MASA: Matching Anything By Segmenting Anything (CVPR24)

[johnnynunez]

Search before asking

• [X] I have searched the Yolov8 Tracking issues and found no similar enhancement requests.

Description

https://github.com/siyuanliii/masa

Use case

No response

Are you willing to submit a PR?

• [ ] Yes I'd like to help by submitting a PR!

[mikel-brostrom]

Wow, this looks very promising

[mikel-brostrom]

From MASA:

"Additionally, our learned detection head speeds up the original SAM dense uniform point proposals for segmenting everything by over tenfold, crucial for tracking applications."

"We treat the SAM outputs as dense object region proposals and learn to match those regions from a vast image collection. We further design a universal MAS encoder: A heavy ViT-based backbone for feature extraction."

From SAM:

"Given a precomputed image embedding, the prompt encoder and mask decoder run in a web browser, on CPU, in ∼50ms."

SAM uses ViT as image embedder, from ViT:

"ViT-B (Base Model): Inference times on high-end GPUs (such as NVIDIA V100 or A100) are typically around 20-40 milliseconds per image."

ViT-L (Large Model): Inference times are generally longer, around 50-80 milliseconds per image, depending on the exact setup and image resolution.

ViT-H (Huge Model): Inference times can exceed 100 milliseconds per image due to the increased model complexity and size.

This would require to get ViT working here as embedder to start with

[mikel-brostrom]

This:

This would require to get ViT working here as embedder to start with

may not be a problem anymore 😄, given that Ultralytics has a ViT encoder

[github-actions[bot]]

👋 Hello, this issue has been automatically marked as stale because it has not had recent activity. Please note it will be closed if no further activity occurs. Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

[mikel-brostrom]

Add model under boxmot/appearance/backbones. And model weights url (https://huggingface.co/dereksiyuanli/masa/resolve/main/gdino_masa.pth). Adapt: https://github.com/siyuanliii/masa/blob/main/masa/apis/masa_inference.py

[johnnynunez]

so is it compatible now your repo?

[mikel-brostrom]

I have been investigating this thoroughly today. The whole architecture build using mmdet needs to be ported to pytorch. This is no easy feat due to many custom implementations and optimizations. I don't have the time for such a research project on my free time at the moment

[rolson24]

I was working on this a few weeks ago to integrate with huggingface, but never got around to finishing it. Would be happy to share my code from it.

On Mon, Sep 9, 2024 at 12:26 PM Mike @.***> wrote:

I have been investigating this throughly today. The whole architecture build using mmdet needs to be ported to pytorch. This is no easy feat due to many custom implementations and optimizations

— Reply to this email directly, view it on GitHub https://github.com/mikel-brostrom/boxmot/issues/1474#issuecomment-2338558512, or unsubscribe https://github.com/notifications/unsubscribe-auth/AX2EJPHM5NNB5OT7BD6FEWTZVXD4VAVCNFSM6AAAAABJEIK5Q6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMZYGU2TQNJRGI . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[mikel-brostrom]

I was working on this a few weeks ago to integrate with huggingface, but never got around to finishing it. Would be happy to share my code from it.

Hey @rolson24! Hope everything is fine. That would be awesome. So just the architecture part?

[microchila]

MASA is very powerful, will it be supported in the future?

[mikel-brostrom]

MASA is very powerful, will it be supported in the future?

It will take some time due to the need of an architecture port from mmdet to pure pytorch. But it is ongoing