Suggestion - Integrate MobileSAM into the pipeline for lightweight and faster inference

[mdimtiazh]

Reference: https://github.com/ChaoningZhang/MobileSAM

Our project performs on par with the original SAM and keeps exactly the same pipeline as the original SAM except for a change on the image encode, therefore, it is easy to Integrate into any project.

MobileSAM is around 60 times smaller and around 50 times faster than original SAM, and it is around 7 times smaller and around 5 times faster than the concurrent FastSAM. The comparison of the whole pipeline is summarzed as follows:

Best Wishes,

Qiao

[Jumpat]

Thanks for your good work and suggestion!🥳 We will try your model and update our project.

[Jumpat]

The integration has been finished.

[akhdanfadh]

https://github.com/Jumpat/SegmentAnythingin3D/blob/33efce5a4c24fe6de6cfa4224e1e92d25b49336e/lib/sam3d.py#L30C1-L31C1

Where does this mobile_encoder from? MobileSAM repo itself installs a package called mobile_sam.

[Jumpat]

Yes, please follow the instruction provided by MobileSAM for the installation🫣, as we mentioned in our Readme:

• 2023/06/29: We now support MobileSAM as the segmentation network. Follow the installation instruction in MobileSAM, and then download mobile_sam.pt into folder ./dependencies/sam_ckpt. You can use --mobile_sam to switch to MobileSAM.

[akhdanfadh]

I already did install them using this

cd dependencies/sam_ckpt/
git clone https://github.com/ChaoningZhang/MobileSAM.git
cd MobileSAM; pip install -e .
cp weights/mobile_sam.pt ../

and now I have mobile_sam package on my pip installed package and their weight checkpoint.

When I try to run python run_seg_gui.py --config=configs/llff/seg/seg_fern.py --segment --sp_name=_gui --num_prompts=20 --render_opt=train --save_ckpt --mobile_sam it throws an error ModuleNotFoundError: No module named 'mobile_encoder'. I am not sure how you integrate MobileSAM to this repo.

[Jumpat]

It seems there have been some modification in MobileSAM since the previous integration. After checking MobileSAM's readme I find currently the backbone loading should be done as follows:

from mobile_sam import sam_model_registry, SamAutomaticMaskGenerator, SamPredictor

model_type = "vit_t"
sam_checkpoint = "./weights/mobile_sam.pt"

device = "cuda" if torch.cuda.is_available() else "cpu"

mobile_sam = sam_model_registry[model_type](checkpoint=sam_checkpoint)
mobile_sam.to(device=device)
mobile_sam.eval()

# the SAM predictor initialization
# predictor = SamPredictor(mobile_sam)

After you get the mobile_sam, just let self.sam = mobile_sam. Change the above mentioned code may help.

We will modify our code to adapt to their update.

[akhdanfadh]

I see. Then please let us know if it is updated, maybe by reopening this issue(?) Thank you!

[Jumpat]

The bug has been fixed😉