Fine-tune SAM on challenging cell segmentation images

[sleeping4cat]

Recently, there was a slew of tutorials and research papers available for SAM with which you can fine-tune SAM for pretty much any task. But, I was recently trying to fine-tune the decider of SAM on my cell segmentation (binary classification) problem and it gave me an error. Most tutorials are made in mind for datasets where the segmentation part can be easily encapsulated in a bounding box but my image can't be done as that. As there are maybe 30 different cells in one image and it is quite difficult to place bounding box around those cells.

The problem is not training SAM as SAM easily trains on my images rather it it is the moment of plotting the predicted masks by the model. I have tried experimenting with a few code and wrote some functions as well but they all fails. That's why, I am turning to the community if they could provide some support/guidance on this issue. It'll greatly help me.

References

• https://medium.com/@rekalantar/segment-anything-model-sam-for-medical-image-segmentation-9344ba57f2ca
• https://github.com/NielsRogge/Transformers-Tutorials/blob/master/SAM/Run_inference_with_MedSAM_using_HuggingFace_Transformers.ipynb
• https://github.com/NielsRogge/Transformers-Tutorials/blob/master/SAM/Fine_tune_SAM_(segment_anything)_on_a_custom_dataset.ipynb

Image in my Dataset (Mask) Image Copyright belongs to me and my team

[lxr-1204]

Hello, I also encountered such a problem. I have two ideas:

1. Use the image algorithm to generate the minimum circumscribed rectangle according to each mask, such as cv2.findContours; but using this method may cause problems in the output of the network.
2. Convert the binary mask into low_res_masks_np, and use SAM to Iterate, but I encountered obstacles when dealing with this step. For example the following picture

If you have similar problems, please discuss with me。

[matiasmsm]

Hi!

Could any of you solve this issue? I'm facing the same problems with vegetation segmentation and classification.

Any information would be much appreciated