search

ibaiGorordo / ONNX-SAM2-Segment-Anything

Python scripts for the Segment Anythin 2 (SAM2) model in ONNX
MIT License
129 stars 7 forks source link

Jitter in the results provided. #1

Closed charchit7 closed 1 month ago

charchit7 commented 1 month ago

Hey, nice work. Just wanted to know if the jitter in the result was due to model or your pipeline? For example, the two dog images you have provided have jitter in the masks.

ibaiGorordo commented 1 month ago

It is possible that I might have missed something in the conversion. Also, some of the postprocessing is not the same (for example the mask upscaling is smoother in the official repo).

Here is the result comparison:

Official repo: dogs_output

This repo: sam2_masked_img

charchit7 commented 1 month ago

Got it, I'll also look into it. Also, you kept the online demo to be 720p, can it handle 4k?

charchit7 commented 1 month ago

@ibaiGorordo would it be possible for you to share the onnx converted decoder and encoder?

ibaiGorordo commented 1 month ago

Added them to the release: https://github.com/ibaiGorordo/ONNX-SAM2-Segment-Anything/releases/tag/0.2.0

ibaiGorordo commented 1 month ago

Results after the latest changes: sam2_masked_img

charchit7 commented 1 month ago

Hey, nice, by not updating the masks = self.sam2.update_mask(True) ?

ibaiGorordo commented 1 month ago

Modified the Pytorch model for the export to multimask_output=False, and changed the code in this repository accordingly

I recommend pulling the latest changes and download the new models from: https://github.com/ibaiGorordo/ONNX-SAM2-Segment-Anything/releases/tag/0.2.0

charchit7 commented 1 month ago

Got it, thank you so much. @ibaiGorordo will update it :)

charchit7 commented 3 weeks ago

Hey, are you planning to convert sam2_hiera_large model too. Currently I am trying to get model from https://huggingface.co/vietanhdev/segment-anything-2-onnx-models/tree/main to try your codebase.

charchit7 commented 3 weeks ago

Also, while converting the model to onnx, my system is getting crashed. 

(cs_seg) charchit@a10x1-4v-16g-s1-0-dy-compute-1:~$ python /home/charchit/private_repo/segment-anything-2/convert_onnx.py
torch.Size([1, 32, 256, 256])
torch.Size([1, 64, 128, 128])
torch.Size([1, 256, 64, 64])
/home/charchit/private_repo/segment-anything-2/sam2/modeling/backbones/utils.py:30: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if pad_h > 0 or pad_w > 0:
/home/charchit/private_repo/segment-anything-2/sam2/modeling/backbones/utils.py:60: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if Hp > H or Wp > W:
Killed

using your colab code only.
charchit7 commented 3 weeks ago

Ram issues, I think.

ibaiGorordo commented 3 weeks ago

Hey, are you planning to convert sam2_hiera_large model too. Currently I am trying to get model from https://huggingface.co/vietanhdev/segment-anything-2-onnx-models/tree/main to try your codebase.

Just added the large model to the latest release and renamed the decoder