search

lxtGH / OMG-Seg

OMG-LLaVA and OMG-Seg codebase [CVPR-24 and NeurIPS-24]
Other
1.3k stars 49 forks source link

How to increase vocabulary? #23

Closed lihengtao closed 3 months ago

lihengtao commented 4 months ago

Hi, thanks for your impressive work!

I am trying to use OMG-Seg to generate masks for some images. I tried the demo and encountered some problems. For example, the mask of the shoes is labeled as "person". I believe this is because "shoe" is not in COCO's vocabulary.

2024-07-12_19-51-47-106413

I tried using LVIS's classes to replace COCO's classes and used tools/gen_cls.py to generate embeddings. I then used these embeddings to generate a mask, but the result is still not satisfactory.

2024-07-12_19-38-58-958744

Could you provide some advice?

lxtGH commented 3 months ago

Hi! @lihengtao. Thanks for providing the feedback on OMG-Seg.

According to your setting, I think the vocabulary size of OMG-Seg is not suitable for your cases since we only evaluate the classes embedding in COCO or ADE label space following the previous open-vocabulary segmentation works.

However, maybe you can try our recent ECCV work for your setting to label objects with increased vocabulary size.

https://github.com/HarborYuan/ovsam