search

lxtGH / OMG-Seg

OMG-LLaVA and OMG-Seg codebase [CVPR-24 and NeurIPS-24]
Other
1.32k stars 50 forks source link

Training code for image-based instance segmentation is same? #31

Closed pkyzh2006 closed 3 months ago

pkyzh2006 commented 4 months ago

I run the train code but receive TypeError: The key argument of Registry.get must be a str, got <class 'type'>. Thanks

pcc-99 commented 3 months ago

Have you solved your problem?

pkyzh2006 commented 3 months ago

Have you solved your problem? The author's code is not written in the mmdet standard config format, so be careful and use import, which can let registration issues generally be resolved. My current problem is that I only want to implement image instance segmentation using my own dataset, and I haven't implemented it yet. Mainly because mmdet's errors always leave people confused, and then I give up.

kanglang123 commented 3 months ago

I solve it! my package version is follow:

Linux

cuda 11.7

torch 2.0.1+cu117 pypi_0 pypi torchaudio 2.0.2+cu117 pypi_0 pypi torchvision 0.15.2+cu117 pypi_0 pypi

mmcv 2.0.1 pypi_0 pypi mmcv-lite 2.2.0 pypi_0 pypi mmdet 3.1.0 pypi_0 pypi mmengine 0.8.5 pypi_0 pypi mmpretrain 1.0.1 pypi_0 pypi mmsegmentation 1.1.1 pypi_0 pypi

be careful : mmcv, mmdet, mmcv-lite

Install mmcv-lite If you need to use PyTorch-related modules, make sure PyTorch has been successfully installed in your environment by referring to the PyTorch official installation guide. pip install mmcv-lite

kanglang123 commented 3 months ago

Have you solved your problem? The author's code is not written in the mmdet standard config format, so be careful and use import, which can let registration issues generally be resolved. My current problem is that I only want to implement image instance segmentation using my own dataset, and I haven't implemented it yet. Mainly because mmdet's errors always leave people confused, and then I give up.

I also want to fine-tuning OMG-Seg on my datasets. I am looking for some useful experience about it.

lxtGH commented 3 months ago

@kanglang123 Thanks for your reply. @pcc-99 @pkyzh2006 These errors are environmental problems. Please make sure you have stall mmdet-3.1.0 with correct mmcv package.

lxtGH commented 3 months ago

@kanglang123 @pkyzh2006 For fine-tuning your own datasets, you can refer to the doc of mmdetection here.

https://mmdetection.readthedocs.io/en/latest/advanced_guides/customize_dataset.html

Generate the COCO-style instance segmentation annotations.

Then, if you use OMG-Seg for fine-tuning, please make sure to extract the class embedding from CLIP.

Hope this can help solve your problems.

We will provide one fine-tuning config for COCO datasets for your reference.

kanglang123 commented 3 months ago

@kanglang123 @pkyzh2006 For fine-tuning your own datasets, you can refer to the doc of mmdetection here.

https://mmdetection.readthedocs.io/en/latest/advanced_guides/customize_dataset.html

Generate the COCO-style instance segmentation annotations.

Then, if you use OMG-Seg for fine-tuning, please make sure to extract the class embedding from CLIP.

Hope this can help solve your problems.

We will provide one fine-tuning config for COCO datasets for your reference.

I want to use my self built Coco instance segmentation dataset for training. Can you provide some guidance? I don't know how to configure for instance segmentation, it seems that there is no detection head applied to instance segmentation.

kanglang123 commented 3 months ago

@kanglang123 @pkyzh2006 For fine-tuning your own datasets, you can refer to the doc of mmdetection here. https://mmdetection.readthedocs.io/en/latest/advanced_guides/customize_dataset.html Generate the COCO-style instance segmentation annotations. Then, if you use OMG-Seg for fine-tuning, please make sure to extract the class embedding from CLIP. Hope this can help solve your problems. We will provide one fine-tuning config for COCO datasets for your reference.

I want to use my self built Coco instance segmentation dataset for training. Can you provide some guidance? I don't know how to configure for instance segmentation, it seems that there is no detection head applied to instance segmentation.

I would like to communicate with you. This is my QQ: 2015945124

lxtGH commented 3 months ago

https://github.com/open-mmlab/mmdetection/blob/main/configs/mask2former/mask2former_r50_8xb2-lsj-50e_coco.py Ref this config for dataloader.

https://github.com/lxtGH/OMG-Seg/blob/main/tools/gen_cls.py Use this to generate customized class embedding.

kanglang123 commented 3 months ago

https://github.com/open-mmlab/mmdetection/blob/main/configs/mask2former/mask2former_r50_8xb2-lsj-50e_coco.py Ref this config for dataloader.

https://github.com/lxtGH/OMG-Seg/blob/main/tools/gen_cls.py Use this to generate customized class embedding.

Thanks for your help. I will try it as you mentioned.