search

om-ai-lab / OmDet

Real-time and accurate open-vocabulary end-to-end object detection
Apache License 2.0
1.48k stars 143 forks source link

Integrating OmDet Turbo in Transformers 🤗 #13

Open yonigozlan opened 2 months ago

yonigozlan commented 2 months ago

Hi Om people!

I am an MLE at Hugging Face, and given the popularity and performance of your model, we wanted to see if you would be interested in working with us to integrate OmDet Turbo into the Transformers 🤗 library. Looking forward to hearing back from you!

Best, Yoni

hx621 commented 1 month ago

@yonigozlan Hello Yoni! Thank you for reaching out and for your interest in integrating OmDet Turbo into the Transformers 🤗 library. We're excited about the possibility of collaborating with Hugging Face on this project. Let's discuss the details and next steps to make this happen.

yonigozlan commented 1 month ago

Hi @hx621, great to hear back from you! The integration of OmDetTurbo is a current work in progress over here 🤗. Please tell us if you spot anything wrongly implemented or any subtleties we have missed! Thank you. Yoni

yonigozlan commented 1 month ago

Hi again @hx621, @P3ngLiu. Does this model support batch inference with different labels and prompt for different images in the same batch? I'm trying to add support for it in the Transformers implementation, however it seems like I'm getting inconsistent results when using labels list of different length in the same batch. Is this a known behavior of the model? It could also be a problem with the Transformers implementation.

hx621 commented 1 month ago

Hi again @hx621, @P3ngLiu. Does this model support batch inference with different labels and prompt for different images in the same batch? I'm trying to add support for it in the Transformers implementation, however it seems like I'm getting inconsistent results when using labels list of different length in the same batch. Is this a known behavior of the model? It could also be a problem with the Transformers implementation.

@yonigozlan During training, each image within the same batch has different labels and prompts. Therefore, inference should also support different images with different labels, and I have already verified this. You can simply modify our demo to implement this feature。 a）provide different labels for each input image https://github.com/om-ai-lab/OmDet/blob/542ce974ee22e16f9e532500e3f84e4702c03abf/omdet/inference/det_engine.py#L72
b) Change here to labels unique to each image https://github.com/om-ai-lab/OmDet/blob/542ce974ee22e16f9e532500e3f84e4702c03abf/omdet/inference/det_engine.py#L85

yonigozlan commented 2 weeks ago

Hi Om team! @hx621 @P3ngLiu OmDet-Turbo is almost ready to be released on Transformers 🤗! As a reminder, here is the PR. Last step before releasing is to upload the Transformers version of the weights on the hub under your organization name. What would be your preferred name? We were thinking of omlab/omdet-turbo-swin-tiny-hf. Indeed, since there is already a checkpoint uploaded for OmdetTurbo swin tiny under your organization name (omlab/OmDet-Turbo_tiny_SWIN_T), and if you would like to keep this one, it would be nice to have a "...-hf" suffix to the Transformers version of the weights. Looking forward to your answer!

P3ngLiu commented 1 week ago

@yonigozlan Thank you very much for integrating our model! We can't wait to see Omdet-Turbo on Transformers. The name omlab/omdet-turbo-swin-tiny-hf sounds good to us as well. Will you be uploading the model, or do we need to upload it ourselves?

yonigozlan commented 2 days ago

@P3ngLiu It might be simpler if we do it on our side since we'll have the weights ready to go if that's ok with you

P3ngLiu commented 2 days ago

@yonigozlan okok, you can do it.