search

huggingface / transformers

🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.
https://huggingface.co/transformers
Apache License 2.0
135.03k stars 27.02k forks source link

LXMERT visual feature extraction during training/fine-tuning phase #7261

Closed mmiakashs closed 3 years ago

mmiakashs commented 4 years ago

🚀 Feature request

Thanks a lot for releasing LXMERT model. In the LXMERT model code samples, the visual feature extraction code (using generalized faster-rcnn: modeling_frcnn) only in the inference step is given. However, the visual feature extraction during the training phase is not given. For this reason if we use the same code for fine-tuning, it raises NotImplementedError as the visual feature extraction during training is not implemented. Is it possible to share the visual feature extraction during training?

TashinAhmed commented 4 years ago

Yes, I also came up with this error. It would be great if the feature gets published. TIA.

LysandreJik commented 4 years ago

Tagging LXMERT's implementation author @eltoto1219

eltoto1219 commented 4 years ago

Haha, yes we only added the FRCNN for evaluation to accommodate lxmert in the demo. I'll add the training code sometime this week, and then post back here once it is done, in the future it may be useable as a publicly available model following the HF api, but for the time being ill just push the changes to where it is now.

TashinAhmed commented 4 years ago

Thanks for the prompt feedback. Looking forward to it. @eltoto1219

mmiakashs commented 4 years ago

@eltoto1219 thanks, that will be quite a help.

xhyandwyy commented 4 years ago

@eltoto1219 Looking forward to it.

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

LetiP commented 3 years ago

Hello, any updates on this? 😃

eltoto1219 commented 3 years ago

Hi @LetiP,

My apologies for the delay! I actually have a couple of conference deadlines mid-January and also some other projects after that, so my free time to implement training code for the FRCNN is unfortunately very limited. I think if I can still manage to add this functionality, it may not be ready until sometime in May. However, the code used for the FRCNN here was majorly adapted from Facebook's detectron2 library. I can point you to the source training code incase you need this functionality sooner!

here is the file for the region proposal network: https://github.com/facebookresearch/detectron2/blob/e0e166d864a2021a15a2bc2c9234d04938066265/detectron2/modeling/proposal_generator/rpn.py#L402

here is the file for the box matcher: https://github.com/facebookresearch/detectron2/blob/master/detectron2/modeling/matcher.py

some utils for the rpn: https://github.com/facebookresearch/detectron2/blob/master/detectron2/modeling/proposal_generator/proposal_utils.py

code for the frcnn output predictions: https://github.com/facebookresearch/detectron2/blob/e0e166d864a2021a15a2bc2c9234d04938066265/detectron2/modeling/roi_heads/fast_rcnn.py#L433

not completely sure if changes are needed in this file for training: https://github.com/facebookresearch/detectron2/blob/master/detectron2/modeling/roi_heads/box_head.py

roi head logic: https://github.com/facebookresearch/detectron2/blob/e0e166d864a2021a15a2bc2c9234d04938066265/detectron2/modeling/roi_heads/roi_heads.py#L307

I may be able to provide some quick pointers if you run into anything that seems impossible to get working by replying more to this thread!

ppwwyyxx commented 3 years ago

Rather than trying to "add training functionality" to the custom copy of an old subset of detectron2 in this repo, I can't see why you cannot just use detectron2 directly. That would not only provide the training functionality out of the box, but also probably reduce the 3000 lines of duplicated unmaintained code here into like 50 lines.

AhmedMasryKU commented 3 years ago

I want to use a frcnn model that is trained on a custom dataset. I followed the tutorials in the original detectron2 repo (Colab Notebooks in https://github.com/facebookresearch/detectron2). However, I noticed that the config file architecture for your pretrained model is different from mine. For example, this is the model part in your config file "model : load_proposals: false device: cpu max_pool: true chkpoint: "" pixel_mean: [102.9801, 115.9465, 122.7717] pixel_std: [1.0, 1.0, 1.0]"

And this is mine: "MODEL: ANCHOR_GENERATOR: ANGLES:

Could you please provide any resources how we can use our own trained frcnn models ?

github-actions[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

mmiakashs commented 3 years ago

Any update on this 🙂 ?