search

amazon-science / tubelet-transformer

This is an official implementation of TubeR: Tubelet Transformer for Video Action Detection
https://openaccess.thecvf.com/content/CVPR2022/supplemental/Zhao_TubeR_Tubelet_Transformer_CVPR_2022_supplemental.pdf
Apache License 2.0
71 stars 17 forks source link

Question about the DETR pretraining process #18

Open jinsingsangsung opened 1 year ago

jinsingsangsung commented 1 year ago

Thanks for the impressive work. I have one question about the pretraining process of DETR (of which you've mentioned here: https://github.com/amazon-science/tubelet-transformer#training)

From here (https://github.com/amazon-science/tubelet-transformer/issues/4#issuecomment-1236167059), I figured that you've brought the DETR weights trained on COCO dataset and re-trained it on AVA to detect human instances.

  1. Could you describe this process in a more detailed way? (e.g., how did you manipulated the DETR structure to only detect human, what exactly was the input, position embedding, ... etc)
  2. Was your intention of this pretraining to make queries focus more on classification after DETR architecture of TubeR learns how to localize actors well enough?
  3. Have you tried training the whole architecture without the pretrained DETR weights? I've tried several times but could not find a good configuration to make the actual learning happen.

Thanks in advance.