Reproduction of MDef-DETR minus Language

[curiosity654]

Hi, thank you for the great work! I met with some problems when trying to reproduce the MDef-DETR minus Language. The first problem is code. I did not find explicit instructions or code for training the minus language version of the model in the main branch, so I switched to the branch mdef_detr_minus_language, is this the correct way? The second problem is the annotation. After switching the branch, I find the annotation file in pretrain.json changed from OpenSource to OpenSource_CategoryID_Corrected, but I didn't find a link to download it. How is such annotation produced, or could you please provide the processed version? I really appreciate any help you can provide.

[curiosity654]

Also, I have another question about the meaning of language structure in Sec 4.2. When the dataloader keeps the same as captioning, will some objects in an image be treated as negative samples in training? Why does this cause no harm to the overall detection performance?

[mmaaz60]

Hi @curiosity654,

Thank you for your interest in our work. You are right, the instructions and code for training the _minuslanguage model are available at _mdef_detr_minuslanguage branch. The corresponding commit is linked as module in our main GitHub repo at https://github.com/mmaaz60/mvits_for_class_agnostic_od/tree/main/training.

The _minuslanguage model has a binary classification head (object vs no object). The original annotations are modified for this and are available at this link for you to download.

Yes, you are right. However, it could be otherwise acting as a regularization. For example, in the same epoch, in one iteration only a subset of the objects are learned. However, at the end of epoch the model has seen every object in a particular image. A further exploration (both quantitative and qualitative) would be helpful in this regard.