Open icecap360 opened 2 months ago
@jinbae
It looks like DETR weights were used rather than EGTR weights. Would you like to try again using this link?
There are several problems with this.
First, for Vis Genome model, the labels (150 total) are abstract (LABEL_0 to LABEL_149). I checked, and OI model's config labels are also abstract. So, how do I know what are the objects detected? The labels are defined in "config.json" file - can we assume to use the first 150 COCO labels? From here: https://github.com/nightrome/cocostuff/blob/master/labels.md
when executing __load_pretrainedmodel, here is what I see for missing_keys (which are the modules in initialized Deformable DETR - modules loaded from checkpoint dict states). Essentially, these are modules that are initialized in DETR (based on config) but are not loaded from state dict, so they are un-initialized (total of 22), not "pre-trained":
['proj_q.1.weight', 'connectivity_layer.layers.0.bias', 'rel_predictor.layers.2.bias', 'rel_predictor.layers.1.bias', 'proj_q.3.bias', 'proj_k.3.weight', 'proj_q.3.weight', 'proj_k.4.bias', 'connectivity_layer.layers.1.weight', 'rel_predictor.layers.2.weight', 'proj_k.5.weight', 'proj_k.1.weight', 'proj_k.1.bias', 'connectivity_layer.layers.0.weight', 'proj_q.5.weight', 'rel_predictor.layers.1.weight', 'rel_predictor.layers.0.weight', 'proj_q.1.bias', 'connectivity_layer.layers.2.bias', 'final_obj_proj.bias', 'proj_q.0.bias', 'proj_k.2.bias', 'proj_q.2.weight', 'proj_q.4.bias', 'final_obj_proj.weight', 'triplet_dist', 'connectivity_layer.layers.1.bias', 'proj_k.0.weight', 'rel_predictor_gate.bias', 'proj_k.2.weight', 'final_sub_proj.weight', 'connectivity_layer.layers.2.weight', 'proj_q.4.weight', 'proj_k.4.weight', 'proj_q.2.bias', 'proj_q.5.bias', 'proj_k.3.bias', 'proj_k.0.bias', 'rel_predictor_gate.weight', 'rel_dist', 'proj_q.0.weight', 'rel_predictor.layers.0.bias', 'proj_k.5.bias', 'final_sub_proj.bias']
It looks like DETR weights were used rather than EGTR weights. Would you like to try again using this link?
@jinbae - this is the same file, it does not contain the modules for relation prediction (see above), so I think this is a checkpoint with DETR's state dict and not a full EGTR model's
@icecap360 and @jinbae: I think I somewhat figured out the steps:
(1) The Deformable DETR model is initialized via call:
model = DetrForSceneGraphGeneration.from_pretrained( args.architecture, config=config, ignore_mismatched_sizes=True )
At this point, there are warnings, because the "architecture" by default is "SenseTime/deformable-detr", which does not match the EGTR config with 150 class labels and 300 queries. But this warning can be ignored.
(2) After, we load the state dict from the specific checkpoint.
Here are two issues:
Two questions for @jinbae :
@jinbae - found 150 labels used in VG-150 dataset. Still, would be nice to have the config.json fixed with these. I am attaching the updated config I did.
So, only one thing remaining - would you be able to post a checkpoint with full EGTR model (and not DETR detector, as is currently)?
Thanks! config.json
I downloaded and checked the files again and there were no problems.
Please check that you have downloaded and loaded the correct file.
dataset/visual_genome/test.json
dataset/open-imagev6/annotations/categories_dict.json
You can find object categories in the paths above. Please note that object categories in the config are not important in the process of training or evaluation.
@jinbae - :
The readme explains how to download datasets. https://github.com/naver-ai/egtr?tab=readme-ov-file#download-dataset
However, since you don't seem to be using those datasets, I've uploaded the object categories directly here.
The Visual Genome project distribution does not seem to have this file - at least I could not find it. It has the object.json file, but it's the annotations of all objects in all images. It has a synsets file, but it's not the same thing, and has no IDs.
Checked with the VG-150 labels that I found elsewhere, and these seem to perfectly align. Many thanks!
Using the checkpoint for visual genome, I get the following errors when I try run
evaluate_egtr.py
. The error relates to missing weights and size mismatches.