Required files for SGDET_on_custom_images.ipynb

[claudio-dg]

Hi @Maelic , first of all thank: you for sharing your precious work! I am currently trying to run the Scene Graph generation on custom images (i.e. SGDET_on_custom_images.ipynb) but I am struggling to understand where to find the correct files to put in the configuration, dictionary and weights paths. To be more clear, I am talking about the following lines of code in your notebook:

config_path = "../checkpoints/VG150/SGDET/penet-yolov8m/config.yml" dict_path = "../datasets/VG150/VG-SGG-dicts-with-attri.json" weights_path = "../checkpoints/VG150/SGDET/penet-yolov8m/best_model_epoch_2.pth"

Could you please help me providing further information and/or links to get the required files?

[Maelic]

Hi @claudio-dg ,

I have just uploaded the model weights for penet-yolov8m on VG150 and two other datasets, please have a look in the MODEL_ZOO.

Just to let you know, you will probably have to modify the config.yml file which I uploaded with the weights as there might be some wrong paths. Don't hesitate to try other models as well!

Good luck!

[claudio-dg]

Thank you for your feedback @Maelic, I really appreciate it! I was trying to modify the config.yaml file and path_catalog.py as you suggested, but now I am stuck at this runtime error regarding the split mask. Would you be so kind to help me understanding how to solve it?

YOLOv8m summary: 295 layers, 25943170 parameters, 25943154 gradients, 79.5 GFLOPs

Split mask: [False False False ... False False False] Traceback (most recent call last): File "/home/claudiodelgaizo/Downloads/SGG-Benchmark-main/my_scripts/SGDET_on_cutom_images.py", line 73, in model = SGG_Model(config_path, dict_path, weights_path, rel_conf=0.01, box_conf=0.1, show_fps=False) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/claudiodelgaizo/Downloads/SGG-Benchmark-main/my_scripts/demo_model.py", line 61, in init self.load_model() File "/home/claudiodelgaizo/Downloads/SGG-Benchmark-main/my_scripts/demo_model.py", line 68, in load_model self.model = build_detection_model(self.cfg) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/claudiodelgaizo/Downloads/SGG-Benchmark-main/sgg_benchmark/modeling/detector/detectors.py", line 11, in build_detection_model return meta_arch(cfg) ^^^^^^^^^^^^^^ File "/home/claudiodelgaizo/Downloads/SGG-Benchmark-main/sgg_benchmark/modeling/detector/generalized_yolo.py", line 28, in init self.roi_heads = build_roi_heads(cfg, self.backbone.out_channels) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/claudiodelgaizo/Downloads/SGG-Benchmark-main/sgg_benchmark/modeling/roi_heads/roi_heads.py", line 69, in build_roi_heads roi_heads.append(("relation", build_roi_relation_head(cfg, in_channels))) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/claudiodelgaizo/Downloads/SGG-Benchmark-main/sgg_benchmark/modeling/roi_heads/relation_head/relation_head.py", line 117, in build_roi_relation_head return ROIRelationHead(cfg, in_channels) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/claudiodelgaizo/Downloads/SGG-Benchmark-main/sgg_benchmark/modeling/roi_heads/relation_head/relation_head.py", line 40, in init self.predictor = make_roi_relation_predictor(cfg, feat_dim) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/claudiodelgaizo/Downloads/SGG-Benchmark-main/sgg_benchmark/modeling/roi_heads/relation_head/roi_relation_predictors.py", line 9, in make_roi_relation_predictor return func(cfg, in_channels) ^^^^^^^^^^^^^^^^^^^^^^ File "/home/claudiodelgaizo/Downloads/SGG-Benchmark-main/sgg_benchmark/modeling/roi_heads/relation_head/predictors/default_predictors.py", line 773, in init super().init(config, in_channels)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/claudiodelgaizo/Downloads/SGG-Benchmark-main/sgg_benchmark/modeling/roi_heads/relation_head/predictors/default_predictors.py", line 34, in init self.statistics = get_dataset_statistics(self.cfg) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/claudiodelgaizo/Downloads/SGG-Benchmark-main/sgg_benchmark/data/build.py", line 54, in get_dataset_statistics statistics.append(dataset.get_statistics()) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/claudiodelgaizo/Downloads/SGG-Benchmark-main/sgg_benchmark/data/datasets/visual_genome.py", line 130, in get_statistics fg_matrix, bg_matrix, predicate_new_order, predicate_new_order_count, pred_prop, triplet_freq, pred_weight = get_VG_statistics(img_dir=self.img_dir, roidb_file=self.roidb_file, dict_file=self.dict_file, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/claudiodelgaizo/Downloads/SGG-Benchmark-main/sgg_benchmark/data/datasets/visual_genome.py", line 275, in get_VG_statistics train_data = VGDataset(split='train', img_dir=img_dir, roidb_file=roidb_file, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/claudiodelgaizo/Downloads/SGG-Benchmark-main/sgg_benchmark/data/datasets/visual_genome.py", line 78, in init self.filenames = [self.filenames[i] for i in np.where(self.split_mask)[0]] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ **File "/home/claudiodelgaizo/Downloads/SGG-Benchmark-main/sgg_benchmark/data/datasets/visual_genome.py", line 78, in self.filenames = [self.filenames[i] for i in np.where(self.split_mask)[0]]

IndexError: list index out of range**

[Maelic]

Did you download the Visual Genome dataset? It is still required to download the dataset, even for inference. I will remove this requirement when I have time. Meanwhile, you can download the images and data file following the DATASET.md

[claudio-dg]

I did but I couldn't unzip it properly into one single folder, I managed to do it now and solved that, thanks again! I hope not to bother you again XD

[Maelic]

No worries, I know the codebase is not very friendly for out-of-the-box inference on custom images, but as I said I have a strictly limited amount of time to work on that right now :)

If it's working now, I'm closing the issue.