Poor Scene Graph Detection Results on Custom Images

[rayush7]

❓ Questions and Help

Hi @KaihuaTang

Thanks for the code for scene graph detection. I was trying to run the code for scene graph generation on custom images but the results I am getting are too poor. Any idea what I am doing wrong?

The command I am using this

CUDA_VISIBLE_DEVICES=0 python tools/relation_test_net.py --config-file "configs/e2e_relation_X_101_32_8_FPN_1x.yaml" MODEL.ROI_RELATION_HEAD.USE_GT_BOX False MODEL.ROI_RELATION_HEAD.USE_GT_OBJECT_LABEL False MODEL.ROI_RELATION_HEAD.PREDICTOR CausalAnalysisPredictor MODEL.ROI_RELATION_HEAD.CAUSAL.EFFECT_TYPE TDE MODEL.ROI_RELATION_HEAD.CAUSAL.FUSION_TYPE sum MODEL.ROI_RELATION_HEAD.CAUSAL.CONTEXT_LAYER motifs TEST.IMS_PER_BATCH 1 DTYPE "float16" GLOVE_DIR /home/ayushrai/glove MODEL.PRETRAINED_DETECTOR_CKPT /home/ayushrai/Scene-Graph-Benchmark.pytorch/pretrained_model/causal-motifs-sgdet OUTPUT_DIR /home/ayushrai/Scene-Graph-Benchmark.pytorch/causal-motifs-sgdet-output TEST.CUSTUM_EVAL True TEST.CUSTUM_PATH /home/ayushrai/utility/test/mediaeval_2020/5122 DETECTED_SGG_DIR /home/ayushrai/Scene-Graph-Benchmark.pytorch/custom_output

This is an example of detection of image I am using.

Please help.

[supremelcs]

Hi, I met the same problem with you when I applied the method on the my custom image. The command I used is as follow: CUDA_VISIBLE_DEVICES=0 python -m torch.distributed.launch --nproc_per_node=1 tools/relation_test_net.py --config-file "configs/e2e_relation_X_101_32_8_FPN_1x.yaml" MODEL.ROI_RELATION_HEAD.USE_GT_BOX False MODEL.ROI_RELATION_HEAD.USE_GT_OBJECT_LABEL False MODEL.ROI_RELATION_HEAD.PREDICTOR CausalAnalysisPredictor MODEL.ROI_RELATION_HEAD.CAUSAL.EFFECT_TYPE TDE MODEL.ROI_RELATION_HEAD.CAUSAL.FUSION_TYPE sum MODEL.ROI_RELATION_HEAD.CAUSAL.CONTEXT_LAYER motifs TEST.IMS_PER_BATCH 1 DTYPE "float16" GLOVE_DIR /data/zhangdan/supremelcs/img_retrieval_based_on_scene_graph/glove MODEL.PRETRAINED_DETECTOR_CKPT /data/zhangdan/supremelcs/img_retrieval_based_on_scene_graph/pretrained_models/pretrained_faster_rcnn/model_final.pth OUTPUT_DIR /data/zhangdan/supremelcs/img_retrieval_based_on_scene_graph/checkpoint/causal-motifs-sgdet TEST.CUSTUM_EVAL True TEST.CUSTUM_PATH /data/zhangdan/supremelcs/img_retrieval_based_on_scene_graph/custom_images DETECTED_SGG_DIR /data/zhangdan/supremelcs/img_retrieval_based_on_scene_graph/generated_sgg

❓ Questions and Help

Hi @KaihuaTang

Thanks for the code for scene graph detection. I was trying to run the code for scene graph generation on custom images but the results I am getting are too poor. Any idea what I am doing wrong?

The command I am using this

CUDA_VISIBLE_DEVICES=0 python tools/relation_test_net.py --config-file "configs/e2e_relation_X_101_32_8_FPN_1x.yaml" MODEL.ROI_RELATION_HEAD.USE_GT_BOX False MODEL.ROI_RELATION_HEAD.USE_GT_OBJECT_LABEL False MODEL.ROI_RELATION_HEAD.PREDICTOR CausalAnalysisPredictor MODEL.ROI_RELATION_HEAD.CAUSAL.EFFECT_TYPE TDE MODEL.ROI_RELATION_HEAD.CAUSAL.FUSION_TYPE sum MODEL.ROI_RELATION_HEAD.CAUSAL.CONTEXT_LAYER motifs TEST.IMS_PER_BATCH 1 DTYPE "float16" GLOVE_DIR /home/ayushrai/glove MODEL.PRETRAINED_DETECTOR_CKPT /home/ayushrai/Scene-Graph-Benchmark.pytorch/pretrained_model/causal-motifs-sgdet OUTPUT_DIR /home/ayushrai/Scene-Graph-Benchmark.pytorch/causal-motifs-sgdet-output TEST.CUSTUM_EVAL True TEST.CUSTUM_PATH /home/ayushrai/utility/test/mediaeval_2020/5122 DETECTED_SGG_DIR /home/ayushrai/Scene-Graph-Benchmark.pytorch/custom_output

This is an example of detection of image I am using.

Please help.

Have you solved the problem ? I see that you have closed the issue. Thanks a lot !

[Nntraveler]

I've faced the problem as well. The result is possibly caused by the failure to load downloaded model (e.g. put model under checkpoints/upload-causal-motifs-sgdet rather than checkpoints/causal-motifs-sgdet which is contained in command) After I unzip upload_causal_motif_sgdet.zip to the right path: /path/to/your_folder/checkpoints/causal-motifs-sgdet and modify last_checkpoint & config, the problem is solved.

[zfchang]

I've faced the problem as well. The result is possibly caused by the failure to load downloaded model (e.g. put model under checkpoints/upload-causal-motifs-sgdet rather than checkpoints/causal-motifs-sgdet which is contained in command) After I unzip upload_causal_motif_sgdet.zip to the right path: /path/to/your_folder/checkpoints/causal-motifs-sgdet and modify last_checkpoint & config, the problem is solved.

I've faced the problem as well. The result is possibly caused by the failure to load downloaded model (e.g. put model under checkpoints/upload-causal-motifs-sgdet rather than checkpoints/causal-motifs-sgdet which is contained in command) After I unzip upload_causal_motif_sgdet.zip to the right path: /path/to/your_folder/checkpoints/causal-motifs-sgdet and modify last_checkpoint & config, the problem is solved.

同学您好，我也遇到了这个问题，路径是对了，但还是有这个问题，想请教下您是怎么修改的last_chekpoint&config，是改的文件名字吗？谢谢

[Nntraveler]

您看一下，就是解压完upload_causal_motif_sgdet.zip以后，里面有last_checkpoint和config.yml两个文件。config文件里 我改了14行glove 72行PRETRAINED_DETECTOR 和332~334 OUTPUT_DIR那边，把他们都改到本机对应路径。然后last_checkpoint打开以后就直接改成pth文件的路径，比如/path/to/your_folder/checkpoints/causal-motifs-sgdet/model_0028000.pth

[zfchang]

您看一下，就是解压完upload_causal_motif_sgdet.zip以后，里面有last_checkpoint和config.yml两个文件。config文件里 我改了14行glove 72行PRETRAINED_DETECTOR 和332~334 OUTPUT_DIR那边，把他们都改到本机对应路径。然后last_checkpoint打开以后就直接改成pth文件的路径，比如/path/to/your_folder/checkpoints/causal-motifs-sgdet/model_0028000.pth

多谢多谢！我按照您的说法，把last_checkpoint还有config.yml文件都改了，但依然存在模型加载不对的问题，检测结果混乱，想请教一下命令行中下面这两处还需要更改吗？ --config-file "/home/changzf/project/Scene-Graph-Benchmark.pytorch/configs/e2e_relation_X_101_32_8_FPN_1x.yaml" MODEL.PRETRAINED_DETECTOR_CKPT /home/changzf/project/Scene-Graph-Benchmark.pytorch/checkpoints/causal-motifs-sgdet

我没看明白PRETRAINED_DETECTOR_CKPT 是指向的存放last_checkpoint和config.yml文件的causal-motifs-sgdet文件夹，并没有指向具体模型，程序怎么才能加载我们修改后的这两个文件呢？因为调了好久，也浏览了这上面很多回答，都没有解决这个问题，感谢同学指点！

[Nntraveler]

命令应该没有什么问题，呃 方便的话您可以贴一下causal-motifs-sgdet文件夹里有哪些东西吗，我这里应该是有

config.yml  labels.json      log.txt            VG_stanford_filtered_with_attribute_train_statistics.cache
inference   last_checkpoint  model_0028000.pth

把模型文件也一起解压到这里了

[Sonaliam]

I did all the things mentioned above but not able to solve the problem yet. I am also facing same issues of poor result.

[ckmstydy]

命令应该没有什么问题，呃 方便的话您可以贴一下causal-motifs-sgdet文件夹里有哪些东西吗，我这里应该是有

config.yml  labels.json      log.txt            VG_stanford_filtered_with_attribute_train_statistics.cache
inference   last_checkpoint  model_0028000.pth

把模型文件也一起解压到这里了

您好，我在运行relation_test_net.py时，cfg.merge_from_file(args.config_file)行出现一个bug：KeyError: 'Non-existent config key: MODEL.RELATION_ON'。。。。但是我看cfg、config_file里面都有这个Key，请问您遇到过吗？

[ch-ho00]

Fixed the issue by changing Line 76 of tools/relation_test_net.py

From

    _ = checkpointer.load(cfg.MODEL.WEIGHT)

to

    _ = checkpointer.load(cfg.MODEL.PRETRAINED_DETECTOR_CKPT)

seems like the pretrained weight were not loading

[leonodelee]

您看一下，就是解压完upload_causal_motif_sgdet.zip以后，里面有last_checkpoint和config.yml两个文件。config文件里 我改了14行glove 72行PRETRAINED_DETECTOR 和332~334 OUTPUT_DIR那边，把他们都改到本机对应路径。然后last_checkpoint打开以后就直接改成pth文件的路径，比如/path/to/your_folder/checkpoints/causal-motifs-sgdet/model_0028000.pth

您好，我配置好文件，但是只能得到custom_data_info.json #得不到custom_prediction.json是咋回事呢，总是运行100%然后被kill

具体问题详情见：

100%|██████████| 12560/12560 [2:59:16<00:00, 1.17it/s] 2021-12-08 10:28:41,551 maskrcnn_benchmark.inference INFO: Total run time: 2:59:16.765582 (0.8564644756246521 s / img per device, on 2 devices) 2021-12-08 10:28:41,551 maskrcnn_benchmark.inference INFO: Model inference time: 2:53:04.057912 (0.8267891167920459 s / img per device, on 2 devices) Traceback (most recent call last): File "/home/anaconda3/envs/scene_graph_benchmark/lib/python3.7/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/home/anaconda3/envs/scene_graph_benchmark/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/anaconda3/envs/scene_graph_benchmark/lib/python3.7/site-packages/torch/distributed/launch.py", line 261, in main() File "/home/anaconda3/envs/scene_graph_benchmark/lib/python3.7/site-packages/torch/distributed/launch.py", line 257, in main cmd=cmd) subprocess.CalledProcessError: Command '['/home/anaconda3/envs/scene_graph_benchmark/bin/python', '-u', 'tools/relation_test_net.py', '--local_rank=1', '--config-file', 'configs/e2e_relation_X_101_32_8_FPN_1x.yaml', 'MODEL.ROI_RELATION_HEAD.USE_GT_BOX', 'False', 'MODEL.ROI_RELATION_HEAD.USE_GT_OBJECT_LABEL', 'False', 'MODEL.ROI_RELATION_HEAD.PREDICTOR', 'CausalAnalysisPredictor', 'MODEL.ROI_RELATION_HEAD.CAUSAL.EFFECT_TYPE', 'TDE', 'MODEL.ROI_RELATION_HEAD.CAUSAL.FUSION_TYPE', 'sum', 'MODEL.ROI_RELATION_HEAD.CAUSAL.CONTEXT_LAYER', 'motifs', 'TEST.IMS_PER_BATCH', '2', 'DTYPE', 'float16', 'GLOVE_DIR', '/data4/glove', 'MODEL.PRETRAINED_DETECTOR_CKPT', '/data4/Scene-Graph-Benchmark.pytorch/checkpoints/causal-motifs-sgdet', 'OUTPUT_DIR', '/data4/Scene-Graph-Benchmark.pytorch/checkpoints/causal-motifs-sgdet', 'TEST.CUSTUM_EVAL', 'True', 'TEST.CUSTUM_PATH', '/data4/Scene-Graph-Benchmark.pytorch/datasets/textcaps/train_images', 'DETECTED_SGG_DIR', '/data4/Scene-Graph-Benchmark.pytorch/datasets']' died with <Signals.SIGKILL: 9>.

[Yudezhi]

因为那个文件特别大，可能长时间存储之类的，被服务器杀死了

[cpperrpr]

是我看cfg、config_file里面都有这个Key，请

你好我也遇到同了同样的问题，请问您解决了吗？

[wywwwww]

命令应该没有什么问题，呃 方便的话您可以贴一下causal-motifs-sgdet文件夹里有哪些东西吗，我这里应该是有

config.yml  labels.json      log.txt            VG_stanford_filtered_with_attribute_train_statistics.cache
inference   last_checkpoint  model_0028000.pth

把模型文件也一起解压到这里了

我想请问一下为什么我下载解压后没有inference这个文件呢？

[zl535320706]

命令应该没有什么问题，呃 方便的话您可以贴一下causal-motifs-sgdet文件夹里有哪些东西吗，我这里应该是有

config.yml  labels.json      log.txt            VG_stanford_filtered_with_attribute_train_statistics.cache
inference   last_checkpoint  model_0028000.pth

把模型文件也一起解压到这里了

您好，我在运行relation_test_net.py时，cfg.merge_from_file(args.config_file)行出现一个bug：KeyError: 'Non-existent config key: MODEL.RELATION_ON'。。。。但是我看cfg、config_file里面都有这个Key，请问您遇到过吗？

please reference #119

[zl535320706]

是我看cfg、config_file里面都有这个Key，请

你好我也遇到同了同样的问题，请问您解决了吗？

please reference #119

[majianbo3]

Hello, I have been customizing my own dataset recently. If you could share your work with me, I would greatly appreciate it，and my QQ:1551924948 email is 1551924948@qq.com