Open Mingyuan1997 opened 1 week ago
I also meet this error.
It seems that the object detector is not detecting anything, I will investigate this issue.
I am not able to reproduce the error, can you share your config file please?
You can copy the content of the .yaml file here
您可以在此处复制 .yaml 文件的内容
This is my config.yml file, but I don't know why the PATHS_DATA path is like this. I have already changed the data loading path in paths_catalog.py, and I am using the VG150 dataset. `DATALOADER: ASPECT_RATIO_GROUPING: true NUM_WORKERS: 4 SIZE_DIVISIBILITY: 32 DATASETS: NAME: '' TEST:
Thanks for your investigation. My yaml as follows:
INPUT: MIN_SIZE_TRAIN: 640 MAX_SIZE_TRAIN: 640 MIN_SIZE_TEST: 640 MAX_SIZE_TEST: 640 PADDING: True MODEL: TEXT_EMBEDDING: glove.6B META_ARCHITECTURE: "GeneralizedYOLO" BOX_HEAD: False WEIGHT: "" PRETRAINED_DETECTOR_CKPT: "/home/haozhen/mingyuan/SGG-Benchmark/yolov8m_vg150.pt" BACKBONE: TYPE: "yolov8" NMS_THRESH: 0.001 FREEZE: True YOLO: SIZE: "yolov8m" OUT_CHANNELS: 192 # 192 for yolov8m, 256 for yolov8l RELATION_ON: True ATTRIBUTE_ON: False FLIP_AUG: False # if there is any left-right relation, FLIP AUG should be false ROI_HEADS: POSITIVE_FRACTION: 0.5 FG_IOU_THRESHOLD: 0.3 BG_IOU_THRESHOLD: 0.1 BATCH_SIZE_PER_IMAGE: 256 DETECTIONS_PER_IMG: 80 NMS_FILTER_DUPLICATES: True NMS: 0.2 ROI_BOX_HEAD: POOLER_SCALES: (0.125, 0.0625, 0.03125) POOLER_RESOLUTION: 7 POOLER_SAMPLING_RATIO: 2 FEATURE_EXTRACTOR: "YOLOV8FeatureExtractor" NUM_CLASSES: 151 # 151 for VG, 1201 for GQA ROI_RELATION_HEAD: EMBED_DIM: 200 USE_GT_BOX: False USE_GT_OBJECT_LABEL: False USE_SPATIAL_FEATURES: True REQUIRE_BOX_OVERLAP: False # for sgdet, during training, only train pairs with overlap ADD_GTBOX_TO_PROPOSAL_IN_TRAIN: True # for sgdet only, in case some gt boxes are missing NUM_CLASSES: 51 # 51 for VG, 201 for GQA (not contain "to the left of" & "to the right of") BATCH_SIZE_PER_IMAGE: 256 # sample as much as possible POSITIVE_FRACTION: 0.25 CONTEXT_POOLING_DIM: 2048 CONTEXT_HIDDEN_DIM: 512 #1024 for VCTree POOLING_ALL_LEVELS: True LABEL_SMOOTHING_LOSS: False FEATURE_EXTRACTOR: "RelationFeatureExtractor" #################### Select Relationship Model ####################
#PREDICTOR: "VCTreePredictor"
#PREDICTOR: "TransformerPredictor"
PREDICTOR: "CausalAnalysisPredictor"
################# Parameters for Motif Predictor ##################
CONTEXT_OBJ_LAYER: 1
CONTEXT_REL_LAYER: 1
############# Parameters for Causal Unbias Predictor ##############
### Implementation for paper "Unbiased Scene Graph Generation from Biased Training"
CAUSAL:
EFFECT_TYPE: 'TDE' # candicates: 'TDE', 'NIE', 'TE', 'none'
FUSION_TYPE: 'sum' # candicates: 'sum', 'gate'
SEPARATE_SPATIAL: False # separate spatial in union feature
CONTEXT_LAYER: "motifs" # candicates: motifs, vctree, vtranse
EFFECT_ANALYSIS: True
############### Parameters for Transformer Predictor ##############
TRANSFORMER:
DROPOUT_RATE: 0.1
OBJ_LAYER: 4
REL_LAYER: 2
NUM_HEAD: 8
KEY_DIM: 64
VAL_DIM: 64
INNER_DIM: 2048
DATASETS: TRAIN: ("VG150_train",) VAL: ("VG150_val",) TEST: ("VG150_test",) DATALOADER: SIZE_DIVISIBILITY: 32 SOLVER: MAX_EPOCH: 20 BIAS_LR_FACTOR: 1 BASE_LR: 0.01 WARMUP_FACTOR: 0.1 WEIGHT_DECAY: 0.0001 MOMENTUM: 0.9 GRAD_NORM_CLIP: 5.0 STEPS: (10000, 16000) MAX_ITER: 40000 VAL_PERIOD: 2000 CHECKPOINT_PERIOD: 2000 PRINT_GRAD_FREQ: 4000 SCHEDULE:
TYPE: "WarmupReduceLROnPlateau" # WarmupMultiStepLR, WarmupReduceLROnPlateau
PATIENCE: 2
THRESHOLD: 0.001
COOLDOWN: 0
FACTOR: 0.1
MAX_DECAY_STEP: 3
OUTPUT_DIR: '/checkpoints' TEST: ALLOW_LOAD_FROM_CACHE: False INFORMATIVE: True RELATION: SYNC_GATHER: True # turn on will slow down the evaluation to solve the sgdet test out of memory problem REQUIRE_OVERLAP: True LATER_NMS_PREDICTION_THRES: 0.5 CUSTUM_EVAL: False # eval SGDet model on custum images, output a json CUSTUM_PATH: '.' # the folder that contains the custum images, only jpg files are allowed
感谢您的调查。我的 yaml 如下:
输入: MIN_SIZE_TRAIN:640 MAX_SIZE_TRAIN:640 MIN_SIZE_TEST:640 MAX_SIZE_TEST:640 PADDING:True 模型: TEXT_EMBEDDING:glove.6B META_ARCHITECTURE:“GeneralizedYOLO” BOX_HEAD:False WEIGHT:“” PRETRAINED_DETECTOR_CKPT:“/home/haozhen/mingyuan/SGG-Benchmark/yolov8m_vg150.pt” BACKBONE: 类型:“yolov8” NMS_THRESH:0.001 FREEZE:True YOLO: SIZE:“yolov8m” OUT_CHANNELS:192 # yolov8m 为 192,yolov8l 为 256 RELATION_ON:True ATTRIBUTE_ON:False FLIP_AUG: False # 如果存在任何左右关系,FLIP AUG 应该为 false ROI_HEADS: POSITIVE_FRACTION: 0.5 FG_IOU_THRESHOLD: 0.3 BG_IOU_THRESHOLD: 0.1 BATCH_SIZE_PER_IMAGE: 256 DETECTIONS_PER_IMG: 80 NMS_FILTER_DUPLICATES: True NMS: 0.2 ROI_BOX_HEAD: POOLER_SCALES: (0.125, 0.0625, 0.03125) POOLER_RESOLUTION: 7 POOLER_SAMPLING_RATIO: 2 FEATURE_EXTRACTOR: "YOLOV8FeatureExtractor" NUM_CLASSES: 151 # 151 表示 VG,1201 表示 GQA ROI_RELATION_HEAD: EMBED_DIM:200 USE_GT_BOX:False USE_GT_OBJECT_LABEL:False USE_SPATIAL_FEATURES:True REQUIRE_BOX_OVERLAP:False # 对于 sgdet,在训练期间,仅训练有重叠的对 ADD_GTBOX_TO_PROPOSAL_IN_TRAIN:True # 仅适用于 sgdet,以防缺少某些 gt 框 NUM_CLASSES:51 # 51 用于 VG,201 用于 GQA(不包含“左侧”和“右侧”) BATCH_SIZE_PER_IMAGE:256 # 尽可能多地采样 POSITIVE_FRACTION:0.25 CONTEXT_POOLING_DIM:2048 CONTEXT_HIDDEN_DIM:512 #1024 用于 VCTree POOLING_ALL_LEVELS:True LABEL_SMOOTHING_LOSS:False FEATURE_EXTRACTOR:“RelationFeatureExtractor” #################### 选择关系模型 #################### #PREDICTOR:“MotifPredictor” #PREDICTOR:“VCTreePredictor” #PREDICTOR:“TransformerPredictor” PREDICTOR:“CausalAnalysisPredictor” ################# Motif 预测器的参数 ################## CONTEXT_OBJ_LAYER:1 CONTEXT_REL_LAYER:1 ############# 因果无偏预测器的参数 ############## ### 论文“从有偏训练中生成无偏场景图”的实现 CAUSAL: EFFECT_TYPE:“TDE” #候选人:“TDE”,'NIE', 'TE', 'none' FUSION_TYPE: 'sum' # 候选: 'sum', 'gate' SEPARATE_SPATIAL: False # 在联合特征中分离空间 CONTEXT_LAYER: "motifs" # 候选者: motifs、vctree、vtranse EFFECT_ANALYSIS: True ############### Transformer Predictor 的参数 ############## TRANSFORMER: DROPOUT_RATE: 0.1 OBJ_LAYER: 4 REL_LAYER: 2 NUM_HEAD: 8 KEY_DIM: 64 VAL_DIM: 64 INNER_DIM: 2048 DATASETS: TRAIN: ("VG150_train",) VAL: ("VG150_val",) TEST: ("VG150_test",) DATALOADER: SIZE_DIVISIBILITY: 32 SOLVER: MAX_EPOCH: 20 BIAS_LR_FACTOR: 1 BASE_LR: 0.01 WARMUP_FACTOR: 0.1 WEIGHT_DECAY: 0.0001 MOMENTUM: 0.9 GRAD_NORM_CLIP: 5.0 STEPS: (10000, 16000) MAX_ITER: 40000 VAL_PERIOD: 2000 CHECKPOINT_PERIOD: 2000 PRINT_GRAD_FREQ: 4000 SCHEDULE: # 以下参数仅用于 WarmupReduceLROnPlateau TYPE: "WarmupReduceLROnPlateau" # WarmupMultiStepLR, WarmupReduceLROnPlateau PATIENCE: 2 THRESHOLD: 0.001 COOLDOWN: 0 FACTOR: 0.1 MAX_DECAY_STEP: 3 OUTPUT_DIR: '/checkpoints' TEST: ALLOW_LOAD_FROM_CACHE: False INFORMATIVE: True RELATION: SYNC_GATHER: True # 开启会减慢评估速度,解决 sgdet 测试内存不足问题 REQUIRE_OVERLAP: True LATER_NMS_PREDICTION_THRES: 0.5 CUSTUM_EVAL: False # 在自定义图片上评估 SGDet 模型,输出一个 json CUSTUM_PATH: '.' # 包含自定义图片的文件夹,只允许 jpg 文件
您好!看到您的回复,我觉得您应该是中国人。我是山东大学研一新生。我有一些关于这个项目复现的问题想向您请教,我是sgg初学者,想和您进行交流。我的邮箱是1339241893@qq.com,我的微信是:XC-992997,希望与您加上好友进行交流。期待得到您的回复,希望与您进行交流学习!(如果可以的话,您可以回复我您的微信,我来加您)
After a lot of digging I think I found out the issue, it is related to the latest ultralytics implementation. In a recent release, ultralytics changed their codebase for yolov8, and this impacted my implementation of the yolov8 backbone here.
As a workaround for now, you can fallback to the previous version of ultralytics as follows:
pip install ultralytics==8.2.70
This should solve the issue. I will update the requirements accordingly.
经过大量挖掘,我想我找到了问题所在,它与最新的 ultralytics 实现有关。在最近的一个版本中,ultralytics 更改了 yolov8 的代码库,这影响了我在这里对 yolov8 主干的实现。 作为一种解决方法,您可以按如下方式回退到以前版本的 ultralytics:
pip install ultralytics==8.2.70
这应该可以解决问题。我会相应地更新要求。
I tried the methods you suggested, and the previous error is indeed gone. However, now there's a new error:
`File "/Dpan/wyc/realtime_rwsg/SGG-Benchmark/tools/relation_train_net.py", line 1514, in
KeyError: 'informative_rels'
`
我尝试了你建议的方法,之前的错误确实消失了。但是现在又出现了一个新的错误:
File "/Dpan/wyc/realtime_rwsg/SGG-Benchmark/tools/relation_train_net.py", line 1514, in ~~~^^^^^^^^^^^^^^^^^^^^ KeyError: 'informative_rels'
Thank you so much and I do appreciate tremendous efforts you put in investigating the issue. The detector release is always unstable lol. I am not aware of this since I have not followed the yolo line of work in recent years. Thanks again!
No worries, hope it helps.
By the way, you should put Informative: False
in your .yaml config files to not get bothered by the informative_rels etc @Mingyuan1997 @Young-Loser
您好!看到您的回复,我觉得您应该是中国人。我是山东大学研一新生。我有一些关于这个项目复现的问题想向您请教,我是sgg初学者,想和您进行交流。我的邮箱是1339241893@qq.com,我的微信是:XC-992997,希望与您加上好友进行交流。期待得到您的回复,希望与您进行交流学习!(如果可以的话,您可以回复我您的微信,我来加您)
Thanks for the great work and a quick fix on the informative file!
However I still got into trouble when I tried running the training command. Could you help me to see why I cannot get prediction results from the detector?
I used pretrained weight in your link (yolov8 trained at VG)
CUDA_VISIBLE_DEVICES=0 python tools/relation_train_net.py --task sgdet --save-best --config-file "configs/VG150/e2e_relation_yolov8m.yaml" MODEL.ROI_RELATION_HEAD.PREDICTOR CausalAnalysisPredictor MODEL.ROI_RELATION_HEAD.CAUSAL.EFFECT_TYPE none MODEL.ROI_RELATION_HEAD.CAUSAL.FUSION_TYPE sum MODEL.ROI_RELATION_HEAD.CAUSAL.CONTEXT_LAYER motifs SOLVER.IMS_PER_BATCH 12 TEST.IMS_PER_BATCH 2 DTYPE "float16" SOLVER.MAX_EPOCH 20 MODEL.PRETRAINED_DETECTOR_CKPT ./yolov8m_vg150.pt OUTPUT_DIR ./checkpoints/
2024-10-14 22:40:25.637 | INFO | sgg_benchmark.utils.logger:setup_logger:31 - Using loguru logger with level: INFO 2024-10-14 22:40:25.638 | INFO | main:main:444 - Using 1 GPUs 2024-10-14 22:40:25.638 | INFO | sgg_benchmark.utils.logger:logger_step:15 - #################### Step 1: Collecting environment info... #################### 2024-10-14 22:40:29.360 | INFO | main:main:457 - Saving config into: ./checkpoints/config.yml 2024-10-14 22:40:29.377 | INFO | sgg_benchmark.utils.logger:logger_step:15 - #################### Step 2: Building model... #################### Overriding model.yaml nc=80 with nc=150 2024-10-14 22:40:30.206 | INFO | sgg_benchmark.data.build:get_dataset_statistics:30 - ---------------------------------------------------------------------------------------------------- 2024-10-14 22:40:30.206 | INFO | sgg_benchmark.data.build:get_dataset_statistics:31 - get dataset statistics... 2024-10-14 22:40:30.206 | INFO | sgg_benchmark.data.build:get_dataset_statistics:42 - Loading data statistics from: ./checkpoints/VG150_train_statistics.cache 2024-10-14 22:40:30.206 | INFO | sgg_benchmark.data.build:get_dataset_statistics:43 - ---------------------------------------------------------------------------------------------------- 2024-10-14 22:40:34.091 | INFO | sgg_benchmark.data.build:get_dataset_statistics:30 - ---------------------------------------------------------------------------------------------------- 2024-10-14 22:40:34.091 | INFO | sgg_benchmark.data.build:get_dataset_statistics:31 - get dataset statistics... 2024-10-14 22:40:34.092 | INFO | sgg_benchmark.data.build:get_dataset_statistics:42 - Loading data statistics from: ./checkpoints/VG150_train_statistics.cache 2024-10-14 22:40:34.092 | INFO | sgg_benchmark.data.build:get_dataset_statistics:43 - ---------------------------------------------------------------------------------------------------- loading word vectors from ./glove.6B.200d.pt 2024-10-14 22:40:39.691 | INFO | sgg_benchmark.utils.logger:logger_step:15 - #################### Step 3: Building optimizer and scheduler... #################### Transferred 439/511 items from pretrained weights 2024-10-14 22:40:39.838 | INFO | sgg_benchmark.utils.logger:logger_step:15 - #################### Step 4: Loading Backbone weights from ./yolov8m_vg150.pt #################### 2024-10-14 22:40:39.839 | INFO | sgg_benchmark.utils.logger:logger_step:15 - #################### Step 5: Building checkpointer #################### 108073 /home/haozhen/mingyuan/SGG-Benchmark/datasets/vg/VG_100K 2024-10-14 22:40:42.501 | INFO | sgg_benchmark.utils.miscellaneous:save_labels:50 - Saving labels mapping into ./checkpoints/labels.json 108073 /home/haozhen/mingyuan/SGG-Benchmark/datasets/vg/VG_100K 2024-10-14 22:40:43.736 | INFO | sgg_benchmark.utils.logger:logger_step:15 - #################### Step 6: Building dataloader #################### 2024-10-14 22:40:43.736 | INFO | main:train:203 - Validate before training 2024-10-14 22:40:43.737 | INFO | sgg_benchmark.engine.inference:inference:263 - Start evaluation on VG150_val dataset(5000 images). 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2501/2501 [02:26<00:00, 17.07it/s] 2024-10-14 22:43:10.251 | INFO | sgg_benchmark.engine.inference:inference:280 - Total run time: 0:02:18 (27.661574461897676 ms / img per device, on 1 devices) 2024-10-14 22:43:10.252 | INFO | sgg_benchmark.engine.inference:inference:289 - Average latency per image: 27.661574461897676ms 2024-10-14 22:43:10.252 | INFO | sgg_benchmark.engine.inference:inference:292 - Standard deviation of latency: 24.626527727053297ms 2024-10-14 22:43:11.539 | INFO | sgg_benchmark.data.datasets.evaluation.vg.vg_eval:do_vg_evaluation:70 - Dataset does not have informative_rels, skipping informative evaluation for dataset: VG150 creating index... index created! Loading and preparing results... Converting ndarray to lists... (0, 7) Traceback (most recent call last): File "/home/haozhen/mingyuan/SGG-Benchmark/tools/relation_train_net.py", line 491, in
main()
File "/home/haozhen/mingyuan/SGG-Benchmark/tools/relation_train_net.py", line 470, in main
model, best_checkpoint = train(
^^^^^^
File "/home/haozhen/mingyuan/SGG-Benchmark/tools/relation_train_net.py", line 204, in train
run_val(cfg, model, val_data_loaders, args['distributed'], logger, device=device)
File "/home/haozhen/mingyuan/SGG-Benchmark/tools/relation_train_net.py", line 323, in run_val
dataset_result = inference(
^^^^^^^^^^
File "/home/haozhen/mingyuan/SGG-Benchmark/sgg_benchmark/engine/inference.py", line 335, in inference
return evaluate(cfg=cfg,
^^^^^^^^^^^^^^^^^
File "/home/haozhen/mingyuan/SGG-Benchmark/sgg_benchmark/data/datasets/evaluation/init.py", line 27, in evaluate
return vg_evaluation(**args)
^^^^^^^^^^^^^^^^^^^^^
File "/home/haozhen/mingyuan/SGG-Benchmark/sgg_benchmark/data/datasets/evaluation/vg/init.py", line 15, in vg_evaluation
return do_vg_evaluation(
^^^^^^^^^^^^^^^^^
File "/home/haozhen/mingyuan/SGG-Benchmark/sgg_benchmark/data/datasets/evaluation/vg/vg_eval.py", line 119, in do_vg_evaluation
res = fauxcoco.loadRes(cocolike_predictions)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/conda/env/scene_graph_benchmark/lib/python3.11/site-packages/pycocotools/coco.py", line 329, in loadRes
if 'caption' in anns[0]: