Maelic
commented
2 months ago Hi @PengPaulWang,
Unfortunately, I made several changes in the architecture since I released those weights and they may not work anymore. I am planning on re-training a VG150 checkpoint with the current architecture but this is not my priority right now. I would recommend you to re-train the model from scratch instead of trying to use my old checkpoint, you could still use the yolov8 checkpoint for object detection and then train a model for SGDet using the current implementation.
Best
Hi,
I tried to evaluate YOLO8vm for SGDet, using
!source activate sgg_benchmark && CUDA_VISIBLE_DEVICES=0 torchrun --master_port 10027 --nproc_per_node=1 tools/relation_test_net.py --config-file "./checkpoints/VG150-penet-yolov8m/config.yml" MODEL.ROI_RELATION_HEAD.USE_GT_BOX False MODEL.ROI_RELATION_HEAD.USE_GT_OBJECT_LABEL False MODEL.ROI_RELATION_HEAD.PREDICTOR CausalAnalysisPredictor MODEL.ROI_RELATION_HEAD.CAUSAL.EFFECT_TYPE TDE MODEL.ROI_RELATION_HEAD.CAUSAL.FUSION_TYPE sum MODEL.ROI_RELATION_HEAD.CAUSAL.CONTEXT_LAYER motifs TEST.IMS_PER_BATCH 1 DTYPE "float16" GLOVE_DIR ./glove_yolo8m MODEL.PRETRAINED_DETECTOR_CKPT ./checkpoints/VG150-penet-yolov8m OUTPUT_DIR ./checkpoints/VG150-penet-yolov8m TEST.CUSTUM_EVAL True TEST.CUSTUM_PATH ./checkpoints/pw_custom_images DETECTED_SGG_DIR ./checkpoints/pw_sgdet_custom_images_yolo_8m
and I got the following issues. It seems there is a mismatch of YOLO versions? Could you please help with this?
2024-08-22 10:04:11.515 | INFO | sgg_benchmark.utils.logger:setup_logger:31 - Using loguru logger with level: INFO 2024-08-22 10:04:11.516 | INFO | main:main:51 - Using 1 GPUs 2024-08-22 10:04:11.516 | INFO | sgg_benchmark.utils.logger:logger_step:15 - #################### Step 1: Collecting environment info... #################### Overriding model.yaml nc=80 with nc=150 2024-08-22 10:04:12.204 | INFO | sgg_benchmark.data.build:get_dataset_statistics:30 - ---------------------------------------------------------------------------------------------------- 2024-08-22 10:04:12.204 | INFO | sgg_benchmark.data.build:get_dataset_statistics:31 - get dataset statistics... <class 'maskrcnn_benchmark.config.paths_catalog.DatasetCatalog'> ('VG150_train',) 2024-08-22 10:04:12.207 | INFO | sgg_benchmark.data.build:get_dataset_statistics:45 - Loading data statistics from: ./checkpoints/VG150-penet-yolov8m/VG150_train_statistics.cache 2024-08-22 10:04:12.208 | INFO | sgg_benchmark.data.build:get_dataset_statistics:46 - ---------------------------------------------------------------------------------------------------- loading word vectors from ./glove_yolo8m/glove.6B.200d.pt DATALOADER: ASPECT_RATIO_GROUPING: True NUM_WORKERS: 4 SIZE_DIVISIBILITY: 32 DATASETS: NAME: TEST: ('VG150_test',) TO_TEST: TRAIN: ('VG150_train',) VAL: ('VG150_val',) DETECTED_SGG_DIR: ./checkpoints/pw_sgdet_custom_images_yolo_8m DTYPE: float16 GLOVE_DIR: ./glove_yolo8m INPUT: BRIGHTNESS: 0.0 CONTRAST: 0.0 FLIP_PROB_TRAIN: 0.5 HUE: 0.0 MAX_SIZE_TEST: 640 MAX_SIZE_TRAIN: 640 MIN_SIZE_TEST: 640 MIN_SIZE_TRAIN: 640 PADDING: True PIXEL_MEAN: [102.9801, 115.9465, 122.7717] PIXEL_STD: [1.0, 1.0, 1.0] SATURATION: 0.0 TO_BGR255: True VERTICAL_FLIP_PROB_TRAIN: 0.0 METRIC_TO_TRACK: mR MODEL: ATTRIBUTE_ON: False BACKBONE: EXTRA_CONFIG: FREEZE: True FREEZE_CONV_BODY_AT: 2 NMS_THRESH: 0.001 TYPE: yolov8 BOX_HEAD: False CLS_AGNOSTIC_BBOX_REG: False DEVICE: cuda FBNET: ARCH: default ARCH_DEF: BN_TYPE: bn DET_HEAD_BLOCKS: [] DET_HEAD_LAST_SCALE: 1.0 DET_HEAD_STRIDE: 0 DW_CONV_SKIP_BN: True DW_CONV_SKIP_RELU: True KPTS_HEAD_BLOCKS: [] KPTS_HEAD_LAST_SCALE: 0.0 KPTS_HEAD_STRIDE: 0 MASK_HEAD_BLOCKS: [] MASK_HEAD_LAST_SCALE: 0.0 MASK_HEAD_STRIDE: 0 RPN_BN_TYPE: RPN_HEAD_BLOCKS: 0 SCALE_FACTOR: 1.0 WIDTH_DIVISOR: 1 FLIP_AUG: False FPN: USE_GN: False USE_RELU: False GROUP_NORM: DIM_PER_GP: -1 EPSILON: 1e-05 NUM_GROUPS: 32 MASK_ON: False META_ARCHITECTURE: GeneralizedYOLO PRETRAINED_DETECTOR_CKPT: ./checkpoints/VG150-penet-yolov8m RELATION_ON: True RESNETS: BACKBONE_OUT_CHANNELS: 1024 DEFORMABLE_GROUPS: 1 NUM_GROUPS: 1 RES2_OUT_CHANNELS: 256 RES5_DILATION: 1 STAGE_WITH_DCN: (False, False, False, False) STEM_FUNC: StemWithFixedBatchNorm STEM_OUT_CHANNELS: 64 STRIDE_IN_1X1: True TRANS_FUNC: BottleneckWithFixedBatchNorm WIDTH_PER_GROUP: 64 WITH_MODULATED_DCN: False ROI_ATTRIBUTE_HEAD: ATTRIBUTE_BGFG_RATIO: 3 ATTRIBUTE_BGFG_SAMPLE: True ATTRIBUTE_LOSS_WEIGHT: 0.1 FEATURE_EXTRACTOR: FPN2MLPFeatureExtractor MAX_ATTRIBUTES: 10 NUM_ATTRIBUTES: 201 POS_WEIGHT: 5.0 PREDICTOR: FPNPredictor SHARE_BOX_FEATURE_EXTRACTOR: True USE_BINARY_LOSS: True ROI_BOX_HEAD: CONV_HEAD_DIM: 256 DILATION: 1 FEATURE_EXTRACTOR: YOLOV8FeatureExtractor MLP_HEAD_DIM: 2048 NUM_CLASSES: 151 NUM_STACKED_CONVS: 4 POOLER_RESOLUTION: 7 POOLER_SAMPLING_RATIO: 2 POOLER_SCALES: (0.0625,) PREDICTOR: FastRCNNPredictor USE_GN: False ROI_HEADS: BATCH_SIZE_PER_IMAGE: 256 BBOX_REG_WEIGHTS: (10.0, 10.0, 5.0, 5.0) BG_IOU_THRESHOLD: 0.1 DETECTIONS_PER_IMG: 80 FG_IOU_THRESHOLD: 0.3 NMS: 0.2 NMS_FILTER_DUPLICATES: True POSITIVE_FRACTION: 0.5 POST_NMS_PER_CLS_TOPN: 300 SCORE_THRESH: 0.01 ROI_MASK_HEAD: CONV_LAYERS: (256, 256, 256, 256) DILATION: 1 FEATURE_EXTRACTOR: ResNet50Conv5ROIFeatureExtractor MLP_HEAD_DIM: 1024 POOLER_RESOLUTION: 14 POOLER_SAMPLING_RATIO: 0 POOLER_SCALES: (0.0625,) POSTPROCESS_MASKS: False POSTPROCESS_MASKS_THRESHOLD: 0.5 PREDICTOR: MaskRCNNC4Predictor RESOLUTION: 14 SHARE_BOX_FEATURE_EXTRACTOR: True USE_GN: False ROI_RELATION_HEAD: ADD_GTBOX_TO_PROPOSAL_IN_TRAIN: True BATCH_SIZE_PER_IMAGE: 256 CAUSAL: CONTEXT_LAYER: motifs EFFECT_ANALYSIS: True EFFECT_TYPE: TDE FUSION_TYPE: sum SEPARATE_SPATIAL: False SPATIAL_FOR_VISION: True CLASSIFIER: linear CONTEXT_DROPOUT_RATE: 0.2 CONTEXT_HIDDEN_DIM: 512 CONTEXT_OBJ_LAYER: 1 CONTEXT_POOLING_DIM: 2048 CONTEXT_REL_LAYER: 1 EMBED_DIM: 200 FEATURE_EXTRACTOR: RelationFeatureExtractor LABEL_SMOOTHING_LOSS: False LOGIT_ADJUSTMENT: False LOGIT_ADJUSTMENT_TAU: 0.3 LOSS: CrossEntropyLoss MLP_HEAD_DIM: 2048 NUM_CLASSES: 51 NUM_SAMPLE_PER_GT_REL: 4 POOLING_ALL_LEVELS: True POSITIVE_FRACTION: 0.25 PREDICTOR: CausalAnalysisPredictor PREDICT_USE_VISION: True REQUIRE_BOX_OVERLAP: False SQUAT_MODULE: BETA: 0.7 NUM_DECODER: 3 PRETRAIN_MASK: False PRETRAIN_MASK_EPOCH: 1 PRE_NORM: False RHO: 0.35 TEXTUAL_FEATURES_ONLY: False TRANSFORMER: DROPOUT_RATE: 0.1 INNER_DIM: 2048 KEY_DIM: 64 NUM_HEAD: 8 OBJ_LAYER: 4 REL_LAYER: 2 VAL_DIM: 64 USE_FREQUENCY_BIAS: True USE_GT_BOX: False USE_GT_OBJECT_LABEL: False USE_SPATIAL_FEATURES: True USE_UNION_FEATURES: True VISUAL_FEATURES_ONLY: False RPN: ANCHOR_SIZES: (32, 64, 128, 256, 512) ANCHOR_STRIDE: (16,) ASPECT_RATIOS: (0.5, 1.0, 2.0) BATCH_SIZE_PER_IMAGE: 256 BG_IOU_THRESHOLD: 0.3 FG_IOU_THRESHOLD: 0.7 FPN_POST_NMS_PER_BATCH: True FPN_POST_NMS_TOP_N_TEST: 2000 FPN_POST_NMS_TOP_N_TRAIN: 2000 MIN_SIZE: 0 POSITIVE_FRACTION: 0.5 POST_NMS_TOP_N_TEST: 1000 POST_NMS_TOP_N_TRAIN: 2000 PRE_NMS_TOP_N_TEST: 6000 PRE_NMS_TOP_N_TRAIN: 12000 RPN_HEAD: SingleConvRPNHead RPN_MID_CHANNEL: 512 STRADDLE_THRESH: 0 USE_FPN: False RPN_ONLY: False TEXT_EMBEDDING: glove.6B VGG: VGG16_OUT_CHANNELS: 512 WEIGHT: YOLO: IMG_SIZE: 640 OUT_CHANNELS: 192 SIZE: yolov8m WEIGHTS: /content/gdrive/MyDrive/SGG-Benchmark/checkpoints/pretrained_yolo8m_vg150/yolov8m_vg150.pt OUTPUT_DIR: ./checkpoints/VG150-penet-yolov8m PATHS_CATALOG: /content/gdrive/MyDrive/SGG-Benchmark/sgg_benchmark/config/paths_catalog.py PATHS_DATA: /content/gdrive/MyDrive/SGG-Benchmark/datasets/vg/ SEED: 42 SOLVER: BASE_LR: 0.01 BIAS_LR_FACTOR: 1 CHECKPOINT_PERIOD: 2000 CLIP_NORM: 5.0 GAMMA: 0.1 GRAD_NORM_CLIP: 5.0 IMS_PER_BATCH: 16 MAX_EPOCH: 20 MAX_ITER: 40000 MOMENTUM: 0.9 OPTIMIZER: SGD PRE_VAL: False PRINT_GRAD_FREQ: 4000 SCHEDULE: COOLDOWN: 0 FACTOR: 0.1 MAX_DECAY_STEP: 3 PATIENCE: 2 THRESHOLD: 0.001 TYPE: WarmupReduceLROnPlateau STEPS: (10000, 16000) TO_VAL: True UPDATE_SCHEDULE_DURING_LOAD: False VAL_PERIOD: 2000 WARMUP_FACTOR: 0.1 WARMUP_ITERS: 500 WARMUP_METHOD: linear WEIGHT_DECAY: 0.0001 WEIGHT_DECAY_BIAS: 0.0 TEST: ALLOW_LOAD_FROM_CACHE: False BBOX_AUG: ENABLED: False H_FLIP: False MAX_SIZE: 4000 SCALES: () SCALE_H_FLIP: False CUSTUM_EVAL: True CUSTUM_PATH: ./checkpoints/pw_custom_images DETECTIONS_PER_IMG: 100 EXPECTED_RESULTS: [] EXPECTED_RESULTS_SIGMA_TOL: 4 IMS_PER_BATCH: 1 INFORMATIVE: True RELATION: IOU_THRESHOLD: 0.5 LATER_NMS_PREDICTION_THRES: 0.5 MULTIPLE_PREDS: False REQUIRE_OVERLAP: True SYNC_GATHER: True SAVE_PROPOSALS: False TOP_K: 100 VERBOSE: INFO 2024-08-22 10:04:16.764 | INFO | main:main:73 - Loading last checkpoint from /content/gdrive/MyDrive/SGG-Benchmark/checkpoints/VG150-penet-yolov8m/best_model_epoch_2.pth... 2024-08-22 10:04:16.764 | INFO | sgg_benchmark.utils.checkpoint:load:65 - Loading checkpoint from /content/gdrive/MyDrive/SGG-Benchmark/checkpoints/VG150-penet-yolov8m/best_model_epoch_2.pth Traceback (most recent call last): File "/content/gdrive/MyDrive/SGG-Benchmark/tools/relation_test_net.py", line 180, in
main()
File "/content/gdrive/MyDrive/SGG-Benchmark/tools/relation_testnet.py", line 74, in main
= checkpointer.load(last_check)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/content/gdrive/MyDrive/SGG-Benchmark/sgg_benchmark/utils/checkpoint.py", line 67, in load
self._load_model(checkpoint, load_mapping, verbose)
File "/content/gdrive/MyDrive/SGG-Benchmark/sgg_benchmark/utils/checkpoint.py", line 107, in _load_model
load_state_dict(self.model, checkpoint.pop("model"), load_mapping, verbose)
File "/content/gdrive/MyDrive/SGG-Benchmark/sgg_benchmark/utils/model_serialization.py", line 94, in load_state_dict
model.load_state_dict(model_state_dict)
File "/usr/local/envs/sgg_benchmark/lib/python3.11/site-packages/torch/nn/modules/module.py", line 2153, in load_state_dict
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for GeneralizedYOLO:
size mismatch for roi_heads.relation.predictor.post_emb.weight: copying a param with shape torch.Size([4096, 2048]) from checkpoint, the shape in current model is torch.Size([1024, 512]).
size mismatch for roi_heads.relation.predictor.post_emb.bias: copying a param with shape torch.Size([4096]) from checkpoint, the shape in current model is torch.Size([1024]).
size mismatch for roi_heads.relation.union_feature_extractor.feature_extractor.pooler.reduce_channel.0.weight: copying a param with shape torch.Size([192, 576, 3, 3]) from checkpoint, the shape in current model is torch.Size([192, 192, 3, 3]).
[2024-08-22 10:04:21,292] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 29416) of binary: /usr/local/envs/sgg_benchmark/bin/python
Traceback (most recent call last):
File "/usr/local/envs/sgg_benchmark/bin/torchrun", line 33, in
sys.exit(load_entry_point('torch==2.2.1', 'console_scripts', 'torchrun')())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/envs/sgg_benchmark/lib/python3.11/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 347, in wrapper
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/usr/local/envs/sgg_benchmark/lib/python3.11/site-packages/torch/distributed/run.py", line 812, in main
run(args)
File "/usr/local/envs/sgg_benchmark/lib/python3.11/site-packages/torch/distributed/run.py", line 803, in run
elastic_launch(
File "/usr/local/envs/sgg_benchmark/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 135, in call
return launch_agent(self._config, self._entrypoint, list(args))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/envs/sgg_benchmark/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 268, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
tools/relation_test_net.py FAILED
Failures: