analyze_logs.py: KeyError: '..../20211102_121731.log.json does not contain metric bbox_mAP'

[pfuerste]

Hey, I am trying to plot my models training progress, but it does not seem to log bbox_mAP. I called python /home/fuerste/mmdetection/tools/analysis_tools/analyze_logs.py plot_curve /home/fuerste/thesis_root/model_compare/models/ddetr/work_dirs/deformable_detr_r50_16x2_50e_coco_cct20/20211102_121731.log.json, but it throws KeyError: '/home/fuerste/thesis_root/model_compare/models/ddetr/work_dirs/deformable_detr_r50_16x2_50e_coco_cct20/20211102_121731.log.json does not contain metric bbox_mAP'

I also trained a faster_rcnn, which I am able to plot the training logs of. I changed the number of classes for both runs to 16. Below is my config file for Deformable-DETR.

_base_ = '/home/fuerste/mmdetection/configs/deformable_detr/deformable_detr_r50_16x2_50e_coco.py'

model = dict(
    bbox_head=dict(
        type='DeformableDETRHead',
        num_classes=16))

dataset_type = 'COCODataset'
classes = (
    'bobcat',
    'opossum',
    'empty',
    'coyote',
    'raccoon',
    'bird',
    'dog',
    'cat',
    'squirrel',
    'rabbit',
    'skunk',
    'rodent',
    'badger',
    'deer',
    'car',
    'fox',
)

data = dict(
    samples_per_gpu=2,
    workers_per_gpu=1,
    train=dict(
        img_prefix='/home/datasets/camera-traps/Caltech-Camera-Traps/CCT20-benchmark/eccv_18_all_images_sm',
        classes=classes,
        filter_empty_gt=False,
        ann_file='/home/fuerste/thesis_root/data/cct20/annotations/train_annotations.json'),
    val=dict(
        img_prefix='/home/datasets/camera-traps/Caltech-Camera-Traps/CCT20-benchmark/eccv_18_all_images_sm',
        classes=classes,
        filter_empty_gt=False,
        ann_file='/home/fuerste/thesis_root/data/cct20/annotations/cis_val_annotations.json'),
    test=dict(
        img_prefix='/home/datasets/camera-traps/Caltech-Camera-Traps/CCT20-benchmark/eccv_18_all_images_sm',
        classes=classes,
        filter_empty_gt=False,
        ann_file='/home/fuerste/thesis_root/data/cct20/annotations/cis_test_annotations.json'))

# runner = dict(type='EpochBasedRunner', max_epochs=50)
log_config = dict(
    interval=10,
    hooks=[
        dict(type='TextLoggerHook'),
        dict(
            type='WandbLoggerHook',
            init_kwargs=dict(
                project='mmdetection_debugging',
                name='ogset_ogsize_2_1_logbug'
            ))])

load_from = '/home/fuerste/mmdetection/checkpoints/deformable_detr_r50_16x2_50e_coco_20210419_220030-a12b9512.pth'
workflow = [('train', 1), ('val', 1)]
runner = dict(type='EpochBasedRunner', max_epochs=5)
evaluation = dict(interval=1, metric='bbox')

Loading and using the model does work. There is no bbox_mAP in either the DDETR nor the faster-rcnn log-files, but plotting bbox_mAP works for faster-rcnn. Could someone help me visualize the training process?

[hhaAndroid]

@pfuerste Please check if the bbox_mAP key exists.

[pfuerste]

It does not. print(log_dict[epochs[0]].keys()) on line 53 in analyze_logs.py prints many losses, but no bbox_mAP:

dict_keys(['mode', 'iter', 'lr', 'memory', 'data_time', 'loss_cls', 'loss_bbox', 'loss_iou', 'd0.loss_cls', 'd0.loss_bbox', 'd0.loss_iou', 'd1.loss_cls', 'd1.loss_bbox', 'd1.loss_iou', 'd2.loss_cls', 'd2.loss_bbox', 'd2.loss_iou', 'd3.loss_cls', 'd3.loss_bbox', 'd3.loss_iou', 'd4.loss_cls', 'd4.loss_bbox', 'd4.loss_iou', 'loss', 'grad_norm', 'time'])

Exactly the same as in the .jog.json.

[pfuerste]

Nevermind, I had no validation in my DDETR workflow. bbox_mAP does only get logged during validation, not training as it seems from the faster-rcnn logs.

Will test and then close.