search

open-mmlab / mmpose

OpenMMLab Pose Estimation Toolbox and Benchmark.
https://mmpose.readthedocs.io/en/latest/
Apache License 2.0
5.86k stars 1.25k forks source link

The evaluation index mAP is 0 #2255

Closed hsp2454 closed 1 year ago

hsp2454 commented 1 year ago

Sorry, because I didn't describe my problem clearly in( https://github.com/open-mmlab/mmpose/issues/2248).. I am using mmpose0.x now, not 1.x. My question is how to use mAP as an evaluation indicator in mmpose0.x. The results of my evaluation are all 0. The following are some of my configuration files.

First, the following is my config file:

Config:
checkpoint_config = dict(interval=2)
log_config = dict(
    interval=1,
    hooks=[dict(type='TextLoggerHook'),        
           dict(type='TensorboardLoggerHook')])
log_level = 'INFO'
load_from = None
resume_from = None
dist_params = dict(backend='nccl')
workflow = [('train', 1), ('val', 1)]
opencv_num_threads = 0
mp_start_method = 'fork'
dataset_info = dict(
    dataset_name='PigAnimal',
    paper_info=dict(
        author=
        'Lin, Tsung-Yi and Maire, Michael and Belongie, Serge and Hays, James and Perona, Pietro and Ramanan, Deva and Doll{\'a}r, Piotr and Zitnick, C Lawrence',        
        title='Microsoft coco: Common objects in context',
        container='European conference on computer vision',
        year='2014',
        homepage='http://cocodataset.org/'),
    keypoint_info=dict({
        0:
        dict(name='1', id=0, color=[51, 153, 255], type='upper', swap=''),
        1:
        dict(name='2', id=1, color=[51, 153, 255], type='upper', swap=''),
        2:
        dict(name='3', id=2, color=[51, 153, 255], type='upper', swap=''),
        3:
        dict(name='4', id=3, color=[51, 153, 255], type='upper', swap=''),
        4:
        dict(name='5', id=4, color=[51, 153, 255], type='upper', swap=''),
        5:
        dict(name='6', id=5, color=[0, 255, 0], type='upper', swap=''),
        6:
        dict(name='7', id=6, color=[255, 128, 0], type='upper', swap=''),
        7:
        dict(name='8', id=7, color=[0, 255, 0], type='upper', swap='9'),
        8:
        dict(name='9', id=8, color=[255, 128, 0], type='upper', swap=''),
        9:
        dict(name='10', id=9, color=[0, 255, 0], type='upper', swap='16'),
        10:
        dict(name='11', id=10, color=[255, 128, 0], type='upper', swap='17'),
        11:
        dict(name='12', id=11, color=[0, 255, 0], type='lower', swap='18'),
        12:
        dict(name='13', id=12, color=[255, 128, 0], type='lower', swap='19'),
        13:
        dict(name='14', id=13, color=[0, 255, 0], type='lower', swap='20'),
        14:
        dict(name='15', id=14, color=[255, 128, 0], type='lower', swap='21'),
        15:
        dict(name='16', id=15, color=[0, 255, 0], type='lower', swap=''),
        16:
        dict(name='17', id=16, color=[255, 128, 0], type='lower', swap=''),
        17:
        dict(name='18', id=17, color=[0, 255, 0], type='lower', swap=''),
        18:
        dict(name='19', id=18, color=[0, 255, 0], type='lower', swap=''),
        19:
        dict(name='20', id=19, color=[0, 255, 0], type='lower', swap=''),
        20:
        dict(name='21', id=20, color=[0, 255, 0], type='lower', swap='')
    }),
    skeleton_info=dict({
        0: dict(link=('1', '2'), id=0, color=[0, 255, 0]),
        1: dict(link=('2', '3'), id=1, color=[0, 255, 0]),
        2: dict(link=('3', '4'), id=2, color=[255, 128, 0]),
        3: dict(link=('4', '5'), id=3, color=[255, 128, 0]),
        4: dict(link=('5', '6'), id=4, color=[51, 153, 255]),
        5: dict(link=('6', '7'), id=5, color=[51, 153, 255]),
        6: dict(link=('7', '8'), id=6, color=[51, 153, 255]),
        7: dict(link=('8', '13'), id=7, color=[51, 153, 255]),
        8: dict(link=('9', '10'), id=8, color=[0, 255, 0]),
        9: dict(link=('9', '16'), id=9, color=[255, 128, 0]),
        10: dict(link=('9', '13'), id=10, color=[0, 255, 0]),
        11: dict(link=('9', '19'), id=11, color=[255, 128, 0]),
        12: dict(link=('10', '11'), id=12, color=[51, 153, 255]),
        13: dict(link=('11', '12'), id=13, color=[51, 153, 255]),
        14: dict(link=('13', '14'), id=14, color=[51, 153, 255]),
        15: dict(link=('14', '15'), id=15, color=[51, 153, 255]),
        16: dict(link=('16', '17'), id=16, color=[51, 153, 255]),
        17: dict(link=('17', '18'), id=17, color=[51, 153, 255]),
        18: dict(link=('19', '20'), id=18, color=[51, 153, 255]),
        19: dict(link=('20', '21'), id=19, color=[51, 123, 255]),
        20: dict(link=('8', '19'), id=20, color=[51, 123, 255]),
        21: dict(link=('2', '10'), id=21, color=[51, 123, 255]),
        22: dict(link=('2', '16'), id=22, color=[51, 123, 255])
    }),
    joint_weights=[
        1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.2, 1.2, 1.5, 1.5, 1.0, 1.0, 1.2,
        1.2, 1.5, 1.5, 1.5, 1.5, 1.5, 1.5
    ],
    sigmas=[
        0.26, 0.25, 0.25, 0.35, 0.35, 0.79, 0.79, 0.72, 0.72, 0.9, 0.9, 0.9,
        1.07, 0.87, 0.87, 0.89, 0.89, 0.9, 0.9, 0.9, 0.9
    ])
evaluation = dict(interval=1, metric='mAP', save_best='AP')
optimizer = dict(type='Adam', lr=0.0005)
optimizer_config = dict(grad_clip=None)
lr_config = dict(
    policy='step',
    warmup='linear',
    warmup_iters=100,
    warmup_ratio=0.001,
    step=[50, 90])
total_epochs = 210
channel_cfg = dict(
    num_output_channels=21,
    dataset_joints=21,
    dataset_channel=[[
        0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19,
        20
    ]],
    inference_channel=[
        0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19,
        20
    ])
model = dict(
    type='TopDown',
    pretrained=
    'https://download.openmmlab.com/mmpose/pretrain_models/hrnet_w48-8ef0771d.pth',
    backbone=dict(
        type='HRNet',
        in_channels=3,
        extra=dict(
            stage1=dict(
                num_modules=1,
                num_branches=1,
                block='BOTTLENECK',
                num_blocks=(4, ),
                num_channels=(64, )),
            stage2=dict(
                num_modules=1,
                num_branches=2,
                block='BASIC',
                num_blocks=(4, 4),
                num_channels=(48, 96)),
            stage3=dict(
                num_modules=4,
                num_branches=3,
                block='BASIC',
                num_blocks=(4, 4, 4),
                num_channels=(48, 96, 192)),
            stage4=dict(
                num_modules=3,
                num_branches=4,
                block='BASIC',
                num_blocks=(4, 4, 4, 4),
                num_channels=(48, 96, 192, 384)))),
    keypoint_head=dict(
        type='TopdownHeatmapSimpleHead',
        in_channels=48,
        out_channels=21,
        num_deconv_layers=0,
        extra=dict(final_conv_kernel=1),
        loss_keypoint=dict(type='JointsMSELoss', use_target_weight=True)),
    train_cfg=dict(),
    test_cfg=dict(
        flip_test=True,
        post_process='default',
        shift_heatmap=True,
        modulate_kernel=11))
data_cfg = dict(
    image_size=[256, 256],
    heatmap_size=[64, 64],
    num_output_channels=21,
    num_joints=21,
    dataset_channel=[[
        0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19,
        20
    ]],
    inference_channel=[
        0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19,
        20
    ],
    soft_nms=False,
    nms_thr=0.9,
    oks_thr=0.9,
    vis_thr=0.4,
    use_gt_bbox=True,
    det_bbox_thr=0.0,
    bbox_file=
    'D:\Code\mmpose2\data\pigBehavior/NewData\coco\instances_val2017.json')
train_pipeline = [
    dict(type='LoadImageFromFile'),
    dict(type='TopDownGetBboxCenterScale', padding=1.25),
    dict(type='TopDownRandomShiftBboxCenter', shift_factor=0.16, prob=0.5),
    dict(type='TopDownRandomFlip', flip_prob=0.7),
    dict(
        type='TopDownHalfBodyTransform',
        num_joints_half_body=8,
        prob_half_body=0.7),
    dict(
        type='TopDownGetRandomScaleRotation', rot_factor=40, scale_factor=0.5),
    dict(type='TopDownAffine'),
    dict(type='ToTensor'),
    dict(
        type='NormalizeTensor',
        mean=[0.485, 0.456, 0.406],
        std=[0.229, 0.224, 0.225]),
    dict(type='TopDownGenerateTarget', sigma=2),
    dict(
        type='Collect',
        keys=['img', 'target', 'target_weight'],
        meta_keys=[
            'image_file', 'joints_3d', 'joints_3d_visible', 'center', 'scale',
            'rotation', 'bbox_score', 'flip_pairs'
        ])
]
val_pipeline = [
    dict(type='LoadImageFromFile'),
    dict(type='TopDownGetBboxCenterScale', padding=1.25),
    dict(type='TopDownAffine'),
    dict(type='ToTensor'),
    dict(
        type='NormalizeTensor',
        mean=[0.485, 0.456, 0.406],
        std=[0.229, 0.224, 0.225]),
    dict(
        type='Collect',
        keys=['img'],
        meta_keys=[
            'image_file', 'center', 'scale', 'rotation', 'bbox_score',
            'flip_pairs'
        ])
]
test_pipeline = [
    dict(type='LoadImageFromFile'),
    dict(type='TopDownGetBboxCenterScale', padding=1.25),
    dict(type='TopDownAffine'),
    dict(type='ToTensor'),
    dict(
        type='NormalizeTensor',
        mean=[0.485, 0.456, 0.406],
        std=[0.229, 0.224, 0.225]),
    dict(
        type='Collect',
        keys=['img'],
        meta_keys=[
            'image_file', 'center', 'scale', 'rotation', 'bbox_score',
            'flip_pairs'
        ])
]
data_root = 'D:\Code\mmpose2\data\pigBehavior\coco'
data = dict(
    samples_per_gpu=16,
    workers_per_gpu=4,
    val_dataloader=dict(samples_per_gpu=16),
    test_dataloader=dict(samples_per_gpu=16),
    train=dict(
        type='PigAnimal',
        ann_file=
        'D:\Code\mmpose2\data\pigBehavior\coco/annotations\keypoints_train.json',
        img_prefix='D:\Code\mmpose2\data\pigBehavior\coco/train/',
        data_cfg=dict(
            image_size=[256, 256],
            heatmap_size=[64, 64],
            num_output_channels=21,
            num_joints=21,
            dataset_channel=[[
                0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17,
                18, 19, 20
            ]],
            inference_channel=[
                0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17,
                18, 19, 20
            ],
            soft_nms=False,
            nms_thr=0.9,
            oks_thr=0.9,
            vis_thr=0.4,
            use_gt_bbox=True,
            det_bbox_thr=0.0,
            bbox_file=
            'D:\Code\mmpose2\data\pigBehavior/NewData\coco\instances_val2017.json'
        ),
        pipeline=[
            dict(type='LoadImageFromFile'),
            dict(type='TopDownGetBboxCenterScale', padding=1.25),
            dict(
                type='TopDownRandomShiftBboxCenter',
                shift_factor=0.16,
                prob=0.5),
            dict(type='TopDownRandomFlip', flip_prob=0.7),
            dict(
                type='TopDownHalfBodyTransform',
                num_joints_half_body=8,
                prob_half_body=0.7),
            dict(
                type='TopDownGetRandomScaleRotation',
                rot_factor=40,
                scale_factor=0.5),
            dict(type='TopDownAffine'),
            dict(type='ToTensor'),
            dict(
                type='NormalizeTensor',
                mean=[0.485, 0.456, 0.406],
                std=[0.229, 0.224, 0.225]),
            dict(type='TopDownGenerateTarget', sigma=2),
            dict(
                type='Collect',
                keys=['img', 'target', 'target_weight'],
                meta_keys=[
                    'image_file', 'joints_3d', 'joints_3d_visible', 'center',
                    'scale', 'rotation', 'bbox_score', 'flip_pairs'
                ])
        ],
        dataset_info=dict(
            dataset_name='PigAnimal',
            paper_info=dict(
                author=
                'Lin, Tsung-Yi and Maire, Michael and Belongie, Serge and Hays, James and Perona, Pietro and Ramanan, Deva and Doll{\'a}r, Piotr and Zitnick, C Lawrence',                title='Microsoft coco: Common objects in context',
                container='European conference on computer vision',
                year='2014',
                homepage='http://cocodataset.org/'),
            keypoint_info=dict({
                0:
                dict(
                    name='1',
                    id=0,
                    color=[51, 153, 255],
                    type='upper',
                    swap=''),
                1:
                dict(
                    name='2',
                    id=1,
                    color=[51, 153, 255],
                    type='upper',
                    swap=''),
                2:
                dict(
                    name='3',
                    id=2,
                    color=[51, 153, 255],
                    type='upper',
                    swap=''),
                3:
                dict(
                    name='4',
                    id=3,
                    color=[51, 153, 255],
                    type='upper',
                    swap=''),
                4:
                dict(
                    name='5',
                    id=4,
                    color=[51, 153, 255],
                    type='upper',
                    swap=''),
                5:
                dict(name='6', id=5, color=[0, 255, 0], type='upper', swap=''),
                6:
                dict(
                    name='7', id=6, color=[255, 128, 0], type='upper',
                    swap=''),
                7:
                dict(
                    name='8', id=7, color=[0, 255, 0], type='upper', swap='9'),
                8:
                dict(
                    name='9', id=8, color=[255, 128, 0], type='upper',
                    swap=''),
                9:
                dict(
                    name='10',
                    id=9,
                    color=[0, 255, 0],
                    type='upper',
                    swap='16'),
                10:
                dict(
                    name='11',
                    id=10,
                    color=[255, 128, 0],
                    type='upper',
                    swap='17'),
                11:
                dict(
                    name='12',
                    id=11,
                    color=[0, 255, 0],
                    type='lower',
                    swap='18'),
                12:
                dict(
                    name='13',
                    id=12,
                    color=[255, 128, 0],
                    type='lower',
                    swap='19'),
                13:
                dict(
                    name='14',
                    id=13,
                    color=[0, 255, 0],
                    type='lower',
                    swap='20'),
                14:
                dict(
                    name='15',
                    id=14,
                    color=[255, 128, 0],
                    type='lower',
                    swap='21'),
                15:
                dict(
                    name='16', id=15, color=[0, 255, 0], type='lower',
                    swap=''),
                16:
                dict(
                    name='17',
                    id=16,
                    color=[255, 128, 0],
                    type='lower',
                    swap=''),
                17:
                dict(
                    name='18', id=17, color=[0, 255, 0], type='lower',
                    swap=''),
                18:
                dict(
                    name='19', id=18, color=[0, 255, 0], type='lower',
                    swap=''),
                19:
                dict(
                    name='20', id=19, color=[0, 255, 0], type='lower',
                    swap=''),
                20:
                dict(
                    name='21', id=20, color=[0, 255, 0], type='lower', swap='')
            }),
            skeleton_info=dict({
                0:
                dict(link=('1', '2'), id=0, color=[0, 255, 0]),
                1:
                dict(link=('2', '3'), id=1, color=[0, 255, 0]),
                2:
                dict(link=('3', '4'), id=2, color=[255, 128, 0]),
                3:
                dict(link=('4', '5'), id=3, color=[255, 128, 0]),
                4:
                dict(link=('5', '6'), id=4, color=[51, 153, 255]),
                5:
                dict(link=('6', '7'), id=5, color=[51, 153, 255]),
                6:
                dict(link=('7', '8'), id=6, color=[51, 153, 255]),
                7:
                dict(link=('8', '13'), id=7, color=[51, 153, 255]),
                8:
                dict(link=('9', '10'), id=8, color=[0, 255, 0]),
                9:
                dict(link=('9', '16'), id=9, color=[255, 128, 0]),
                10:
                dict(link=('9', '13'), id=10, color=[0, 255, 0]),
                11:
                dict(link=('9', '19'), id=11, color=[255, 128, 0]),
                12:
                dict(link=('10', '11'), id=12, color=[51, 153, 255]),
                13:
                dict(link=('11', '12'), id=13, color=[51, 153, 255]),
                14:
                dict(link=('13', '14'), id=14, color=[51, 153, 255]),
                15:
                dict(link=('14', '15'), id=15, color=[51, 153, 255]),
                16:
                dict(link=('16', '17'), id=16, color=[51, 153, 255]),
                17:
                dict(link=('17', '18'), id=17, color=[51, 153, 255]),
                18:
                dict(link=('19', '20'), id=18, color=[51, 153, 255]),
                19:
                dict(link=('20', '21'), id=19, color=[51, 123, 255]),
                20:
                dict(link=('8', '19'), id=20, color=[51, 123, 255]),
                21:
                dict(link=('2', '10'), id=21, color=[51, 123, 255]),
                22:
                dict(link=('2', '16'), id=22, color=[51, 123, 255])
            }),
            joint_weights=[
                1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.2, 1.2, 1.5, 1.5, 1.0,
                1.0, 1.2, 1.2, 1.5, 1.5, 1.5, 1.5, 1.5, 1.5
            ],
            sigmas=[
                0.26, 0.25, 0.25, 0.35, 0.35, 0.79, 0.79, 0.72, 0.72, 0.9, 0.9,
                0.9, 1.07, 0.87, 0.87, 0.89, 0.89, 0.9, 0.9, 0.9, 0.9
            ])),
    val=dict(
        type='PigAnimal',
        ann_file=
        'D:\Code\mmpose2\data\pigBehavior\coco/annotations\keypoints_val.json',
        img_prefix='D:\Code\mmpose2\data\pigBehavior\coco/val/',
        data_cfg=dict(
            image_size=[256, 256],
            heatmap_size=[64, 64],
            num_output_channels=21,
            num_joints=21,
            dataset_channel=[[
                0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17,
                18, 19, 20
            ]],
            inference_channel=[
                0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17,
                18, 19, 20
            ],
            soft_nms=False,
            nms_thr=0.9,
            oks_thr=0.9,
            vis_thr=0.4,
            use_gt_bbox=True,
            det_bbox_thr=0.0,
            bbox_file=
            'D:\Code\mmpose2\data\pigBehavior/NewData\coco\instances_val2017.json'
        ),
        pipeline=[
            dict(type='LoadImageFromFile'),
            dict(type='TopDownGetBboxCenterScale', padding=1.25),
            dict(type='TopDownAffine'),
            dict(type='ToTensor'),
            dict(
                type='NormalizeTensor',
                mean=[0.485, 0.456, 0.406],
                std=[0.229, 0.224, 0.225]),
            dict(
                type='Collect',
                keys=['img'],
                meta_keys=[
                    'image_file', 'center', 'scale', 'rotation', 'bbox_score',
                    'flip_pairs'
                ])
        ],
        dataset_info=dict(
            dataset_name='PigAnimal',
            paper_info=dict(
                author=
                'Lin, Tsung-Yi and Maire, Michael and Belongie, Serge and Hays, James and Perona, Pietro and Ramanan, Deva and Doll{\'a}r, Piotr and Zitnick, C Lawrence',                title='Microsoft coco: Common objects in context',
                container='European conference on computer vision',
                year='2014',
                homepage='http://cocodataset.org/'),
            keypoint_info=dict({
                0:
                dict(
                    name='1',
                    id=0,
                    color=[51, 153, 255],
                    type='upper',
                    swap=''),
                1:
                dict(
                    name='2',
                    id=1,
                    color=[51, 153, 255],
                    type='upper',
                    swap=''),
                2:
                dict(
                    name='3',
                    id=2,
                    color=[51, 153, 255],
                    type='upper',
                    swap=''),
                3:
                dict(
                    name='4',
                    id=3,
                    color=[51, 153, 255],
                    type='upper',
                    swap=''),
                4:
                dict(
                    name='5',
                    id=4,
                    color=[51, 153, 255],
                    type='upper',
                    swap=''),
                5:
                dict(name='6', id=5, color=[0, 255, 0], type='upper', swap=''),
                6:
                dict(
                    name='7', id=6, color=[255, 128, 0], type='upper',
                    swap=''),
                7:
                dict(
                    name='8', id=7, color=[0, 255, 0], type='upper', swap='9'),
                8:
                dict(
                    name='9', id=8, color=[255, 128, 0], type='upper',
                    swap=''),
                9:
                dict(
                    name='10',
                    id=9,
                    color=[0, 255, 0],
                    type='upper',
                    swap='16'),
                10:
                dict(
                    name='11',
                    id=10,
                    color=[255, 128, 0],
                    type='upper',
                    swap='17'),
                11:
                dict(
                    name='12',
                    id=11,
                    color=[0, 255, 0],
                    type='lower',
                    swap='18'),
                12:
                dict(
                    name='13',
                    id=12,
                    color=[255, 128, 0],
                    type='lower',
                    swap='19'),
                13:
                dict(
                    name='14',
                    id=13,
                    color=[0, 255, 0],
                    type='lower',
                    swap='20'),
                14:
                dict(
                    name='15',
                    id=14,
                    color=[255, 128, 0],
                    type='lower',
                    swap='21'),
                15:
                dict(
                    name='16', id=15, color=[0, 255, 0], type='lower',
                    swap=''),
                16:
                dict(
                    name='17',
                    id=16,
                    color=[255, 128, 0],
                    type='lower',
                    swap=''),
                17:
                dict(
                    name='18', id=17, color=[0, 255, 0], type='lower',
                    swap=''),
                18:
                dict(
                    name='19', id=18, color=[0, 255, 0], type='lower',
                    swap=''),
                19:
                dict(
                    name='20', id=19, color=[0, 255, 0], type='lower',
                    swap=''),
                20:
                dict(
                    name='21', id=20, color=[0, 255, 0], type='lower', swap='')
            }),
            skeleton_info=dict({
                0:
                dict(link=('1', '2'), id=0, color=[0, 255, 0]),
                1:
                dict(link=('2', '3'), id=1, color=[0, 255, 0]),
                2:
                dict(link=('3', '4'), id=2, color=[255, 128, 0]),
                3:
                dict(link=('4', '5'), id=3, color=[255, 128, 0]),
                4:
                dict(link=('5', '6'), id=4, color=[51, 153, 255]),
                5:
                dict(link=('6', '7'), id=5, color=[51, 153, 255]),
                6:
                dict(link=('7', '8'), id=6, color=[51, 153, 255]),
                7:
                dict(link=('8', '13'), id=7, color=[51, 153, 255]),
                8:
                dict(link=('9', '10'), id=8, color=[0, 255, 0]),
                9:
                dict(link=('9', '16'), id=9, color=[255, 128, 0]),
                10:
                dict(link=('9', '13'), id=10, color=[0, 255, 0]),
                11:
                dict(link=('9', '19'), id=11, color=[255, 128, 0]),
                12:
                dict(link=('10', '11'), id=12, color=[51, 153, 255]),
                13:
                dict(link=('11', '12'), id=13, color=[51, 153, 255]),
                14:
                dict(link=('13', '14'), id=14, color=[51, 153, 255]),
                15:
                dict(link=('14', '15'), id=15, color=[51, 153, 255]),
                16:
                dict(link=('16', '17'), id=16, color=[51, 153, 255]),
                17:
                dict(link=('17', '18'), id=17, color=[51, 153, 255]),
                18:
                dict(link=('19', '20'), id=18, color=[51, 153, 255]),
                19:
                dict(link=('20', '21'), id=19, color=[51, 123, 255]),
                20:
                dict(link=('8', '19'), id=20, color=[51, 123, 255]),
                21:
                dict(link=('2', '10'), id=21, color=[51, 123, 255]),
                22:
                dict(link=('2', '16'), id=22, color=[51, 123, 255])
            }),
            joint_weights=[
                1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.2, 1.2, 1.5, 1.5, 1.0,
                1.0, 1.2, 1.2, 1.5, 1.5, 1.5, 1.5, 1.5, 1.5
            ],
            sigmas=[
                0.26, 0.25, 0.25, 0.35, 0.35, 0.79, 0.79, 0.72, 0.72, 0.9, 0.9,
                0.9, 1.07, 0.87, 0.87, 0.89, 0.89, 0.9, 0.9, 0.9, 0.9
            ])),
    test=dict(
        type='PigAnimal',
        ann_file=
        'D:\Code\mmpose2\data\pigBehavior\coco/annotations\keypoints_val.json',
        img_prefix='D:\Code\mmpose2\data\pigBehavior\coco/val/',
        data_cfg=dict(
            image_size=[256, 256],
            heatmap_size=[64, 64],
            num_output_channels=21,
            num_joints=21,
            dataset_channel=[[
                0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17,
                18, 19, 20
            ]],
            inference_channel=[
                0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17,
                18, 19, 20
            ],
            soft_nms=False,
            nms_thr=0.9,
            oks_thr=0.9,
            vis_thr=0.4,
            use_gt_bbox=True,
            det_bbox_thr=0.0,
            bbox_file=
            'D:\Code\mmpose2\data\pigBehavior/NewData\coco\instances_val2017.json'
        ),
        pipeline=[
            dict(type='LoadImageFromFile'),
            dict(type='TopDownGetBboxCenterScale', padding=1.25),
            dict(type='TopDownAffine'),
            dict(type='ToTensor'),
            dict(
                type='NormalizeTensor',
                mean=[0.485, 0.456, 0.406],
                std=[0.229, 0.224, 0.225]),
            dict(
                type='Collect',
                keys=['img'],
                meta_keys=[
                    'image_file', 'center', 'scale', 'rotation', 'bbox_score',
                    'flip_pairs'
                ])
        ],
        dataset_info=dict(
            dataset_name='PigAnimal',
            paper_info=dict(
                author=
                'Lin, Tsung-Yi and Maire, Michael and Belongie, Serge and Hays, James and Perona, Pietro and Ramanan, Deva and Doll{\'a}r, Piotr and Zitnick, C Lawrence',                title='Microsoft coco: Common objects in context',
                container='European conference on computer vision',
                year='2014',
                homepage='http://cocodataset.org/'),
            keypoint_info=dict({
                0:
                dict(
                    name='1',
                    id=0,
                    color=[51, 153, 255],
                    type='upper',
                    swap=''),
                1:
                dict(
                    name='2',
                    id=1,
                    color=[51, 153, 255],
                    type='upper',
                    swap=''),
                2:
                dict(
                    name='3',
                    id=2,
                    color=[51, 153, 255],
                    type='upper',
                    swap=''),
                3:
                dict(
                    name='4',
                    id=3,
                    color=[51, 153, 255],
                    type='upper',
                    swap=''),
                4:
                dict(
                    name='5',
                    id=4,
                    color=[51, 153, 255],
                    type='upper',
                    swap=''),
                5:
                dict(name='6', id=5, color=[0, 255, 0], type='upper', swap=''),
                6:
                dict(
                    name='7', id=6, color=[255, 128, 0], type='upper',
                    swap=''),
                7:
                dict(
                    name='8', id=7, color=[0, 255, 0], type='upper', swap='9'),
                8:
                dict(
                    name='9', id=8, color=[255, 128, 0], type='upper',
                    swap=''),
                9:
                dict(
                    name='10',
                    id=9,
                    color=[0, 255, 0],
                    type='upper',
                    swap='16'),
                10:
                dict(
                    name='11',
                    id=10,
                    color=[255, 128, 0],
                    type='upper',
                    swap='17'),
                11:
                dict(
                    name='12',
                    id=11,
                    color=[0, 255, 0],
                    type='lower',
                    swap='18'),
                12:
                dict(
                    name='13',
                    id=12,
                    color=[255, 128, 0],
                    type='lower',
                    swap='19'),
                13:
                dict(
                    name='14',
                    id=13,
                    color=[0, 255, 0],
                    type='lower',
                    swap='20'),
                14:
                dict(
                    name='15',
                    id=14,
                    color=[255, 128, 0],
                    type='lower',
                    swap='21'),
                15:
                dict(
                    name='16', id=15, color=[0, 255, 0], type='lower',
                    swap=''),
                16:
                dict(
                    name='17',
                    id=16,
                    color=[255, 128, 0],
                    type='lower',
                    swap=''),
                17:
                dict(
                    name='18', id=17, color=[0, 255, 0], type='lower',
                    swap=''),
                18:
                dict(
                    name='19', id=18, color=[0, 255, 0], type='lower',
                    swap=''),
                19:
                dict(
                    name='20', id=19, color=[0, 255, 0], type='lower',
                    swap=''),
                20:
                dict(
                    name='21', id=20, color=[0, 255, 0], type='lower', swap='')
            }),
            skeleton_info=dict({
                0:
                dict(link=('1', '2'), id=0, color=[0, 255, 0]),
                1:
                dict(link=('2', '3'), id=1, color=[0, 255, 0]),
                2:
                dict(link=('3', '4'), id=2, color=[255, 128, 0]),
                3:
                dict(link=('4', '5'), id=3, color=[255, 128, 0]),
                4:
                dict(link=('5', '6'), id=4, color=[51, 153, 255]),
                5:
                dict(link=('6', '7'), id=5, color=[51, 153, 255]),
                6:
                dict(link=('7', '8'), id=6, color=[51, 153, 255]),
                7:
                dict(link=('8', '13'), id=7, color=[51, 153, 255]),
                8:
                dict(link=('9', '10'), id=8, color=[0, 255, 0]),
                9:
                dict(link=('9', '16'), id=9, color=[255, 128, 0]),
                10:
                dict(link=('9', '13'), id=10, color=[0, 255, 0]),
                11:
                dict(link=('9', '19'), id=11, color=[255, 128, 0]),
                12:
                dict(link=('10', '11'), id=12, color=[51, 153, 255]),
                13:
                dict(link=('11', '12'), id=13, color=[51, 153, 255]),
                14:
                dict(link=('13', '14'), id=14, color=[51, 153, 255]),
                15:
                dict(link=('14', '15'), id=15, color=[51, 153, 255]),
                16:
                dict(link=('16', '17'), id=16, color=[51, 153, 255]),
                17:
                dict(link=('17', '18'), id=17, color=[51, 153, 255]),
                18:
                dict(link=('19', '20'), id=18, color=[51, 153, 255]),
                19:
                dict(link=('20', '21'), id=19, color=[51, 123, 255]),
                20:
                dict(link=('8', '19'), id=20, color=[51, 123, 255]),
                21:
                dict(link=('2', '10'), id=21, color=[51, 123, 255]),
                22:
                dict(link=('2', '16'), id=22, color=[51, 123, 255])
            }),
            joint_weights=[
                1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.2, 1.2, 1.5, 1.5, 1.0,
                1.0, 1.2, 1.2, 1.5, 1.5, 1.5, 1.5, 1.5, 1.5
            ],
            sigmas=[
                0.26, 0.25, 0.25, 0.35, 0.35, 0.79, 0.79, 0.72, 0.72, 0.9, 0.9,
                0.9, 1.07, 0.87, 0.87, 0.89, 0.89, 0.9, 0.9, 0.9, 0.9
            ])))

Below is my mmpose\datasets\datasets\animal\pig_animal.py file

# Copyright (c) OpenMMLab. All rights reserved.
import os.path as osp
import tempfile
import warnings
from collections import OrderedDict, defaultdict

import json_tricks as json
import numpy as np
from mmcv import Config, deprecated_api_warning
from xtcocotools.cocoeval import COCOeval

from ....core.post_processing import oks_nms, soft_oks_nms
from ...builder import DATASETS
from ..base import Kpt2dSviewRgbImgTopDownDataset

@DATASETS.register_module(name='PigAnimal')
class PigAnimal(Kpt2dSviewRgbImgTopDownDataset):
    """CocoDataset dataset for top-down pose estimation.

    "Microsoft COCO: Common Objects in Context", ECCV'2014.
    More details can be found in the `paper
    <https://arxiv.org/abs/1405.0312>`__ .

    The dataset loads raw features and apply specified transforms
    to return a dict containing the image tensors and other information.

    COCO keypoint indexes::

        0: 'nose',
        1: 'left_eye',
        2: 'right_eye',
        3: 'left_ear',
        4: 'right_ear',
        5: 'left_shoulder',
        6: 'right_shoulder',
        7: 'left_elbow',
        8: 'right_elbow',
        9: 'left_wrist',
        10: 'right_wrist',
        11: 'left_hip',
        12: 'right_hip',
        13: 'left_knee',
        14: 'right_knee',
        15: 'left_ankle',
        16: 'right_ankle'

    Args:
        ann_file (str): Path to the annotation file.
        img_prefix (str): Path to a directory where images are held.
            Default: None.
        data_cfg (dict): config
        pipeline (list[dict | callable]): A sequence of data transforms.
        dataset_info (DatasetInfo): A class containing all dataset info.
        test_mode (bool): Store True when building test or
            validation dataset. Default: False.
    """

    def __init__(self,
                 ann_file,
                 img_prefix,
                 data_cfg,
                 pipeline,
                 dataset_info=None,
                 test_mode=False):

        if dataset_info is None:
            warnings.warn(
                'dataset_info is missing. '
                'Check https://github.com/open-mmlab/mmpose/pull/663 '
                'for details.', DeprecationWarning)
            cfg = Config.fromfile('D:\Code\mmpose2\configs\_base_\datasets\pig_animal.py')
            dataset_info = cfg._cfg_dict['dataset_info']

        super().__init__(
            ann_file,
            img_prefix,
            data_cfg,
            pipeline,
            dataset_info=dataset_info,
            test_mode=test_mode)

        self.use_gt_bbox = data_cfg['use_gt_bbox']
        self.bbox_file = data_cfg['bbox_file']
        self.det_bbox_thr = data_cfg.get('det_bbox_thr', 0.0)
        self.use_nms = data_cfg.get('use_nms', True)
        self.soft_nms = data_cfg['soft_nms']
        self.nms_thr = data_cfg['nms_thr']
        self.oks_thr = data_cfg['oks_thr']
        self.vis_thr = data_cfg['vis_thr']

        self.db = self._get_db()

        print(f'=> num_images: {self.num_images}')
        print(f'=> load {len(self.db)} samples')

    def _get_db(self):
        """Load dataset."""
        if (not self.test_mode) or self.use_gt_bbox:
            # use ground truth bbox
            gt_db = self._load_coco_keypoint_annotations()
        else:
            # use bbox from detection
            gt_db = self._load_coco_person_detection_results()
        return gt_db

    def _load_coco_keypoint_annotations(self):
        """Ground truth bbox and keypoints."""
        gt_db = []
        for img_id in self.img_ids:
            gt_db.extend(self._load_coco_keypoint_annotation_kernel(img_id))
        return gt_db

    def _load_coco_keypoint_annotation_kernel(self, img_id):
        """load annotation from COCOAPI.

        Note:
            bbox:[x1, y1, w, h]

        Args:
            img_id: coco image id

        Returns:
            dict: db entry
        """
        img_ann = self.coco.loadImgs(img_id)[0]
        width = img_ann['width']
        height = img_ann['height']
        num_joints = self.ann_info['num_joints']

        ann_ids = self.coco.getAnnIds(imgIds=img_id, iscrowd=False)
        objs = self.coco.loadAnns(ann_ids)

        # sanitize bboxes
        valid_objs = []
        for obj in objs:
            if 'bbox' not in obj:
                continue
            x, y, w, h = obj['bbox']
            x1 = max(0, x)
            y1 = max(0, y)
            x2 = min(width - 1, x1 + max(0, w))
            y2 = min(height - 1, y1 + max(0, h))
            if ('area' not in obj or obj['area'] > 0) and x2 > x1 and y2 > y1:
                obj['clean_bbox'] = [x1, y1, x2 - x1, y2 - y1]
                valid_objs.append(obj)
        objs = valid_objs

        bbox_id = 0
        rec = []
        for obj in objs:
            if 'keypoints' not in obj:
                continue
            if max(obj['keypoints']) == 0:
                continue
            if 'num_keypoints' in obj and obj['num_keypoints'] == 0:
                continue
            joints_3d = np.zeros((num_joints, 3), dtype=np.float32)
            joints_3d_visible = np.zeros((num_joints, 3), dtype=np.float32)

            keypoints = np.array(obj['keypoints']).reshape(-1, 3)
            joints_3d[:, :2] = keypoints[:, :2]
            joints_3d_visible[:, :2] = np.minimum(1, keypoints[:, 2:3])

            image_file = osp.join(self.img_prefix, self.id2name[img_id])
            rec.append({
                'image_file': image_file,
                'bbox': obj['clean_bbox'][:4],
                'rotation': 0,
                'joints_3d': joints_3d,
                'joints_3d_visible': joints_3d_visible,
                'dataset': self.dataset_name,
                'bbox_score': 1,
                'bbox_id': bbox_id
            })
            bbox_id = bbox_id + 1

        return rec

    def _load_coco_person_detection_results(self):
        """Load coco person detection results."""
        num_joints = self.ann_info['num_joints']
        all_boxes = None
        with open(self.bbox_file, 'r') as f:
            all_boxes = json.load(f)

        if not all_boxes:
            raise ValueError('=> Load %s fail!' % self.bbox_file)

        print(f'=> Total boxes: {len(all_boxes)}')

        kpt_db = []
        bbox_id = 0
        for det_res in all_boxes:
            if det_res['category_id'] != 1:
                continue

            image_file = osp.join(self.img_prefix,
                                  self.id2name[det_res['image_id']])
            box = det_res['bbox']
            score = det_res['score']

            if score < self.det_bbox_thr:
                continue

            joints_3d = np.zeros((num_joints, 3), dtype=np.float32)
            joints_3d_visible = np.ones((num_joints, 3), dtype=np.float32)
            kpt_db.append({
                'image_file': image_file,
                'rotation': 0,
                'bbox': box[:4],
                'bbox_score': score,
                'dataset': self.dataset_name,
                'joints_3d': joints_3d,
                'joints_3d_visible': joints_3d_visible,
                'bbox_id': bbox_id
            })
            bbox_id = bbox_id + 1
        print(f'=> Total boxes after filter '
              f'low score@{self.det_bbox_thr}: {bbox_id}')
        return kpt_db

    @deprecated_api_warning(name_dict=dict(outputs='results'))
    def evaluate(self, results, res_folder=None, metric='mAP', **kwargs):
        """Evaluate coco keypoint results. The pose prediction results will be
        saved in ``${res_folder}/result_keypoints.json``.

        Note:
            - batch_size: N
            - num_keypoints: K
            - heatmap height: H
            - heatmap width: W

        Args:
            results (list[dict]): Testing results containing the following
                items:

                - preds (np.ndarray[N,K,3]): The first two dimensions are \
                    coordinates, score is the third dimension of the array.
                - boxes (np.ndarray[N,6]): [center[0], center[1], scale[0], \
                    scale[1],area, score]
                - image_paths (list[str]): For example, ['data/coco/val2017\
                    /000000393226.jpg']
                - heatmap (np.ndarray[N, K, H, W]): model output heatmap
                - bbox_id (list(int)).
            res_folder (str, optional): The folder to save the testing
                results. If not specified, a temp folder will be created.
                Default: None.
            metric (str | list[str]): Metric to be performed. Defaults: 'mAP'.

        Returns:
            dict: Evaluation results for evaluation metric.
        """
        metrics = metric if isinstance(metric, list) else [metric]
        allowed_metrics = ['mAP']
        for metric in metrics:
            if metric not in allowed_metrics:
                raise KeyError(f'metric {metric} is not supported')

        if res_folder is not None:
            tmp_folder = None
            res_file = osp.join(res_folder, 'result_keypoints.json')
        else:
            tmp_folder = tempfile.TemporaryDirectory()
            res_file = osp.join(tmp_folder.name, 'result_keypoints.json')

        kpts = defaultdict(list)

        for result in results:
            preds = result['preds']
            boxes = result['boxes']
            image_paths = result['image_paths']
            bbox_ids = result['bbox_ids']

            batch_size = len(image_paths)
            for i in range(batch_size):
                image_id = self.name2id[image_paths[i][len(self.img_prefix):]]
                kpts[image_id].append({
                    'keypoints': preds[i],
                    'center': boxes[i][0:2],
                    'scale': boxes[i][2:4],
                    'area': boxes[i][4],
                    'score': boxes[i][5],
                    'image_id': image_id,
                    'bbox_id': bbox_ids[i]
                })
        kpts = self._sort_and_unique_bboxes(kpts)

        # rescoring and oks nms
        num_joints = self.ann_info['num_joints']
        vis_thr = self.vis_thr
        oks_thr = self.oks_thr
        valid_kpts = []
        for image_id in kpts.keys():
            img_kpts = kpts[image_id]
            for n_p in img_kpts:
                box_score = n_p['score']
                if kwargs.get('rle_score', False):
                    pose_score = n_p['keypoints'][:, 2]
                    n_p['score'] = float(box_score + np.mean(pose_score) +
                                         np.max(pose_score))
                else:
                    kpt_score = 0
                    valid_num = 0
                    for n_jt in range(0, num_joints):
                        t_s = n_p['keypoints'][n_jt][2]
                        if t_s > vis_thr:
                            kpt_score = kpt_score + t_s
                            valid_num = valid_num + 1
                    if valid_num != 0:
                        kpt_score = kpt_score / valid_num
                    # rescoring
                    n_p['score'] = kpt_score * box_score

            if self.use_nms:
                nms = soft_oks_nms if self.soft_nms else oks_nms
                keep = nms(img_kpts, oks_thr, sigmas=self.sigmas)
                valid_kpts.append([img_kpts[_keep] for _keep in keep])
            else:
                valid_kpts.append(img_kpts)

        self._write_coco_keypoint_results(valid_kpts, res_file)

        # do evaluation only if the ground truth keypoint annotations exist
        if 'annotations' in self.coco.dataset:
            info_str = self._do_python_keypoint_eval(res_file)
            name_value = OrderedDict(info_str)

            if tmp_folder is not None:
                tmp_folder.cleanup()
        else:
            warnings.warn(f'Due to the absence of ground truth keypoint'
                          f'annotations, the quantitative evaluation can not'
                          f'be conducted. The prediction results have been'
                          f'saved at: {osp.abspath(res_file)}')
            name_value = {}

        return name_value

    def _write_coco_keypoint_results(self, keypoints, res_file):
        """Write results into a json file."""
        data_pack = [{
            'cat_id': self._class_to_coco_ind[cls],
            'cls_ind': cls_ind,
            'cls': cls,
            'ann_type': 'keypoints',
            'keypoints': keypoints
        } for cls_ind, cls in enumerate(self.classes)
                     if not cls == '__background__']

        results = self._coco_keypoint_results_one_category_kernel(data_pack[0])

        with open(res_file, 'w') as f:
            json.dump(results, f, sort_keys=True, indent=4)

    def _coco_keypoint_results_one_category_kernel(self, data_pack):
        """Get coco keypoint results."""
        cat_id = data_pack['cat_id']
        keypoints = data_pack['keypoints']
        cat_results = []

        for img_kpts in keypoints:
            if len(img_kpts) == 0:
                continue

            _key_points = np.array(
                [img_kpt['keypoints'] for img_kpt in img_kpts])
            key_points = _key_points.reshape(-1,
                                             self.ann_info['num_joints'] * 3)

            result = [{
                'image_id': img_kpt['image_id'],
                'category_id': cat_id,
                'keypoints': key_point.tolist(),
                'score': float(img_kpt['score']),
                'center': img_kpt['center'].tolist(),
                'scale': img_kpt['scale'].tolist()
            } for img_kpt, key_point in zip(img_kpts, key_points)]

            cat_results.extend(result)

        return cat_results

    def _do_python_keypoint_eval(self, res_file):
        """Keypoint evaluation using COCOAPI."""
        coco_det = self.coco.loadRes(res_file)
        coco_eval = COCOeval(self.coco, coco_det, 'keypoints', self.sigmas)
        coco_eval.params.useSegm = None
        coco_eval.evaluate()
        coco_eval.accumulate()
        coco_eval.summarize()

        stats_names = [
            'AP', 'AP .5', 'AP .75', 'AP (M)', 'AP (L)', 'AR', 'AR .5',
            'AR .75', 'AR (M)', 'AR (L)'
        ]

        info_str = list(zip(stats_names, coco_eval.stats))

        return info_str

    def _sort_and_unique_bboxes(self, kpts, key='bbox_id'):
        """sort kpts and remove the repeated ones."""
        for img_id, persons in kpts.items():
            num = len(persons)
            kpts[img_id] = sorted(kpts[img_id], key=lambda x: x[key])
            for i in range(num - 1, 0, -1):
                if kpts[img_id][i][key] == kpts[img_id][i - 1][key]:
                    del kpts[img_id][i]

        return kpts

Will the calculation of AP be related to my bbox_file? The label in this bbox_file is the frame information marked with labelme according to the pictures in the verification set after I divide the verification set.

Ben-Louis commented 1 year ago

Hi @hsp2454, thanks for using MMPose. Typically, the 'bbox_file' represents bounding boxes identified by object detectors. To assess the evaluation results' plausibility, you can remove the 'bbox_file' entries from the configuration and use the annotated bounding boxes instead.

hsp2454 commented 1 year ago

If I remove bbox_file and set use_gt_bbox to true, he reported the following error during training image

Ben-Louis commented 1 year ago

please following https://github.com/open-mmlab/mmpose/blob/0.x/configs/animal/2d_kpt_sview_rgb_img/topdown_heatmap/ap10k/res101_ap10k_256x256.py on how to avoid using 'bbox_file'

hsp2454 commented 1 year ago

Yes, I solved the bbox_file problem, but my AP is still 0

hsp2454 commented 1 year ago

Can someone answer my question, maybe which file I haven't modified?

Ben-Louis commented 1 year ago

There are several factors that could potentially lead to a zero AP score. It might be helpful to visualize the model's predictions on your validation data, as this could provide valuable insights into whether the issue lies within the data, the model, or the evaluation process itself.

hsp2454 commented 1 year ago

My data set is labeled with labelme. There are four targets in a picture. I first label each target box, then mark the points, save it as json data, and then use the script on the Internet to convert it into a coco data set. The following is my data. I read other issues, and it seems that someone mentioned that it is related to my area, because the script for converting data sets all areas to 1. If I want to modify the value of area by myself, then I should How to calculate it, I can’t calculate the value of area correctly through your case, for example, how to calculate area through bbox in the following.

https://drive.google.com/file/d/1in7t4okYiLiAvOtvtctCYyjnwHnXAjst/view?usp=sharing IDDJFB8 }6%$PFG2WCMH`78