search

open-mmlab / mmpose

OpenMMLab Pose Estimation Toolbox and Benchmark.
https://mmpose.readthedocs.io/en/latest/
Apache License 2.0
5.4k stars 1.18k forks source link

how to evalutate on a subset of coco keypoints #2098

Closed darcula1993 closed 1 year ago

darcula1993 commented 1 year ago

I am trying to build a pose detector which only detect upper body keypoints of coco2017. The CombinedDataset works pretty well for training part. But I cannot find a way to do coco metric evaluation on CombinedDataset. Following is my config:

# keypoint mappings
keypoint_mapping_coco = [
    (0, 0),
    (5, 1),
    (6, 2),
    (7, 3),
    (8, 4),
    (9, 5),
    (10, 6),
    (11, 7),
    (12, 8)
]

# train datasets
dataset_coco = dict(
    type='CocoDataset',
    data_root='/lixinwei/mmpose/data/coco/',
    data_mode='topdown',
    ann_file='annotations/person_keypoints_train2017.json',
    data_prefix=dict(img='train2017/'),
    pipeline=[
        dict(
            type='KeypointConverter',
            num_keypoints=9,
            mapping=keypoint_mapping_coco)
    ],
)

dataset_coco_val = dict(
    type='CocoDataset',
    data_root='/lixinwei/mmpose/data/coco/',
    data_mode='topdown',
    ann_file='annotations/person_keypoints_val2017_easy.json',
    data_prefix=dict(img='val2017/'),
    pipeline=[
        dict(
            type='KeypointConverter',
            num_keypoints=9,
            mapping=keypoint_mapping_coco)
    ],
)

train_dataloader = dict(
    batch_size=512,
    num_workers=20,
    persistent_workers=True,
    sampler=dict(type='DefaultSampler', shuffle=True),
    dataset=dict(
        type='CombinedDataset',
        metainfo=dict(from_file='configs/_base_/datasets/cocopose_halfbody.py'),
        datasets=[dataset_coco],
        pipeline=train_pipeline,
        test_mode=False,
    ))

val_dataloader = dict(
    batch_size=512,
    num_workers=20,
    persistent_workers=True,
    drop_last=False,
    sampler=dict(type='DefaultSampler', shuffle=False, round_up=False),
    dataset=dict(
        type='CombinedDataset',
        metainfo=dict(from_file='configs/_base_/datasets/cocopose_halfbody.py'),
        datasets=[dataset_coco_val],
        pipeline=val_pipeline,
        test_mode=True,
    ))
test_dataloader = val_dataloader

# evaluators
val_evaluator = dict(
    type='CocoMetric',
    ann_file = '/lixinwei/mmpose/data/coco/' + 'annotations/person_keypoints_val2017_easy.json'
    )
test_evaluator = val_evaluator

but it seems not work for evaluation part. I get the error: ValueError: operands could not be broadcast together with shapes (9,) (17,)

Any idea?

Ben-Louis commented 1 year ago

If you give CocoMetric the ann_file, it will load the ground truth from a local annotation file instead of the val_dataset. This means it will have 17 keypoints, not just 9 keypoints. You could try removing the ann_file argument and give it another go.

darcula1993 commented 1 year ago

I try but got another error:

[523] Traceback (most recent call last):
[524]   File "./tools/train.py", line 161, in <module>
[525]     main()
[526]   File "./tools/train.py", line 157, in main
[527]     runner.train()
[528]   File "/usr/local/lib/python3.6/dist-packages/mmengine/runner/runner.py", line 1684, in train
[529]     model = self.train_loop.run()  # type: ignore
[530]   File "/usr/local/lib/python3.6/dist-packages/mmengine/runner/loops.py", line 96, in run
[531]     self.runner.val_loop.run()
[532]   File "/usr/local/lib/python3.6/dist-packages/mmengine/runner/loops.py", line 350, in run
[533]     metrics = self.evaluator.evaluate(len(self.dataloader.dataset))
[534]   File "/usr/local/lib/python3.6/dist-packages/mmengine/evaluator/evaluator.py", line 79, in evaluate
[535]     _results = metric.evaluate(size)
[536]   File "/usr/local/lib/python3.6/dist-packages/mmengine/evaluator/metric.py", line 110, in evaluate
[537]     _metrics = self.compute_metrics(results)  # type: ignore
[538]   File "/lixinwei/mmpose-1.0.0rc0/mmpose/evaluation/metrics/coco_metric.py", line 334, in compute_metrics
[539]     gt_dicts=gts, outfile_prefix=outfile_prefix)
[540]   File "/lixinwei/mmpose-1.0.0rc0/mmpose/evaluation/metrics/coco_metric.py", line 300, in gt_to_coco_json
[541]     categories=self.dataset_meta['CLASSES'],
[542] KeyError: 'CLASSES'
Ben-Louis commented 1 year ago

This problem is strange. We're unable to replicate it. We suggest updating your mmegine and mmpose version. If the problem doesn't go away, you could put your code in a part of your forked mmpose repo if you wish, and we can help find what's wrong.

darcula1993 commented 1 year ago

I update mmegine and mmpose and mmc. The version is mmcv 2.0.0rc4 mmengine 0.7.0 mmpose 1.0.0rc1 but problem still there. I upload my dataset definition and config code here。the model backbone just pick random one. halfbody.zip

darcula1993 commented 1 year ago

@Ben-Louis could you replicate the problem with these config files?

Ben-Louis commented 1 year ago

Hi @darcula1993, sorry for the late reply. We've managed to reproduce the issues you mentioned:

val_dataloader = dict( batch_size=512, num_workers=20, persistent_workers=True, drop_last=False, sampler=dict(type='DefaultSampler', shuffle=False, round_up=False), dataset=dataset_coco_val) test_dataloader = val_dataloader



- The `ValueError: operands could not be broadcast together with shapes (9,) (17,)` issue arises because the `KeypointConverter` is unable to convert keypoint annotations in the `raw_ann_info` item, which is used by `CocoMetric`.

We appreciate your information and will work on fixing these issues in the coming days. If you're interested in contributing to MMPose by addressing these problems, we would be truly grateful for your assistance.
darcula1993 commented 1 year ago

hi @Ben-Louis thank you very much for your rely! So I just change the config file like you show? should I remove the val_evaluator part? I try to keep the val_evaluator but got error 

[405] Traceback (most recent call last):
[406]   File "/lixinwei/mmpose-1.x/./tools/train.py", line 157, in <module>
[407]     main()
[408]   File "/lixinwei/mmpose-1.x/./tools/train.py", line 153, in main
[409]     runner.train()
[410]   File "/usr/local/lib/python3.9/site-packages/mmengine/runner/runner.py", line 1701, in train
[411]     model = self.train_loop.run()  # type: ignore
[412]   File "/usr/local/lib/python3.9/site-packages/mmengine/runner/loops.py", line 102, in run
[413]     self.runner.val_loop.run()
[414]   File "/usr/local/lib/python3.9/site-packages/mmengine/runner/loops.py", line 366, in run
[415]     metrics = self.evaluator.evaluate(len(self.dataloader.dataset))
[416]   File "/usr/local/lib/python3.9/site-packages/mmengine/evaluator/evaluator.py", line 79, in evaluate
[417]     _results = metric.evaluate(size)
[418]   File "/usr/local/lib/python3.9/site-packages/mmengine/evaluator/metric.py", line 115, in evaluate
[419]     _metrics = self.compute_metrics(results)  # type: ignore
[420]   File "/lixinwei/mmpose-1.x/mmpose/evaluation/metrics/coco_metric.py", line 405, in compute_metrics
[421]     kpt_score = instance['keypoint_scores'][kpt_idx]
[422] IndexError: index 9 is out of bounds for axis 0 with size 9
Ben-Louis commented 1 year ago

Oh, my fault for missing the metainfo argument in the dataset_coco_val in the above code. BTW, val_evaluator should be kept.

darcula1993 commented 1 year ago

@Ben-Louis OK！But the error come back to ValueError: operands could not be broadcast together with shapes (9,) (17,). I guess that's the bug we should try to fix. Do you have any suggestion which part I should take a look at first?

Ben-Louis commented 1 year ago

To resolve this issue, you need to transform the keypoints in the raw_ann_info within the KeypointConverter.transform. To achieve this, you'll need to

  1. Locate where raw_ann_info is within the results
  2. Determine the keys within raw_ann_info and identify which of them are related to keypoints.
  3. Transform the keypoints using the given argument num_keypoints and mapping

Once you have completed these steps, the metric should run smoothly. For more information on the reasoning behind these actions, please refer to the BaseCocoStyleDataset to understand how raw_ann_info is formed, and CocoMetric on how to use raw_ann_info to recover ground truth data.

If you successfully resolve the problem, welcome to submit a PR to MMPose. A detailed guide is available for reference. Your contribution will be highly appreciated. Feel free to contact us if you have any further question