search

open-mmlab / mmpose

OpenMMLab Pose Estimation Toolbox and Benchmark.
https://mmpose.readthedocs.io/en/latest/
Apache License 2.0
5.84k stars 1.25k forks source link

How can I train bottom up models in latest version of mmpose? [Bug] #2669

Closed seon-creator closed 1 year ago

seon-creator commented 1 year ago

Prerequisite

Environment

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
_openmp_mutex             5.1                       1_gnu  
addict                    2.4.0                    pypi_0    pypi
aliyun-python-sdk-core    2.13.36                  pypi_0    pypi
aliyun-python-sdk-kms     2.16.1                   pypi_0    pypi
attrs                     23.1.0                   pypi_0    pypi
ca-certificates           2023.05.30           h06a4308_0  
certifi                   2022.12.7                pypi_0    pypi
cffi                      1.15.1                   pypi_0    pypi
charset-normalizer        2.1.1                    pypi_0    pypi
chumpy                    0.70                     pypi_0    pypi
click                     8.1.7                    pypi_0    pypi
cmake                     3.25.0                   pypi_0    pypi
colorama                  0.4.6                    pypi_0    pypi
contourpy                 1.1.0                    pypi_0    pypi
coverage                  7.3.0                    pypi_0    pypi
crcmod                    1.7                      pypi_0    pypi
cryptography              41.0.3                   pypi_0    pypi
cycler                    0.11.0                   pypi_0    pypi
cython                    3.0.2                    pypi_0    pypi
exceptiongroup            1.1.3                    pypi_0    pypi
filelock                  3.9.0                    pypi_0    pypi
flake8                    6.1.0                    pypi_0    pypi
fonttools                 4.42.1                   pypi_0    pypi
idna                      3.4                      pypi_0    pypi
importlib-metadata        6.8.0                    pypi_0    pypi
importlib-resources       6.0.1                    pypi_0    pypi
iniconfig                 2.0.0                    pypi_0    pypi
interrogate               1.5.0                    pypi_0    pypi
isort                     4.3.21                   pypi_0    pypi
jinja2                    3.1.2                    pypi_0    pypi
jmespath                  0.10.0                   pypi_0    pypi
json-tricks               3.17.3                   pypi_0    pypi
kiwisolver                1.4.5                    pypi_0    pypi
ld_impl_linux-64          2.38                 h1181459_1  
libffi                    3.4.4                h6a678d5_0  
libgcc-ng                 11.2.0               h1234567_1  
libgomp                   11.2.0               h1234567_1  
libstdcxx-ng              11.2.0               h1234567_1  
lit                       15.0.7                   pypi_0    pypi
markdown                  3.4.4                    pypi_0    pypi
markdown-it-py            3.0.0                    pypi_0    pypi
markupsafe                2.1.2                    pypi_0    pypi
matplotlib                3.7.2                    pypi_0    pypi
mccabe                    0.7.0                    pypi_0    pypi
mdurl                     0.1.2                    pypi_0    pypi
mmcv                      2.0.1                    pypi_0    pypi
mmdet                     3.1.0                    pypi_0    pypi
mmengine                  0.8.4                    pypi_0    pypi
mmpose                    1.1.0                     dev_0    <develop>
model-index               0.1.11                   pypi_0    pypi
mpmath                    1.2.1                    pypi_0    pypi
munkres                   1.1.4                    pypi_0    pypi
ncurses                   6.4                  h6a678d5_0  
networkx                  3.0                      pypi_0    pypi
numpy                     1.24.1                   pypi_0    pypi
opencv-python             4.8.0.76                 pypi_0    pypi
opendatalab               0.0.10                   pypi_0    pypi
openmim                   0.3.9                    pypi_0    pypi
openssl                   3.0.10               h7f8727e_2  
openxlab                  0.0.23                   pypi_0    pypi
ordered-set               4.1.0                    pypi_0    pypi
oss2                      2.17.0                   pypi_0    pypi
packaging                 23.1                     pypi_0    pypi
pandas                    2.0.3                    pypi_0    pypi
parameterized             0.9.0                    pypi_0    pypi
pillow                    9.3.0                    pypi_0    pypi
pip                       23.2.1           py38h06a4308_0  
platformdirs              3.10.0                   pypi_0    pypi
pluggy                    1.3.0                    pypi_0    pypi
py                        1.11.0                   pypi_0    pypi
pycocotools               2.0.7                    pypi_0    pypi
pycodestyle               2.11.0                   pypi_0    pypi
pycparser                 2.21                     pypi_0    pypi
pycryptodome              3.18.0                   pypi_0    pypi
pyflakes                  3.1.0                    pypi_0    pypi
pygments                  2.16.1                   pypi_0    pypi
pyparsing                 3.0.9                    pypi_0    pypi
pytest                    7.4.0                    pypi_0    pypi
pytest-runner             6.0.0                    pypi_0    pypi
python                    3.8.17               h955ad1f_0  
python-dateutil           2.8.2                    pypi_0    pypi
pytz                      2023.3                   pypi_0    pypi
pyyaml                    6.0.1                    pypi_0    pypi
readline                  8.2                  h5eee18b_0  
requests                  2.28.2                   pypi_0    pypi
rich                      13.4.2                   pypi_0    pypi
scipy                     1.10.1                   pypi_0    pypi
setuptools                60.2.0                   pypi_0    pypi
shapely                   2.0.1                    pypi_0    pypi
six                       1.16.0                   pypi_0    pypi
sqlite                    3.41.2               h5eee18b_0  
sympy                     1.11.1                   pypi_0    pypi
tabulate                  0.9.0                    pypi_0    pypi
termcolor                 2.3.0                    pypi_0    pypi
terminaltables            3.1.10                   pypi_0    pypi
tk                        8.6.12               h1ccaba5_0  
toml                      0.10.2                   pypi_0    pypi
tomli                     2.0.1                    pypi_0    pypi
torch                     2.0.1+cu118              pypi_0    pypi
torchaudio                2.0.2+cu118              pypi_0    pypi
torchvision               0.15.2+cu118             pypi_0    pypi
tqdm                      4.65.2                   pypi_0    pypi
triton                    2.0.0                    pypi_0    pypi
typing-extensions         4.4.0                    pypi_0    pypi
tzdata                    2023.3                   pypi_0    pypi
urllib3                   1.26.13                  pypi_0    pypi
wheel                     0.38.4           py38h06a4308_0  
xdoctest                  1.1.1                    pypi_0    pypi
xtcocotools               1.13                     pypi_0    pypi
xz                        5.4.2                h5eee18b_0  
yapf                      0.40.1                   pypi_0    pypi
zipp                      3.16.2                   pypi_0    pypi
zlib                      1.2.13               h5eee18b_0

The environment is followed by mmpose_Tutorial.ipynb.

Reproduces the problem - code sample

_base_ = ['../../../_base_/default_runtime.py']

# runtime
train_cfg = dict(max_epochs=200, val_interval=10)

# optimizer
optim_wrapper = dict(optimizer=dict(
    type='Adam',
    lr=1e-3,
))

# learning policy
param_scheduler = [
    dict(
        type='LinearLR', begin=0, end=500, start_factor=0.001,
        by_epoch=False),  # warm-up
    dict(
        type='MultiStepLR',
        begin=0,
        end=140,
        milestones=[90, 120],
        gamma=0.1,
        by_epoch=True)
]

# automatically scaling LR based on the actual training batch size
auto_scale_lr = dict(base_batch_size=80)

# hooks
default_hooks = dict(checkpoint=dict(save_best='coco/AP', rule='greater'))

# codec settings
codec = dict(
    type='SPR',
    input_size=(512, 512),
    heatmap_size=(128, 128),
    sigma=(4, 2),
    minimal_diagonal_length=32**0.5,
    generate_keypoint_heatmaps=True,
    decode_max_instances=30)

# model settings
model = dict(
    type='BottomupPoseEstimator',
    data_preprocessor=dict(
        type='PoseDataPreprocessor',
        mean=[123.675, 116.28, 103.53],
        std=[58.395, 57.12, 57.375],
        bgr_to_rgb=True),
    backbone=dict(
        type='HRNet',
        in_channels=3,
        extra=dict(
            stage1=dict(
                num_modules=1,
                num_branches=1,
                block='BOTTLENECK',
                num_blocks=(4, ),
                num_channels=(64, )),
            stage2=dict(
                num_modules=1,
                num_branches=2,
                block='BASIC',
                num_blocks=(4, 4),
                num_channels=(32, 64)),
            stage3=dict(
                num_modules=4,
                num_branches=3,
                block='BASIC',
                num_blocks=(4, 4, 4),
                num_channels=(32, 64, 128)),
            stage4=dict(
                num_modules=3,
                num_branches=4,
                block='BASIC',
                num_blocks=(4, 4, 4, 4),
                num_channels=(32, 64, 128, 256),
                multiscale_output=True)),
        init_cfg=dict(
            type='Pretrained',
            checkpoint='configs/body_2d_keypoint/dekr/custom_hrnet_arm/dekr_hrnet-w32_8xb10-140e_coco-512x512_ac7c17bf-20221228.pth'),
    ),
    neck=dict(
        type='FeatureMapProcessor',
        concat=True,
    ),
    head=dict(
        type='DEKRHead',
        in_channels=480,
        num_keypoints=5, # edit from 17
        heatmap_loss=dict(type='KeypointMSELoss', use_target_weight=True),
        displacement_loss=dict(
            type='SoftWeightSmoothL1Loss',
            use_target_weight=True,
            supervise_empty=False,
            beta=1 / 9,
            loss_weight=0.002,
        ),
        decoder=codec,

        rescore_cfg=dict(
            in_channels=74,
            norm_indexes=(5, 6),
        #     init_cfg=dict(
        #         type='Pretrained',
        #         checkpoint='https://download.openmmlab.com/mmpose/'
        #         'pretrain_models/kpt_rescore_coco-33d58c5c.pth'),
        )
    ),
    test_cfg=dict(
        multiscale_test=False,
        flip_test=True,
        nms_dist_thr=0.05,
        shift_heatmap=True,
        align_corners=False))

# enable DDP training when rescore net is used
find_unused_parameters = False  # edit from True

# base dataset settings
dataset_type = 'CocoArm'
data_mode = 'bottomup'
data_root = 'data/coco/'

# pipelines
train_pipeline = [
    dict(type='LoadImage'),
    dict(type='BottomupRandomAffine', input_size=codec['input_size']),
    dict(type='RandomFlip', direction='horizontal'),
    dict(type='GenerateTarget', encoder=codec),
    dict(type='BottomupGetHeatmapMask'),
    dict(type='PackPoseInputs'),
]
val_pipeline = [
    dict(type='LoadImage'),
    dict(
        type='BottomupResize',
        input_size=codec['input_size'],
        size_factor=32,
        resize_mode='expand'),
    dict(
        type='PackPoseInputs',
        meta_keys=('id', 'img_id', 'img_path', 'crowd_index', 'ori_shape',
                   'img_shape', 'input_size', 'input_center', 'input_scale',
                   'flip', 'flip_direction', 'flip_indices', 'raw_ann_info',
                   'skeleton_links'))
]

# data loaders
train_dataloader = dict(
    batch_size=8,
    num_workers=1,
    persistent_workers=True,
    sampler=dict(type='DefaultSampler', shuffle=True),
    dataset=dict(
        type=dataset_type,
        data_root='/data/home/seondeok/Project/acupoint/mmpose/datasets/arm/PK_arm_Train/',
        data_mode=data_mode,
        ann_file='/data/home/seondeok/Project/acupoint/mmpose/datasets/arm/json_file/PK_arm_Train.json',
        data_prefix=dict(img='/data/home/seondeok/Project/acupoint/mmpose/datasets/arm/PK_arm_Train/'),
        pipeline=train_pipeline,
    ))
val_dataloader = dict(
    batch_size=1,
    num_workers=1,
    persistent_workers=True,
    drop_last=False,
    sampler=dict(type='DefaultSampler', shuffle=False, round_up=False),
    dataset=dict(
        type=dataset_type,
        data_root='/data/home/seondeok/Project/acupoint/mmpose/datasets/arm/PK_arm_Test/',
        data_mode=data_mode,
        ann_file='/data/home/seondeok/Project/acupoint/mmpose/datasets/arm/json_file/PK_arm_Test.json',
        data_prefix=dict(img='/data/home/seondeok/Project/acupoint/mmpose/datasets/arm/PK_arm_Test/'),
        test_mode=True,
        pipeline=val_pipeline,
    ))
test_dataloader = val_dataloader

# evaluators
val_evaluator = dict(
    type='CocoMetric',
    ann_file='/data/home/seondeok/Project/acupoint/mmpose/datasets/arm/json_file/PK_arm_Test.json',
    nms_mode='none',
    score_mode='keypoint',
)
test_evaluator = val_evaluator

Reproduces the problem - command or script

python tools/train.py configs/body_2d_keypoint/topdown_heatmap/custom_hrnet_coco_arm/td-hm_hrnet-w32_8xb64-210e_coco-256x192.py

Reproduces the problem - error message

Traceback (most recent call last):
  File "tools/train.py", line 161, in <module>
    main()
  File "tools/train.py", line 157, in main
    runner.train()
  File "/data/home/seondeok/.conda/envs/openmmlab/lib/python3.8/site-packages/mmengine/runner/runner.py", line 1745, in train
    model = self.train_loop.run()  # type: ignore
  File "/data/home/seondeok/.conda/envs/openmmlab/lib/python3.8/site-packages/mmengine/runner/loops.py", line 96, in run
    self.run_epoch()
  File "/data/home/seondeok/.conda/envs/openmmlab/lib/python3.8/site-packages/mmengine/runner/loops.py", line 111, in run_epoch
    for idx, data_batch in enumerate(self.dataloader):
  File "/data/home/seondeok/.conda/envs/openmmlab/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 633, in __next__
    data = self._next_data()
  File "/data/home/seondeok/.conda/envs/openmmlab/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1345, in _next_data
    return self._process_data(data)
  File "/data/home/seondeok/.conda/envs/openmmlab/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1371, in _process_data
    data.reraise()
  File "/data/home/seondeok/.conda/envs/openmmlab/lib/python3.8/site-packages/torch/_utils.py", line 644, in reraise
    raise exception
IndexError: Caught IndexError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/data/home/seondeok/.conda/envs/openmmlab/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loop
    data = fetcher.fetch(index)
  File "/data/home/seondeok/.conda/envs/openmmlab/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 51, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/data/home/seondeok/.conda/envs/openmmlab/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 51, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/data/home/seondeok/.conda/envs/openmmlab/lib/python3.8/site-packages/mmengine/dataset/base_dataset.py", line 408, in __getitem__
    data = self.prepare_data(idx)
  File "/data/home/seondeok/.conda/envs/openmmlab/lib/python3.8/site-packages/mmengine/dataset/base_dataset.py", line 113, in wrapper
    return old_func(obj, *args, **kwargs)
  File "/data/home/seondeok/Project/acupoint/mmpose/mmpose/datasets/datasets/base/base_coco_style_dataset.py", line 150, in prepare_data
    return self.pipeline(data_info)
  File "/data/home/seondeok/.conda/envs/openmmlab/lib/python3.8/site-packages/mmengine/dataset/base_dataset.py", line 58, in __call__
    data = t(data)
  File "/data/home/seondeok/.conda/envs/openmmlab/lib/python3.8/site-packages/mmcv/transforms/base.py", line 12, in __call__
    return self.transform(results)
  File "/data/home/seondeok/Project/acupoint/mmpose/mmpose/datasets/transforms/bottomup_transforms.py", line 89, in transform
    mask = 1 - self._segs_to_mask(invalid_segs, img_shape)
  File "/data/home/seondeok/Project/acupoint/mmpose/mmpose/datasets/transforms/bottomup_transforms.py", line 53, in _segs_to_mask
    rle = cocomask.frPyObjects(seg, img_shape[0], img_shape[1])
  File "xtcocotools/_mask.pyx", line 292, in xtcocotools._mask.frPyObjects
IndexError: list index out of range

Additional information

  1. In Top down case the coco dataset is not problem. But, in Bottom up case, It has problem.
  2. This is my coco labeling data.(Get from coco-annotator) --> ex) train_data.json
{
    "images": [
        {
            "id": 73032,
            "dataset_id": 98,
            "category_ids": [],
            "path": "/datasets/Revise_done_arm_Test/001_arm1_0.png",
            "width": 1488,
            "height": 837,
            "file_name": "001_arm1_0.png",
            "annotated": false,
            "annotating": [],
            "num_annotations": 0,
            "metadata": {},
            "deleted": false,
            "milliseconds": 0,
            "events": [],
            "regenerate_thumbnail": false
        },
    "categories": [
        {
            "id": 9,
            "name": "armpoint",
            "supercategory": "arm",
            "color": "#3bcb02",
            "metadata": {},
            "keypoint_colors": [
                "#bf5c4d",
                "#d99100",
                "#4d8068",
                "#0d2b80",
                "#9c73bf"
            ],
            "keypoints": [
                "li11",
                "li10",
                "te5",
                "li4",
                "te3"
            ],
            "skeleton": []
        }
    ],
    "annotations": [
        {
            "id": 77655,
            "image_id": 73032,
            "category_id": 9,
            "segmentation": [
                [
                    542.3,
                    294.8,
                    618.3,
                    351.7,
                    810.8,
                    385.9,
                    818.4,
                    389.3,
                    956.1,
                    389.3,
                    1125.8,
                    334.8,
                    1225.1,
                    388.5,
                    1281.9,
                    455.7,
                    1266.7,
                    538.2,
                    1189.9,
                    596.6,
                    1021.8,
                    590.2,
                    817.6,
                    575.8,
                    602.3,
                    539,
                    599.9,
                    539,
                    599.9,
                    540.6,
                    407.8,
                    469.3,
                    367,
                    388.5,
                    466.2,
                    354.9,
                    543.1,
                    296.4
                ]
            ],
            "area": 170625,
            "bbox": [
                367,
                296,
                915,
                301
            ],
            "iscrowd": false,
            "isbbox": false,
            "color": "#f2ae60",
            "keypoints": [
                479,
                377,
                2,
                566,
                406,
                2,
                858,
                485,
                2,
                1031,
                428,
                2,
                1020,
                528,
                2
            ],
            "metadata": {},
            "num_keypoints": 5
        },
}

Is the format in bottom up training has problem?

Ben-Louis commented 1 year ago

If your dataset lacks mask annotations, you can set up your configuration file in line with the CrowdPose example found here: https://github.com/open-mmlab/mmpose/blob/537bd8e543ab463fb55120d5caaa1ae22d6aaf06/configs/body_2d_keypoint/dekr/crowdpose/dekr_hrnet-w32_8xb10-300e_crowdpose-512x512.py#L124-L131

seon-creator commented 1 year ago

Thank you for your advice! After then I tried to train and it's work. But when evaluation steps, It gets error.

09/01 21:55:02 - mmengine - WARNING - "FileClient" will be deprecated in future. Please use io functions in https://mmengine.readthedocs.io/en/latest/api/fileio.html#file-io
09/01 21:55:02 - mmengine - WARNING - "HardDiskBackend" is the alias of "LocalBackend" and the former will be deprecated in future.
09/01 21:55:02 - mmengine - INFO - Checkpoints will be saved to /data/home/seondeok/Project/acupoint/mmpose/work_dirs/Bottom_up/DEKR_hrnet_w32_0901.
09/01 21:55:21 - mmengine - INFO - Epoch(train)   [1][ 50/106]  lr: 9.909820e-05  eta: 2:10:22  time: 0.369846  data_time: 0.191051  memory: 7741  loss: 0.001332  loss/heatmap: 0.000879  loss/displacement: 0.000454
09/01 21:55:38 - mmengine - INFO - Epoch(train)   [1][100/106]  lr: 1.991984e-04  eta: 2:06:07  time: 0.347460  data_time: 0.189022  memory: 7741  loss: 0.001064  loss/heatmap: 0.000614  loss/displacement: 0.000450
09/01 21:55:40 - mmengine - INFO - Exp name: dekr_hrnet-w32_8xb10-140e_coco-512x512_20230901_215454
09/01 21:55:49 - mmengine - INFO - Epoch(train)   [2][ 50/106]  lr: 3.113106e-04  eta: 1:44:02  time: 0.167862  data_time: 0.009961  memory: 7741  loss: 0.000945  loss/heatmap: 0.000509  loss/displacement: 0.000436
09/01 21:55:57 - mmengine - INFO - Epoch(train)   [2][100/106]  lr: 4.114108e-04  eta: 1:32:45  time: 0.166636  data_time: 0.008195  memory: 7741  loss: 0.000918  loss/heatmap: 0.000494  loss/displacement: 0.000424
09/01 21:55:58 - mmengine - INFO - Exp name: dekr_hrnet-w32_8xb10-140e_coco-512x512_20230901_215454
09/01 21:56:06 - mmengine - INFO - Epoch(train)   [3][ 50/106]  lr: 5.235230e-04  eta: 1:25:16  time: 0.169351  data_time: 0.011285  memory: 7741  loss: 0.000854  loss/heatmap: 0.000450  loss/displacement: 0.000404
09/01 21:56:15 - mmengine - INFO - Epoch(train)   [3][100/106]  lr: 6.236232e-04  eta: 1:20:43  time: 0.166557  data_time: 0.007964  memory: 7741  loss: 0.000805  loss/heatmap: 0.000432  loss/displacement: 0.000372
09/01 21:56:16 - mmengine - INFO - Exp name: dekr_hrnet-w32_8xb10-140e_coco-512x512_20230901_215454
09/01 21:56:24 - mmengine - INFO - Epoch(train)   [4][ 50/106]  lr: 7.357355e-04  eta: 1:17:05  time: 0.167826  data_time: 0.009669  memory: 7741  loss: 0.000706  loss/heatmap: 0.000408  loss/displacement: 0.000298
09/01 21:56:32 - mmengine - INFO - Epoch(train)   [4][100/106]  lr: 8.358357e-04  eta: 1:14:40  time: 0.168171  data_time: 0.009046  memory: 7742  loss: 0.000585  loss/heatmap: 0.000378  loss/displacement: 0.000208
09/01 21:56:33 - mmengine - INFO - Exp name: dekr_hrnet-w32_8xb10-140e_coco-512x512_20230901_215454
09/01 21:56:42 - mmengine - INFO - Epoch(train)   [5][ 50/106]  lr: 9.479479e-04  eta: 1:12:30  time: 0.168413  data_time: 0.009599  memory: 7742  loss: 0.000525  loss/heatmap: 0.000376  loss/displacement: 0.000148
09/01 21:56:50 - mmengine - INFO - Epoch(train)   [5][100/106]  lr: 1.000000e-03  eta: 1:10:54  time: 0.166663  data_time: 0.008285  memory: 7742  loss: 0.000498  loss/heatmap: 0.000380  loss/displacement: 0.000118
09/01 21:56:51 - mmengine - INFO - Exp name: dekr_hrnet-w32_8xb10-140e_coco-512x512_20230901_215454
09/01 21:56:59 - mmengine - INFO - Epoch(train)   [6][ 50/106]  lr: 1.000000e-03  eta: 1:09:26  time: 0.168582  data_time: 0.009675  memory: 7742  loss: 0.000448  loss/heatmap: 0.000346  loss/displacement: 0.000101
09/01 21:57:08 - mmengine - INFO - Epoch(train)   [6][100/106]  lr: 1.000000e-03  eta: 1:08:19  time: 0.167408  data_time: 0.008816  memory: 7742  loss: 0.000419  loss/heatmap: 0.000335  loss/displacement: 0.000083
09/01 21:57:09 - mmengine - INFO - Exp name: dekr_hrnet-w32_8xb10-140e_coco-512x512_20230901_215454
09/01 21:57:17 - mmengine - INFO - Epoch(train)   [7][ 50/106]  lr: 1.000000e-03  eta: 1:07:16  time: 0.169302  data_time: 0.009737  memory: 7742  loss: 0.000415  loss/heatmap: 0.000336  loss/displacement: 0.000079
09/01 21:57:26 - mmengine - INFO - Epoch(train)   [7][100/106]  lr: 1.000000e-03  eta: 1:06:24  time: 0.166421  data_time: 0.007720  memory: 7741  loss: 0.000364  loss/heatmap: 0.000298  loss/displacement: 0.000067
09/01 21:57:27 - mmengine - INFO - Exp name: dekr_hrnet-w32_8xb10-140e_coco-512x512_20230901_215454
09/01 21:57:35 - mmengine - INFO - Epoch(train)   [8][ 50/106]  lr: 1.000000e-03  eta: 1:05:34  time: 0.168857  data_time: 0.009598  memory: 7742  loss: 0.000360  loss/heatmap: 0.000296  loss/displacement: 0.000064
09/01 21:57:43 - mmengine - INFO - Epoch(train)   [8][100/106]  lr: 1.000000e-03  eta: 1:04:55  time: 0.168401  data_time: 0.009147  memory: 7741  loss: 0.000356  loss/heatmap: 0.000292  loss/displacement: 0.000064
09/01 21:57:44 - mmengine - INFO - Exp name: dekr_hrnet-w32_8xb10-140e_coco-512x512_20230901_215454
09/01 21:57:53 - mmengine - INFO - Epoch(train)   [9][ 50/106]  lr: 1.000000e-03  eta: 1:04:14  time: 0.168675  data_time: 0.009869  memory: 7741  loss: 0.000342  loss/heatmap: 0.000284  loss/displacement: 0.000058
09/01 21:58:01 - mmengine - INFO - Epoch(train)   [9][100/106]  lr: 1.000000e-03  eta: 1:03:42  time: 0.168189  data_time: 0.008862  memory: 7741  loss: 0.000346  loss/heatmap: 0.000290  loss/displacement: 0.000056
09/01 21:58:02 - mmengine - INFO - Exp name: dekr_hrnet-w32_8xb10-140e_coco-512x512_20230901_215454
09/01 21:58:10 - mmengine - INFO - Exp name: dekr_hrnet-w32_8xb10-140e_coco-512x512_20230901_215454
09/01 21:58:11 - mmengine - INFO - Epoch(train)  [10][ 50/106]  lr: 1.000000e-03  eta: 1:03:07  time: 0.168612  data_time: 0.009756  memory: 7741  loss: 0.000323  loss/heatmap: 0.000273  loss/displacement: 0.000050
09/01 21:58:19 - mmengine - INFO - Epoch(train)  [10][100/106]  lr: 1.000000e-03  eta: 1:02:38  time: 0.167257  data_time: 0.007840  memory: 7741  loss: 0.000335  loss/heatmap: 0.000280  loss/displacement: 0.000055
09/01 21:58:20 - mmengine - INFO - Exp name: dekr_hrnet-w32_8xb10-140e_coco-512x512_20230901_215454
09/01 21:58:20 - mmengine - INFO - Saving checkpoint at 10 epochs
/data/home/seondeok/.conda/envs/openmmlab/lib/python3.8/site-packages/torch/nn/functional.py:4236: UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for det$
  warnings.warn(
Traceback (most recent call last):
  File "tools/train.py", line 161, in <module>
    main()
  File "tools/train.py", line 157, in main
    runner.train()
  File "/data/home/seondeok/.conda/envs/openmmlab/lib/python3.8/site-packages/mmengine/runner/runner.py", line 1745, in train
    model = self.train_loop.run()  # type: ignore
  File "/data/home/seondeok/.conda/envs/openmmlab/lib/python3.8/site-packages/mmengine/runner/loops.py", line 102, in run
    self.runner.val_loop.run()
  File "/data/home/seondeok/.conda/envs/openmmlab/lib/python3.8/site-packages/mmengine/runner/loops.py", line 363, in run
    self.run_iter(idx, data_batch)
  File "/data/home/seondeok/.conda/envs/openmmlab/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/data/home/seondeok/.conda/envs/openmmlab/lib/python3.8/site-packages/mmengine/runner/loops.py", line 383, in run_iter
    outputs = self.runner.model.val_step(data_batch)
  File "/data/home/seondeok/.conda/envs/openmmlab/lib/python3.8/site-packages/mmengine/model/base_model/base_model.py", line 133, in val_step
    return self._run_forward(data, mode='predict')  # type: ignore
  File "/data/home/seondeok/.conda/envs/openmmlab/lib/python3.8/site-packages/mmengine/model/base_model/base_model.py", line 340, in _run_forward
    results = self(**data, mode=mode)
  File "/data/home/seondeok/.conda/envs/openmmlab/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/data/home/seondeok/Project/acupoint/mmpose/mmpose/models/pose_estimators/base.py", line 142, in forward
    return self.predict(inputs, data_samples)
  File "/data/home/seondeok/Project/acupoint/mmpose/mmpose/models/pose_estimators/bottomup.py", line 127, in predict
    preds = self.head.predict(feats, data_samples, test_cfg=self.test_cfg)
  File "/data/home/seondeok/Project/acupoint/mmpose/mmpose/models/heads/hybrid_heads/dekr_head.py", line 460, in predict
    preds = self.decode(heatmaps, displacements, test_cfg, metainfo)
  File "/data/home/seondeok/Project/acupoint/mmpose/mmpose/models/heads/hybrid_heads/dekr_head.py", line 520, in decode
    instance_scores = self.rescore_net(keypoints, keypoint_scores,
  File "/data/home/seondeok/.conda/envs/openmmlab/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/data/home/seondeok/Project/acupoint/mmpose/mmpose/models/heads/hybrid_heads/dekr_head.py", line 161, in forward
    feature = self.make_feature(keypoints, keypoint_scores, skeleton)
  File "/data/home/seondeok/Project/acupoint/mmpose/mmpose/models/heads/hybrid_heads/dekr_head.py", line 147, in make_feature
    normalize = (joint_length[:, self.norm_indexes[0]] +
IndexError: index 5 is out of bounds for dimension 1 with size 5

Should I have to change another things in config? My dataset has 5 keypoints.

Ben-Louis commented 1 year ago

Please remove the rescore_cfg from the config for custom datasets that have a different number of keypoints than COCO or CrowdPose. The rescore nets are pretrained on respective datasets and will not be updated during training.

seon-creator commented 1 year ago

Thank you, so I removed the rescore_cfg and It's work. But all of the AP is 0.000. How to solve it?

Loading and preparing results...
DONE (t=0.01s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *keypoints*
DONE (t=0.01s).
Accumulating evaluation results...
DONE (t=0.00s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets= 20 ] =  0.000
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets= 20 ] =  0.000
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets= 20 ] =  0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets= 20 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets= 20 ] =  0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 20 ] =  0.000
 Average Recall     (AR) @[ IoU=0.50      | area=   all | maxDets= 20 ] =  0.000
 Average Recall     (AR) @[ IoU=0.75      | area=   all | maxDets= 20 ] =  0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets= 20 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets= 20 ] =  0.000
Ben-Louis commented 1 year ago

Have you visualized the predictions of the model on your data? You can follow this doc to check the model outputs

seon-creator commented 1 year ago

Yes, the visualized results works and keypoints are also well presented at 200epochs or 300epochs. But the AP values are all zero, so I don't know which epochs is the best AP.

Ben-Louis commented 1 year ago

The metric AP is highly sensitive to the parameter 'sigmas' defined in the dataset metainformation file. Perhaps you could try to enlarge the value of 'sigmas' for each keypoint and check if the AP goes up

bc-arl commented 1 year ago

I am getting the same problem training DEKR, my model gives good results on validation data but AP values are all 0. Increasing sigma does not help. Did you ever figure this out?