search

open-mmlab / mmpose

OpenMMLab Pose Estimation Toolbox and Benchmark.
https://mmpose.readthedocs.io/en/latest/
Apache License 2.0
5.61k stars 1.22k forks source link

[Bug] Why the RTMO model train has CUDA kernel errors? #2920

Open seon-creator opened 8 months ago

seon-creator commented 8 months ago

Prerequisite

Environment

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
_openmp_mutex             5.1                       1_gnu  
addict                    2.4.0                    pypi_0    pypi
aliyun-python-sdk-core    2.14.0                   pypi_0    pypi
aliyun-python-sdk-kms     2.16.2                   pypi_0    pypi
attrs                     23.2.0                   pypi_0    pypi
ca-certificates           2023.12.12           h06a4308_0  
certifi                   2022.12.7                pypi_0    pypi
cffi                      1.16.0                   pypi_0    pypi
charset-normalizer        2.1.1                    pypi_0    pypi
chumpy                    0.70                     pypi_0    pypi
click                     8.1.7                    pypi_0    pypi
colorama                  0.4.6                    pypi_0    pypi
contourpy                 1.1.1                    pypi_0    pypi
coverage                  7.4.0                    pypi_0    pypi
crcmod                    1.7                      pypi_0    pypi
cryptography              41.0.7                   pypi_0    pypi
cycler                    0.12.1                   pypi_0    pypi
cython                    3.0.8                    pypi_0    pypi
exceptiongroup            1.2.0                    pypi_0    pypi
filelock                  3.9.0                    pypi_0    pypi
flake8                    7.0.0                    pypi_0    pypi
fonttools                 4.47.0                   pypi_0    pypi
fsspec                    2023.4.0                 pypi_0    pypi
idna                      3.4                      pypi_0    pypi
importlib-metadata        7.0.1                    pypi_0    pypi
importlib-resources       6.1.1                    pypi_0    pypi
iniconfig                 2.0.0                    pypi_0    pypi
interrogate               1.5.0                    pypi_0    pypi
isort                     4.3.21                   pypi_0    pypi
jinja2                    3.1.2                    pypi_0    pypi
jmespath                  0.10.0                   pypi_0    pypi
json-tricks               3.17.3                   pypi_0    pypi
kiwisolver                1.4.5                    pypi_0    pypi
ld_impl_linux-64          2.38                 h1181459_1  
libffi                    3.4.4                h6a678d5_0  
libgcc-ng                 11.2.0               h1234567_1  
libgomp                   11.2.0               h1234567_1  
libstdcxx-ng              11.2.0               h1234567_1  
markdown                  3.5.2                    pypi_0    pypi
markdown-it-py            3.0.0                    pypi_0    pypi
markupsafe                2.1.3                    pypi_0    pypi
matplotlib                3.7.4                    pypi_0    pypi
mccabe                    0.7.0                    pypi_0    pypi
mdurl                     0.1.2                    pypi_0    pypi
mmcv                      2.1.0                    pypi_0    pypi
mmdet                     3.2.0                    pypi_0    pypi
mmengine                  0.10.2                   pypi_0    pypi
mmpose                    1.3.0                     dev_0    <develop>
model-index               0.1.11                   pypi_0    pypi
mpmath                    1.3.0                    pypi_0    pypi
munkres                   1.1.4                    pypi_0    pypi
ncurses                   6.4                  h6a678d5_0  
networkx                  3.0                      pypi_0    pypi
numpy                     1.24.1                   pypi_0    pypi
opencv-python             4.9.0.80                 pypi_0    pypi
opendatalab               0.0.10                   pypi_0    pypi
openmim                   0.3.9                    pypi_0    pypi
openssl                   3.0.12               h7f8727e_0  
openxlab                  0.0.33                   pypi_0    pypi
ordered-set               4.1.0                    pypi_0    pypi
oss2                      2.17.0                   pypi_0    pypi
packaging                 23.2                     pypi_0    pypi
pandas                    2.0.3                    pypi_0    pypi
parameterized             0.9.0                    pypi_0    pypi
pillow                    9.3.0                    pypi_0    pypi
pip                       23.3.1           py38h06a4308_0  
platformdirs              4.1.0                    pypi_0    pypi
pluggy                    1.3.0                    pypi_0    pypi
py                        1.11.0                   pypi_0    pypi
pycocotools               2.0.7                    pypi_0    pypi
pycodestyle               2.11.1                   pypi_0    pypi
pycparser                 2.21                     pypi_0    pypi
pycryptodome              3.20.0                   pypi_0    pypi
pyflakes                  3.2.0                    pypi_0    pypi
pygments                  2.17.2                   pypi_0    pypi
pyparsing                 3.1.1                    pypi_0    pypi
pytest                    7.4.4                    pypi_0    pypi
pytest-runner             6.0.1                    pypi_0    pypi
python                    3.8.18               h955ad1f_0  
python-dateutil           2.8.2                    pypi_0    pypi
pytz                      2023.3.post1             pypi_0    pypi
pyyaml                    6.0.1                    pypi_0    pypi
readline                  8.2                  h5eee18b_0  
requests                  2.28.2                   pypi_0    pypi
rich                      13.4.2                   pypi_0    pypi
scipy                     1.10.1                   pypi_0    pypi
setuptools                60.2.0                   pypi_0    pypi
shapely                   2.0.2                    pypi_0    pypi
six                       1.16.0                   pypi_0    pypi
sqlite                    3.41.2               h5eee18b_0  
sympy                     1.12                     pypi_0    pypi
tabulate                  0.9.0                    pypi_0    pypi
termcolor                 2.4.0                    pypi_0    pypi
terminaltables            3.1.10                   pypi_0    pypi
tk                        8.6.12               h1ccaba5_0  
toml                      0.10.2                   pypi_0    pypi
tomli                     2.0.1                    pypi_0    pypi
torch                     2.1.2+cu118              pypi_0    pypi
torchaudio                2.1.2+cu118              pypi_0    pypi
torchvision               0.16.2+cu118             pypi_0    pypi
tqdm                      4.65.2                   pypi_0    pypi
triton                    2.1.0                    pypi_0    pypi
typing-extensions         4.4.0                    pypi_0    pypi
tzdata                    2023.4                   pypi_0    pypi
urllib3                   1.26.13                  pypi_0    pypi
wheel                     0.41.2           py38h06a4308_0  
xdoctest                  1.1.2                    pypi_0    pypi
xtcocotools               1.14.3                   pypi_0    pypi
xz                        5.4.5                h5eee18b_0  
yapf                      0.40.2                   pypi_0    pypi
zipp                      3.17.0                   pypi_0    pypi
zlib                      1.2.13               h5eee18b_0

Reproduces the problem - code sample

configs/base/datasets/coco_arm.py

dataset_info = dict(

    dataset_name='coco', 
    paper_info=dict(
        author='Lin, Tsung-Yi and Maire, Michael and '
        'Belongie, Serge and Hays, James and '
        'Perona, Pietro and Ramanan, Deva and '
        r'Doll{\'a}r, Piotr and Zitnick, C Lawrence',
        title='Microsoft coco: Common objects in context',
        container='European conference on computer vision',
        year='2014',
        homepage='http://cocodataset.org/',
    ),
    keypoint_info={ 
        0:
        dict(
            name='LI11',
            id=0,
            color=[255,255,255],
            type='upper',
            swap='LI11'),
        1:
        dict(
            name='LI10',
            id=1,
            color=[255,255,255],
            type='upper',
            swap='LI10'),
        2:
        dict(
            name='TE5',
            id=2,
            color=[255,255,255],
            type='upper',
            swap='TE5'),
        3:
        dict(
            name='LI4',
            id=3,
            color=[255,255,255],
            type='upper',
            swap='LI4'),
        4:
        dict(
            name='TE3',
            id=4,
            color=[255,255,255],
            type='upper',
            swap='TE3')

    },
    skeleton_info={   

        0:
        dict(link=('LI11', 'LI11'), id=0, color=[255, 0, 0]),
        1:
        dict(link=('LI10', 'LI10'), id=1, color=[255, 0, 0]),
        2:
        dict(link=('TE5', 'TE5'), id=2, color=[255, 0, 0]),
        3:
        dict(link=('LI4', 'LI4'), id=3, color=[255, 0, 0]),
        4:
        dict(link=('TE3', 'TE3'), id=4, color=[255, 0, 0]),
    },
    joint_weights=[ # COCO dataset에서 사용하는 keypoint에 대한 가중치 (대부분 1)
        1., 1., 1., 1., 1.
    ],
    sigmas=[ 
        0.015, 0.015, 0.015, 0.015, 0.015 # edit : 0.025 -> 0.005 -> 0.015
    ]
)

RTMO-l model mmpose/configs/body_2d_keypoint/rtmo/coco/rtmo-l_16xb16-600e_coco-640x640.py

_base_ = ['../../../_base_/default_runtime.py']

# runtime
train_cfg = dict(max_epochs=600, val_interval=20, dynamic_intervals=[(580, 1)])

auto_scale_lr = dict(base_batch_size=256)

default_hooks = dict(
    checkpoint=dict(type='CheckpointHook', interval=40, max_keep_ckpts=3))

optim_wrapper = dict(
    type='OptimWrapper',
    constructor='ForceDefaultOptimWrapperConstructor',
    optimizer=dict(type='AdamW', lr=0.004, weight_decay=0.05),
    paramwise_cfg=dict(
        norm_decay_mult=0,
        bias_decay_mult=0,
        bypass_duplicate=True,
        force_default_settings=True,
        custom_keys=dict({'neck.encoder': dict(lr_mult=0.05)})),
    clip_grad=dict(max_norm=0.1, norm_type=2))

param_scheduler = [
    dict(
        type='QuadraticWarmupLR',
        by_epoch=True,
        begin=0,
        end=5,
        convert_to_iter_based=True),
    dict(
        type='CosineAnnealingLR',
        eta_min=0.0002,
        begin=5,
        T_max=280,
        end=280,
        by_epoch=True,
        convert_to_iter_based=True),
    # this scheduler is used to increase the lr from 2e-4 to 5e-4
    dict(type='ConstantLR', by_epoch=True, factor=2.5, begin=280, end=281),
    dict(
        type='CosineAnnealingLR',
        eta_min=0.0002,
        begin=281,
        T_max=300,
        end=580,
        by_epoch=True,
        convert_to_iter_based=True),
    dict(type='ConstantLR', by_epoch=True, factor=1, begin=580, end=600),
]

# data
input_size = (640, 640)
metafile = 'configs/_base_/datasets/coco_arm.py'
codec = dict(type='YOLOXPoseAnnotationProcessor', input_size=input_size)

train_pipeline_stage1 = [
    dict(type='LoadImage', backend_args=None),
    dict(
        type='Mosaic',
        img_scale=(640, 640),
        pad_val=114.0,
        pre_transform=[dict(type='LoadImage', backend_args=None)]),
    dict(
        type='BottomupRandomAffine',
        input_size=(640, 640),
        shift_factor=0.1,
        rotate_factor=10,
        scale_factor=(0.75, 1.0),
        pad_val=114,
        distribution='uniform',
        transform_mode='perspective',
        bbox_keep_corner=False,
        clip_border=True,
    ),
    dict(
        type='YOLOXMixUp',
        img_scale=(640, 640),
        ratio_range=(0.8, 1.6),
        pad_val=114.0,
        pre_transform=[dict(type='LoadImage', backend_args=None)]),
    dict(type='YOLOXHSVRandomAug'),
    dict(type='RandomFlip'),
    dict(type='FilterAnnotations', by_kpt=True, by_box=True, keep_empty=False),
    dict(type='GenerateTarget', encoder=codec),
    dict(type='PackPoseInputs'),
]
train_pipeline_stage2 = [
    dict(type='LoadImage'),
    dict(
        type='BottomupRandomAffine',
        input_size=(640, 640),
        scale_type='long',
        pad_val=(114, 114, 114),
        bbox_keep_corner=False,
        clip_border=True,
    ),
    dict(type='YOLOXHSVRandomAug'),
    dict(type='RandomFlip'),
    dict(type='BottomupGetHeatmapMask', get_invalid=True),
    dict(type='FilterAnnotations', by_kpt=True, by_box=True, keep_empty=False),
    dict(type='GenerateTarget', encoder=codec),
    dict(type='PackPoseInputs'),
]

data_type = 'CocoArm'
data_mode = 'bottomup'
data_root = '/data/home/seondeok/Project/acupoint/coco_dataset/arm/revised_dataset/PK_dataset/Train1596/'
test_root = '/data/home/seondeok/Project/acupoint/coco_dataset/arm/revised_dataset/PK_dataset/Test392/'
annotation_root = '/data/home/seondeok/Project/acupoint/coco_dataset/arm/revised_dataset/PK_dataset/Train1596_v1.json'
annotation_root_val = '/data/home/seondeok/Project/acupoint/coco_dataset/arm/revised_dataset/PK_dataset/Test392_v1.json'

# train datasets
dataset_coco = dict(
    type=data_type,
    data_root=data_root,    # edit
    data_mode=data_mode,   
    ann_file=annotation_root,   # edit
    data_prefix=dict(img=data_root),    # edit
    pipeline=train_pipeline_stage1,
)

train_dataloader = dict(
    batch_size=16,
    num_workers=8,
    persistent_workers=True,
    pin_memory=True,
    sampler=dict(type='DefaultSampler', shuffle=True),
    dataset=dataset_coco)

val_pipeline = [
    dict(type='LoadImage'),
    dict(
        type='BottomupResize', input_size=input_size, pad_val=(114, 114, 114)),
    dict(
        type='PackPoseInputs',
        meta_keys=('id', 'img_id', 'img_path', 'ori_shape', 'img_shape',
                   'input_size', 'input_center', 'input_scale'))
]

val_dataloader = dict(
    batch_size=1,
    num_workers=2,
    persistent_workers=True,
    pin_memory=True,
    drop_last=False,
    sampler=dict(type='DefaultSampler', shuffle=False, round_up=False),
    dataset=dict(
        type=data_type, # edit
        data_root=data_root,    # edit
        data_mode=data_mode,   
        ann_file=annotation_root_val,   # edit
        data_prefix=dict(img=test_root),    # edit
        test_mode=True,
        pipeline=val_pipeline,
    ))
test_dataloader = val_dataloader

# evaluators
val_evaluator = dict(
    type='CocoMetric',
    ann_file=annotation_root_val,   # edit
    score_mode='bbox',  
    nms_mode='none',
)
test_evaluator = val_evaluator

# hooks
custom_hooks = [
    dict(
        type='YOLOXPoseModeSwitchHook',
        num_last_epochs=20,
        new_train_pipeline=train_pipeline_stage2,
        priority=48),
    dict(
        type='RTMOModeSwitchHook',
        epoch_attributes={
            280: {
                'proxy_target_cc': True,
                'overlaps_power': 1.0,
                'loss_cls.loss_weight': 2.0,
                'loss_mle.loss_weight': 5.0,
                'loss_oks.loss_weight': 10.0
            },
        },
        priority=48),
    dict(type='SyncNormHook', priority=48),
    dict(
        type='EMAHook',
        ema_type='ExpMomentumEMA',
        momentum=0.0002,
        update_buffers=True,
        strict_load=False,
        priority=49),
]

# model
widen_factor = 1.0
deepen_factor = 1.0

model = dict(
    type='BottomupPoseEstimator',
    init_cfg=dict(
        type='Kaiming',
        layer='Conv2d',
        a=2.23606797749979,
        distribution='uniform',
        mode='fan_in',
        nonlinearity='leaky_relu'),
    data_preprocessor=dict(
        type='PoseDataPreprocessor',
        pad_size_divisor=32,
        mean=[0, 0, 0],
        std=[1, 1, 1],
        batch_augments=[
            dict(
                type='BatchSyncRandomResize',
                random_size_range=(480, 800),
                size_divisor=32,
                interval=1),
        ]),
    backbone=dict(
        type='CSPDarknet',
        deepen_factor=deepen_factor,
        widen_factor=widen_factor,
        out_indices=(2, 3, 4),
        spp_kernal_sizes=(5, 9, 13),
        norm_cfg=dict(type='BN', momentum=0.03, eps=0.001),
        act_cfg=dict(type='Swish'),
        init_cfg=dict(
            type='Pretrained',
            checkpoint='/data/home/seondeok/MMPose/mmpose/configs/body_2d_keypoint/rtmo/pretrained/yolox_l_8x8_300e_coco_20211126_140236-d3bd2b23.pth',
            prefix='backbone.',
        )),
    neck=dict(
        type='HybridEncoder',
        in_channels=[256, 512, 1024],
        deepen_factor=deepen_factor,
        widen_factor=widen_factor,
        hidden_dim=256,
        output_indices=[1, 2],
        encoder_cfg=dict(
            self_attn_cfg=dict(embed_dims=256, num_heads=8, dropout=0.0),
            ffn_cfg=dict(
                embed_dims=256,
                feedforward_channels=1024,
                ffn_drop=0.0,
                act_cfg=dict(type='GELU'))),
        projector=dict(
            type='ChannelMapper',
            in_channels=[256, 256],
            kernel_size=1,
            out_channels=512,
            act_cfg=None,
            norm_cfg=dict(type='BN'),
            num_outs=2)),
    head=dict(
        type='RTMOHead',
        num_keypoints=5,    # edit keypoints
        featmap_strides=(16, 32),
        head_module_cfg=dict(
            num_classes=1,  # edit
            in_channels=256,
            cls_feat_channels=256,
            channels_per_group=36,
            pose_vec_channels=512,
            widen_factor=widen_factor,
            stacked_convs=2,
            norm_cfg=dict(type='BN', momentum=0.03, eps=0.001),
            act_cfg=dict(type='Swish')),
        assigner=dict(
            type='SimOTAAssigner',
            dynamic_k_indicator='oks',
            oks_calculator=dict(type='PoseOKS', metainfo=metafile)),
        prior_generator=dict(
            type='MlvlPointGenerator',
            centralize_points=True,
            strides=[16, 32]),
        dcc_cfg=dict(
            in_channels=512,
            feat_channels=128,
            num_bins=(192, 256),
            spe_channels=128,
            gau_cfg=dict(
                s=128,
                expansion_factor=2,
                dropout_rate=0.0,
                drop_path=0.0,
                act_fn='SiLU',
                pos_enc='add')),
        overlaps_power=0.5,
        loss_cls=dict(
            type='VariFocalLoss',
            reduction='sum',
            use_target_weight=True,
            loss_weight=1.0),
        loss_bbox=dict(
            type='IoULoss',
            mode='square',
            eps=1e-16,
            reduction='sum',
            loss_weight=5.0),
        loss_oks=dict(
            type='OKSLoss',
            reduction='none',
            metainfo=metafile,
            loss_weight=30.0),
        loss_vis=dict(
            type='BCELoss',
            use_target_weight=True,
            reduction='mean',
            loss_weight=1.0),
        loss_mle=dict(
            type='MLECCLoss',
            use_target_weight=True,
            loss_weight=1e-2,
        ),
        loss_bbox_aux=dict(type='L1Loss', reduction='sum', loss_weight=1.0),
    ),
    test_cfg=dict(
        input_size=input_size,
        score_thr=0.1,
        nms_thr=0.65,
    ))

Reproduces the problem - command or script

$ python tools/train.py configs/body_2d_keypoint/rtmo/custom_coco/rtmo-l_16xb16-600e_coco-640x640.py

Reproduces the problem - error message

../aten/src/ATen/native/cuda/ScatterGatherKernel.cu:365: operator(): block: [0,0,0], thread: [0,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/ScatterGatherKernel.cu:365: operator(): block: [0,0,0], thread: [1,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/data/home/seondeok/.conda/envs/openmmlab_v2/lib/python3.8/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3526.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
Traceback (most recent call last):
  File "tools/train.py", line 162, in <module>
    main()
  File "tools/train.py", line 158, in main
    runner.train()
  File "/data/home/seondeok/.conda/envs/openmmlab_v2/lib/python3.8/site-packages/mmengine/runner/runner.py", line 1777, in train
    model = self.train_loop.run()  # type: ignore
  File "/data/home/seondeok/.conda/envs/openmmlab_v2/lib/python3.8/site-packages/mmengine/runner/loops.py", line 96, in run
    self.run_epoch()
  File "/data/home/seondeok/.conda/envs/openmmlab_v2/lib/python3.8/site-packages/mmengine/runner/loops.py", line 112, in run_epoch
    self.run_iter(idx, data_batch)
  File "/data/home/seondeok/.conda/envs/openmmlab_v2/lib/python3.8/site-packages/mmengine/runner/loops.py", line 128, in run_iter
    outputs = self.runner.model.train_step(
  File "/data/home/seondeok/.conda/envs/openmmlab_v2/lib/python3.8/site-packages/mmengine/model/base_model/base_model.py", line 114, in train_step
    losses = self._run_forward(data, mode='loss')  # type: ignore
  File "/data/home/seondeok/.conda/envs/openmmlab_v2/lib/python3.8/site-packages/mmengine/model/base_model/base_model.py", line 346, in _run_forward
    results = self(**data, mode=mode)
  File "/data/home/seondeok/.conda/envs/openmmlab_v2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/data/home/seondeok/.conda/envs/openmmlab_v2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/data/home/seondeok/MMPose/mmpose/mmpose/models/pose_estimators/base.py", line 155, in forward
    return self.loss(inputs, data_samples)
  File "/data/home/seondeok/MMPose/mmpose/mmpose/models/pose_estimators/bottomup.py", line 70, in loss
    self.head.loss(feats, data_samples, train_cfg=self.train_cfg))
  File "/data/home/seondeok/MMPose/mmpose/mmpose/models/heads/hybrid_heads/rtmo_head.py", line 793, in loss
    targets = self._get_targets(flatten_priors,
  File "/data/home/seondeok/.conda/envs/openmmlab_v2/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/data/home/seondeok/MMPose/mmpose/mmpose/models/heads/hybrid_heads/yoloxpose_head.py", line 410, in _get_targets
    target = self._get_targets_single(priors, batch_cls_scores[i],
  File "/data/home/seondeok/.conda/envs/openmmlab_v2/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/data/home/seondeok/MMPose/mmpose/mmpose/models/heads/hybrid_heads/yoloxpose_head.py", line 532, in _get_targets_single
    assign_result = self.assigner.assign(
  File "/data/home/seondeok/MMPose/mmpose/mmpose/models/task_modules/assigners/sim_ota_assigner.py", line 166, in assign
    F.binary_cross_entropy(
  File "/data/home/seondeok/.conda/envs/openmmlab_v2/lib/python3.8/site-packages/torch/nn/functional.py", line 3122, in binary_cross_entropy
    return torch._C._nn.binary_cross_entropy(input, target, weight, reduction_enum)
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Additional information

  1. When I try to train RTMO model, CUDA Kernal error accur.
  2. So I tried other model like topdown-heatmap : HRNet model : mmpose/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-210e_coco-256x192.py
  3. For HRNet model, train code works well

But, with RTMO model, It stopped with this step and CUDA error accur.

loading annotations into memory...
Done (t=0.00s)
creating index...
index created!
loading annotations into memory...
Done (t=0.00s)
creating index...
index created!
01/11 17:09:16 - mmengine - INFO - load backbone. in model from: /data/home/seondeok/MMPose/mmpose/configs/body_2d_keypoint/rtmo/pretrained/yolox_l_8x8_300e_coco_20211126_140236-d3bd2b23.pth
Loads checkpoint by local backend from path: /data/home/seondeok/MMPose/mmpose/configs/body_2d_keypoint/rtmo/pretrained/yolox_l_8x8_300e_coco_20211126_140236-d3bd2b23.pth
01/11 17:09:18 - mmengine - WARNING - "FileClient" will be deprecated in future. Please use io functions in https://mmengine.readthedocs.io/en/latest/api/fileio.html#file-io
01/11 17:09:18 - mmengine - WARNING - "HardDiskBackend" is the alias of "LocalBackend" and the former will be deprecated in future.
01/11 17:09:18 - mmengine - INFO - Checkpoints will be saved to /data/home/seondeok/MMPose/mmpose/work_dirs/rtmo-l_16xb16-600e_coco-640x640.

How to solve this problem?

Ben-Louis commented 8 months ago

Please make sure the "category_id" in your data annotation file matches the one in the COCO annotation file.

seon-creator commented 8 months ago

Thank you for advice! My category_id is 9, and how to match the id in the config code? My custom coco annotation id is matched with 9. Should I have to change this number to 0?

lishiyu005 commented 7 months ago

I have the same error. Category id can only be 1. If you have other id like 0 or 9, you can get error.