Prerequisite

[X] I have searched Issues and Discussions but cannot get the expected help.
[X] The bug has not been fixed in the latest version(https://github.com/open-mmlab/mmpose).

Environment

OrderedDict([('sys.platform', 'win32'), ('Python', '3.8.19 (default, Mar 20 2024, 19:55:45) [MSC v.1916 64 bit (AMD64)]'), ('CUDA available', True), ('MUSA available', False), ('numpy_random_seed', 2147483648), ('GPU 0', 'NVIDIA RTX 3500 Ada Generation Laptop GPU'), ('CUDA_HOME', 'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4'), ('NVCC', 'Cuda compilation tools, release 12.4, V12.4.99'), ('GCC', 'n/a'), ('PyTorch', '2.2.2'), ('PyTorch compiling details', 'PyTorch built with:\n - C++ Version: 201703\n - MSVC 192930151\n - Intel(R) Math Kernel Library Version 2020.0.2 Product Build 20200624 for Intel(R) 64 architecture applications\n - Intel(R) MKL-DNN v3.3.2 (Git Hash 2dc95a2ad0841e29db8b22fbccaf3e5da7992b01)\n - OpenMP 2019\n - LAPACK is enabled (usually provided by MKL)\n - CPU capability usage: AVX2\n - CUDA Runtime 12.1\n - NVCC architecture flags: -gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90\n - CuDNN 8.8.1 (built against CUDA 12.0)\n - Magma 2.5.4\n - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=8.8.1, CXX_COMPILER=C:/cb/pytorch_1000000000000/work/tmp_bin/sccache-cl.exe, CXX_FLAGS=/DWIN32 /D_WINDOWS /GR /EHsc /Zc:__cplusplus /bigobj /FS /utf-8 -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE /wd4624 /wd4068 /wd4067 /wd4267 /wd4661 /wd4717 /wd4244 /wd4804 /wd4273, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.2.2, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=OFF, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, \n'), ('TorchVision', '0.17.2'), ('OpenCV', '4.9.0'), ('MMEngine', '0.10.3'), ('MMPose', '1.3.1+5a3be94')])

mmcv 2.1.0 mmdet 3.3.0 mmengine 0.10.3 mmpose 1.3.1

Reproduces the problem - code sample

base = ['../../configs/base/default_runtime.py']

runtime

train_cfg = dict(max_epochs=210, val_interval=1)

optimizer

optim_wrapper = dict(optimizer=dict( type='Adam', lr=5e-4, ))

learning policy

param_scheduler = [ dict( type='LinearLR', begin=0, end=500, start_factor=0.001, by_epoch=False), # warm-up dict( type='MultiStepLR', begin=0, end=210, milestones=[100, 150], gamma=0.1, by_epoch=True) ]

automatically scaling LR based on the actual training batch size

auto_scale_lr = dict(base_batch_size=512)

hooks

default_hooks = dict(checkpoint=dict(save_best='coco/AP', rule='greater'))

codec settings

codec = dict( type='MSRAHeatmap', input_size=(384, 384), heatmap_size=(96, 96), sigma=3)

model settings

model = dict( type='TopdownPoseEstimator', data_preprocessor=dict( type='PoseDataPreprocessor', mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], bgr_to_rgb=True), backbone=dict( type='HourglassNet', num_stacks=1, init_cfg=dict( type='Pretrained', prefix="backbone.", checkpoint='models/hg/model1.pth' ), ), head=dict( type='CPMHead', in_channels=256, out_channels=17, num_stages=1, deconv_out_channels=None, loss=dict(type='KeypointMSELoss', use_target_weight=True), decoder=codec), test_cfg=dict( flip_test=True, flip_mode='heatmap', shift_heatmap=True, ))

base dataset settings

dataset_type = 'HumanArtDataset' data_mode = 'topdown' data_root = 'data/humanart/'

visualizer = dict(vis_backends=[ dict(type='LocalVisBackend'), dict(type='TensorboardVisBackend'), ])

pipelines

train_pipeline = [ dict(type='LoadImage'), dict(type='GetBBoxCenterScale'), dict(type='RandomHalfBody'), dict(type='RandomBBoxTransform'), dict(type='TopdownAffine', input_size=codec['input_size']), dict(type='GenerateTarget', encoder=codec), dict(type='PackPoseInputs') ] val_pipeline = [ dict(type='LoadImage'), dict(type='GetBBoxCenterScale'), dict(type='TopdownAffine', input_size=codec['input_size']), dict(type='PackPoseInputs') ]

####### TRAIN PART###########

train_water = dict( type=dataset_type, data_root=data_root, ann_file='HumanArt/annotations/training_humanart_watercolor.json', pipeline=[], ) train_oil = dict( type=dataset_type, data_root=data_root, ann_file='HumanArt/annotations/training_humanart_oil_painting.json', pipeline=[], ) train_ink = dict( type=dataset_type, data_root=data_root, ann_file='HumanArt/annotations/training_humanart_ink_painting.json', pipeline=[], )

Data loaders

train_dataloader = dict( batch_size=32, num_workers=2, persistent_workers=True,sampler=dict(type='DefaultSampler', shuffle=False, round_up=False), dataset=dict( type='CombinedDataset', metainfo=dict(from_file='configs/base/datasets/coco.py'), datasets=[train_water, train_oil, train_ink], pipeline=train_pipeline, test_mode=False, ), )

######## VAL PART ########### water_colo_val = dict( type=dataset_type, data_root=data_root, data_mode=data_mode, ann_file='HumanArt/annotations/validation_humanart_watercolor.json', bbox_file='data/humanart/HumanArt/person_detection_results/HumanArt_watercolor_validation_detections_AP_H_56_person.json', test_mode=True, pipeline=[]
)

oil_val = dict( type=dataset_type, data_root=data_root, data_mode=data_mode, ann_file='HumanArt/annotations/validation_humanart_oil_painting.json', bbox_file='data/humanart/HumanArt/person_detection_results/HumanArt_oil_painting_validation_detections_AP_H_56_person.json', test_mode=True, pipeline=[]
)

ink_val = dict( type=dataset_type, data_root=data_root, data_mode=data_mode, ann_file='HumanArt/annotations/validation_humanart_ink_painting.json', bbox_file='data/humanart/HumanArt/person_detection_results/HumanArt_ink_painting_validation_detections_AP_H_56_person.json', test_mode=True, pipeline=[]
)

val_dataset = dict( type='CombinedDataset', metainfo=dict(from_file='configs/base/datasets/coco.py'), datasets=[water_colo_val, oil_val, ink_val], pipeline=val_pipeline, test_mode=True, )

val_dataloader = dict( batch_size=32, num_workers=2, persistent_workers=True, drop_last=False, sampler=dict(type='DefaultSampler', shuffle=False, round_up=False), dataset=val_dataset, )

test_dataloader = val_dataloader

evaluators

val_evaluator = dict( type='MultiDatasetEvaluator', metrics=[ dict( type='CocoMetric', ann_file=data_root + 'HumanArt/annotations/validation_humanart_watercolor.json', ), dict( type='CocoMetric', ann_file=data_root + 'HumanArt/annotations/validation_humanart_oil_painting.json',

    ),
    dict(
        type='CocoMetric',
        ann_file=data_root + 'HumanArt/annotations/validation_humanart_ink_painting.json',
    ),
],
datasets=val_dataset['datasets'],

)

test_evaluator = val_evaluator

Reproduces the problem - command or script

python tools/train.py models/new/model1.py --work-dir models/new/train/ --resume

Reproduces the problem - error message

05/28 00:55:25 - mmengine - INFO -

System environment: sys.platform: win32 Python: 3.8.19 (default, Mar 20 2024, 19:55:45) [MSC v.1916 64 bit (AMD64)] CUDA available: True MUSA available: False numpy_random_seed: 1986547594 GPU 0: NVIDIA RTX 3500 Ada Generation Laptop GPU CUDA_HOME: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4 NVCC: Cuda compilation tools, release 12.4, V12.4.99 GCC: n/a PyTorch: 2.2.2 PyTorch compiling details: PyTorch built with:

C++ Version: 201703
MSVC 192930151
Intel(R) Math Kernel Library Version 2020.0.2 Product Build 20200624 for Intel(R) 64 architecture applications
Intel(R) MKL-DNN v3.3.2 (Git Hash 2dc95a2ad0841e29db8b22fbccaf3e5da7992b01)
OpenMP 2019
LAPACK is enabled (usually provided by MKL)
CPU capability usage: AVX2
CUDA Runtime 12.1
NVCC architecture flags: -gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90
CuDNN 8.8.1 (built against CUDA 12.0)
Magma 2.5.4
Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=8.8.1, CXX_COMPILER=C:/cb/pytorch_1000000000000/work/tmp_bin/sccache-cl.exe, CXX_FLAGS=/DWIN32 /D_WINDOWS /GR /EHsc /Zc:__cplusplus /bigobj /FS /utf-8 -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE /wd4624 /wd4068 /wd4067 /wd4267 /wd4661 /wd4717 /wd4244 /wd4804 /wd4273, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.2.2, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=OFF, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF,

TorchVision: 0.17.2 OpenCV: 4.9.0 MMEngine: 0.10.3

Runtime environment: cudnn_benchmark: False mp_cfg: {'mp_start_method': 'fork', 'opencv_num_threads': 0} dist_cfg: {'backend': 'nccl'} seed: 1986547594 Distributed launcher: none Distributed training: False GPU number: 1

05/28 00:55:29 - mmengine - INFO - Config: auto_scale_lr = dict(base_batch_size=512) backend_args = dict(backend='local') codec = dict( heatmap_size=( 96, 96, ), input_size=( 384, 384, ), sigma=3, type='MSRAHeatmap') custom_hooks = [ dict(type='SyncBuffersHook'), ] data_mode = 'topdown' data_root = 'data/humanart/' dataset_type = 'HumanArtDataset' default_hooks = dict( badcase=dict( badcase_thr=5, enable=False, metric_type='loss', out_dir='badcase', type='BadCaseAnalysisHook'), checkpoint=dict( interval=10, rule='greater', save_best='coco/AP', type='CheckpointHook'), logger=dict(interval=50, type='LoggerHook'), param_scheduler=dict(type='ParamSchedulerHook'), sampler_seed=dict(type='DistSamplerSeedHook'), timer=dict(type='IterTimerHook'), visualization=dict(enable=False, type='PoseVisualizationHook')) default_scope = 'mmpose' env_cfg = dict( cudnn_benchmark=False, dist_cfg=dict(backend='nccl'), mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0)) ink_val = dict( ann_file='HumanArt/annotations/validation_humanart_ink_painting.json', bbox_file= 'data/humanart/HumanArt/person_detection_results/HumanArt_ink_painting_validation_detections_AP_H_56_person.json', data_mode='topdown', data_root='data/humanart/', pipeline=[], test_mode=True, type='HumanArtDataset') launcher = 'none' load_from = None log_level = 'INFO' log_processor = dict( by_epoch=True, num_digits=6, type='LogProcessor', window_size=50) model = dict( backbone=dict( init_cfg=dict( checkpoint='models/hg/model1.pth', prefix='backbone.', type='Pretrained'), num_stacks=1, type='HourglassNet'), data_preprocessor=dict( bgr_to_rgb=True, mean=[ 123.675, 116.28, 103.53, ], std=[ 58.395, 57.12, 57.375, ], type='PoseDataPreprocessor'), head=dict( decoder=dict( heatmap_size=( 96, 96, ), input_size=( 384, 384, ), sigma=3, type='MSRAHeatmap'), deconv_out_channels=None, in_channels=256, loss=dict(type='KeypointMSELoss', use_target_weight=True), num_stages=1, out_channels=17, type='CPMHead'), test_cfg=dict(flip_mode='heatmap', flip_test=True, shift_heatmap=True), type='TopdownPoseEstimator') oil_val = dict( ann_file='HumanArt/annotations/validation_humanart_oil_painting.json', bbox_file= 'data/humanart/HumanArt/person_detection_results/HumanArt_oil_painting_validation_detections_AP_H_56_person.json', data_mode='topdown', data_root='data/humanart/', pipeline=[], test_mode=True, type='HumanArtDataset') optim_wrapper = dict(optimizer=dict(lr=0.0005, type='Adam')) param_scheduler = [ dict( begin=0, by_epoch=False, end=500, start_factor=0.001, type='LinearLR'), dict( begin=0, by_epoch=True, end=210, gamma=0.1, milestones=[ 100, 150, ], type='MultiStepLR'), ] resume = True test_cfg = dict() test_dataloader = dict( batch_size=32, dataset=dict( datasets=[ dict( ann_file= 'HumanArt/annotations/validation_humanart_watercolor.json', bbox_file= 'data/humanart/HumanArt/person_detection_results/HumanArt_watercolor_validation_detections_AP_H_56_person.json', data_mode='topdown', data_root='data/humanart/', pipeline=[], test_mode=True, type='HumanArtDataset'), dict( ann_file= 'HumanArt/annotations/validation_humanart_oil_painting.json', bbox_file= 'data/humanart/HumanArt/person_detection_results/HumanArt_oil_painting_validation_detections_AP_H_56_person.json', data_mode='topdown', data_root='data/humanart/', pipeline=[], test_mode=True, type='HumanArtDataset'), dict( ann_file= 'HumanArt/annotations/validation_humanart_ink_painting.json', bbox_file= 'data/humanart/HumanArt/person_detection_results/HumanArt_ink_painting_validation_detections_AP_H_56_person.json', data_mode='topdown', data_root='data/humanart/', pipeline=[], test_mode=True, type='HumanArtDataset'), ], metainfo=dict(from_file='configs/base/datasets/coco.py'), pipeline=[ dict(type='LoadImage'), dict(type='GetBBoxCenterScale'), dict(input_size=( 384, 384, ), type='TopdownAffine'), dict(type='PackPoseInputs'), ], test_mode=True, type='CombinedDataset'), drop_last=False, num_workers=2, persistent_workers=True, sampler=dict(round_up=False, shuffle=False, type='DefaultSampler')) test_evaluator = dict( datasets=[ dict( ann_file='HumanArt/annotations/validation_humanart_watercolor.json', bbox_file= 'data/humanart/HumanArt/person_detection_results/HumanArt_watercolor_validation_detections_AP_H_56_person.json', data_mode='topdown', data_root='data/humanart/', pipeline=[], test_mode=True, type='HumanArtDataset'), dict( ann_file= 'HumanArt/annotations/validation_humanart_oil_painting.json', bbox_file= 'data/humanart/HumanArt/person_detection_results/HumanArt_oil_painting_validation_detections_AP_H_56_person.json', data_mode='topdown', data_root='data/humanart/', pipeline=[], test_mode=True, type='HumanArtDataset'), dict( ann_file= 'HumanArt/annotations/validation_humanart_ink_painting.json', bbox_file= 'data/humanart/HumanArt/person_detection_results/HumanArt_ink_painting_validation_detections_AP_H_56_person.json', data_mode='topdown', data_root='data/humanart/', pipeline=[], test_mode=True, type='HumanArtDataset'), ], metrics=[ dict( ann_file= 'data/humanart/HumanArt/annotations/validation_humanart_watercolor.json', prefix='water', type='CocoMetric'), dict( ann_file= 'data/humanart/HumanArt/annotations/validation_humanart_oil_painting.json', prefix='oil', type='CocoMetric'), dict( ann_file= 'data/humanart/HumanArt/annotations/validation_humanart_ink_painting.json', prefix='ink', type='CocoMetric'), ], type='MultiDatasetEvaluator') train_cfg = dict(by_epoch=True, max_epochs=210, val_interval=1) train_dataloader = dict( batch_size=32, dataset=dict( datasets=[ dict( ann_file= 'HumanArt/annotations/training_humanart_watercolor.json', data_root='data/humanart/', pipeline=[], type='HumanArtDataset'), dict( ann_file= 'HumanArt/annotations/training_humanart_oil_painting.json', data_root='data/humanart/', pipeline=[], type='HumanArtDataset'), dict( ann_file= 'HumanArt/annotations/training_humanart_ink_painting.json', data_root='data/humanart/', pipeline=[], type='HumanArtDataset'), ], metainfo=dict(from_file='configs/base/datasets/coco.py'), pipeline=[ dict(type='LoadImage'), dict(type='GetBBoxCenterScale'), dict(type='RandomHalfBody'), dict(type='RandomBBoxTransform'), dict(input_size=( 384, 384, ), type='TopdownAffine'), dict( encoder=dict( heatmap_size=( 96, 96, ), input_size=( 384, 384, ), sigma=3, type='MSRAHeatmap'), type='GenerateTarget'), dict(type='PackPoseInputs'), ], test_mode=False, type='CombinedDataset'), num_workers=2, persistent_workers=True, sampler=dict(round_up=False, shuffle=False, type='DefaultSampler')) train_ink = dict( ann_file='HumanArt/annotations/training_humanart_ink_painting.json', data_root='data/humanart/', pipeline=[], type='HumanArtDataset') train_oil = dict( ann_file='HumanArt/annotations/training_humanart_oil_painting.json', data_root='data/humanart/', pipeline=[], type='HumanArtDataset') train_pipeline = [ dict(type='LoadImage'), dict(type='GetBBoxCenterScale'), dict(type='RandomHalfBody'), dict(type='RandomBBoxTransform'), dict(input_size=( 384, 384, ), type='TopdownAffine'), dict( encoder=dict( heatmap_size=( 96, 96, ), input_size=( 384, 384, ), sigma=3, type='MSRAHeatmap'), type='GenerateTarget'), dict(type='PackPoseInputs'), ] train_water = dict( ann_file='HumanArt/annotations/training_humanart_watercolor.json', data_root='data/humanart/', pipeline=[], type='HumanArtDataset') val_cfg = dict() val_dataloader = dict( batch_size=32, dataset=dict( datasets=[ dict( ann_file= 'HumanArt/annotations/validation_humanart_watercolor.json', bbox_file= 'data/humanart/HumanArt/person_detection_results/HumanArt_watercolor_validation_detections_AP_H_56_person.json', data_mode='topdown', data_root='data/humanart/', pipeline=[], test_mode=True, type='HumanArtDataset'), dict( ann_file= 'HumanArt/annotations/validation_humanart_oil_painting.json', bbox_file= 'data/humanart/HumanArt/person_detection_results/HumanArt_oil_painting_validation_detections_AP_H_56_person.json', data_mode='topdown', data_root='data/humanart/', pipeline=[], test_mode=True, type='HumanArtDataset'), dict( ann_file= 'HumanArt/annotations/validation_humanart_ink_painting.json', bbox_file= 'data/humanart/HumanArt/person_detection_results/HumanArt_ink_painting_validation_detections_AP_H_56_person.json', data_mode='topdown', data_root='data/humanart/', pipeline=[], test_mode=True, type='HumanArtDataset'), ], metainfo=dict(from_file='configs/base/datasets/coco.py'), pipeline=[ dict(type='LoadImage'), dict(type='GetBBoxCenterScale'), dict(input_size=( 384, 384, ), type='TopdownAffine'), dict(type='PackPoseInputs'), ], test_mode=True, type='CombinedDataset'), drop_last=False, num_workers=2, persistent_workers=True, sampler=dict(round_up=False, shuffle=False, type='DefaultSampler')) val_dataset = dict( datasets=[ dict( ann_file='HumanArt/annotations/validation_humanart_watercolor.json', bbox_file= 'data/humanart/HumanArt/person_detection_results/HumanArt_watercolor_validation_detections_AP_H_56_person.json', data_mode='topdown', data_root='data/humanart/', pipeline=[], test_mode=True, type='HumanArtDataset'), dict( ann_file= 'HumanArt/annotations/validation_humanart_oil_painting.json', bbox_file= 'data/humanart/HumanArt/person_detection_results/HumanArt_oil_painting_validation_detections_AP_H_56_person.json', data_mode='topdown', data_root='data/humanart/', pipeline=[], test_mode=True, type='HumanArtDataset'), dict( ann_file= 'HumanArt/annotations/validation_humanart_ink_painting.json', bbox_file= 'data/humanart/HumanArt/person_detection_results/HumanArt_ink_painting_validation_detections_AP_H_56_person.json', data_mode='topdown', data_root='data/humanart/', pipeline=[], test_mode=True, type='HumanArtDataset'), ], metainfo=dict(from_file='configs/base/datasets/coco.py'), pipeline=[ dict(type='LoadImage'), dict(type='GetBBoxCenterScale'), dict(input_size=( 384, 384, ), type='TopdownAffine'), dict(type='PackPoseInputs'), ], test_mode=True, type='CombinedDataset') val_evaluator = dict( datasets=[ dict( ann_file='HumanArt/annotations/validation_humanart_watercolor.json', bbox_file= 'data/humanart/HumanArt/person_detection_results/HumanArt_watercolor_validation_detections_AP_H_56_person.json', data_mode='topdown', data_root='data/humanart/', pipeline=[], test_mode=True, type='HumanArtDataset'), dict( ann_file= 'HumanArt/annotations/validation_humanart_oil_painting.json', bbox_file= 'data/humanart/HumanArt/person_detection_results/HumanArt_oil_painting_validation_detections_AP_H_56_person.json', data_mode='topdown', data_root='data/humanart/', pipeline=[], test_mode=True, type='HumanArtDataset'), dict( ann_file= 'HumanArt/annotations/validation_humanart_ink_painting.json', bbox_file= 'data/humanart/HumanArt/person_detection_results/HumanArt_ink_painting_validation_detections_AP_H_56_person.json', data_mode='topdown', data_root='data/humanart/', pipeline=[], test_mode=True, type='HumanArtDataset'), ], metrics=[ dict( ann_file= 'data/humanart/HumanArt/annotations/validation_humanart_watercolor.json', prefix='water', type='CocoMetric'), dict( ann_file= 'data/humanart/HumanArt/annotations/validation_humanart_oil_painting.json', prefix='oil', type='CocoMetric'), dict( ann_file= 'data/humanart/HumanArt/annotations/validation_humanart_ink_painting.json', prefix='ink', type='CocoMetric'), ], type='MultiDatasetEvaluator') val_pipeline = [ dict(type='LoadImage'), dict(type='GetBBoxCenterScale'), dict(input_size=( 384, 384, ), type='TopdownAffine'), dict(type='PackPoseInputs'), ] vis_backends = [ dict(type='LocalVisBackend'), ] visualizer = dict( name='visualizer', type='PoseLocalVisualizer', vis_backends=[ dict(type='LocalVisBackend'), dict(type='TensorboardVisBackend'), ]) water_colo_val = dict( ann_file='HumanArt/annotations/validation_humanart_watercolor.json', bbox_file= 'data/humanart/HumanArt/person_detection_results/HumanArt_watercolor_validation_detections_AP_H_56_person.json', data_mode='topdown', data_root='data/humanart/', pipeline=[], test_mode=True, type='HumanArtDataset') workdir = 'models/new/train/'

05/28 00:55:40 - mmengine - INFO - Distributed training is not used, all SyncBatchNorm (SyncBN) layers in the model will be automatically reverted to BatchNormXd layers if they are used. 05/28 00:55:40 - mmengine - INFO - Hooks will be executed in the following order: before_run: (VERY_HIGH ) RuntimeInfoHook (BELOW_NORMAL) LoggerHook

before_train: (VERY_HIGH ) RuntimeInfoHook (NORMAL ) IterTimerHook (VERY_LOW ) CheckpointHook

before_train_epoch: (VERY_HIGH ) RuntimeInfoHook (NORMAL ) IterTimerHook (NORMAL ) DistSamplerSeedHook

before_train_iter: (VERY_HIGH ) RuntimeInfoHook (NORMAL ) IterTimerHook

after_train_iter: (VERY_HIGH ) RuntimeInfoHook (NORMAL ) IterTimerHook (BELOW_NORMAL) LoggerHook (LOW ) ParamSchedulerHook (VERY_LOW ) CheckpointHook

after_train_epoch: (NORMAL ) IterTimerHook (NORMAL ) SyncBuffersHook (LOW ) ParamSchedulerHook (VERY_LOW ) CheckpointHook

before_val: (VERY_HIGH ) RuntimeInfoHook

before_val_epoch: (NORMAL ) IterTimerHook (NORMAL ) SyncBuffersHook

before_val_iter: (NORMAL ) IterTimerHook

after_val_iter: (NORMAL ) IterTimerHook (NORMAL ) PoseVisualizationHook (BELOW_NORMAL) LoggerHook

after_val_epoch: (VERY_HIGH ) RuntimeInfoHook (NORMAL ) IterTimerHook (BELOW_NORMAL) LoggerHook (LOW ) ParamSchedulerHook (VERY_LOW ) CheckpointHook

after_val: (VERY_HIGH ) RuntimeInfoHook

after_train: (VERY_HIGH ) RuntimeInfoHook (VERY_LOW ) CheckpointHook

before_test: (VERY_HIGH ) RuntimeInfoHook

before_test_epoch: (NORMAL ) IterTimerHook

before_test_iter: (NORMAL ) IterTimerHook

after_test_iter: (NORMAL ) IterTimerHook (NORMAL ) PoseVisualizationHook (NORMAL ) BadCaseAnalysisHook (BELOW_NORMAL) LoggerHook

after_test_epoch: (VERY_HIGH ) RuntimeInfoHook (NORMAL ) IterTimerHook (NORMAL ) BadCaseAnalysisHook (BELOW_NORMAL) LoggerHook

after_test: (VERY_HIGH ) RuntimeInfoHook

after_run: (BELOW_NORMAL) LoggerHook

loading annotations into memory... Done (t=0.07s) creating index... index created! loading annotations into memory... Done (t=0.11s) creating index... index created! loading annotations into memory... Done (t=0.09s) creating index... index created! loading annotations into memory... Done (t=0.01s) creating index... index created! loading annotations into memory... Done (t=0.02s) creating index... index created! loading annotations into memory... Done (t=0.01s) creating index... index created! loading annotations into memory... Done (t=0.01s) creating index... index created! loading annotations into memory... Done (t=0.02s) creating index... index created! loading annotations into memory... Done (t=0.01s) creating index... index created! 05/28 00:55:51 - mmengine - INFO - load backbone. in model from: models/hg/model1.pth Loads checkpoint by local backend from path: models/hg/model1.pth Did not find lastcheckpoint to be resumed. 05/28 00:55:52 - mmengine - INFO - Auto resumed from the latest checkpoint None. 05/28 00:55:52 - mmengine - WARNING - "FileClient" will be deprecated in future. Please use io functions in https://mmengine.readthedocs.io/en/latest/api/fileio.html#file-io 05/28 00:55:52 - mmengine - WARNING - "HardDiskBackend" is the alias of "LocalBackend" and the former will be deprecated in future. 05/28 00:55:52 - mmengine - INFO - Checkpoints will be saved to C:\Users\Me\mmpose\models\new\train. 05/28 00:57:54 - mmengine - INFO - Epoch(train) [1][ 50/487] lr: 4.954910e-05 eta: 2 days, 21:23:05 time: 2.443607 data_time: 0.182674 memory: 10300 loss: 0.002342 loss_kpt: 0.002342 acc_pose: 0.280381 05/28 00:59:58 - mmengine - INFO - Epoch(train) [1][100/487] lr: 9.959920e-05 eta: 2 days, 22:03:22 time: 2.493321 data_time: 0.006862 memory: 10300 loss: 0.001887 loss_kpt: 0.001887 acc_pose: 0.734947 05/28 01:02:09 - mmengine - INFO - Epoch(train) [1][150/487] lr: 1.496493e-04 eta: 2 days, 23:23:07 time: 2.612638 data_time: 0.018283 memory: 10300 loss: 0.001341 loss_kpt: 0.001341 acc_pose: 0.665533 05/28 01:04:30 - mmengine - INFO - Epoch(train) [1][200/487] lr: 1.996994e-04 eta: 3 days, 1:25:52 time: 2.810086 data_time: 0.195886 memory: 10300 loss: 0.001245 loss_kpt: 0.001245 acc_pose: 0.679139 05/28 01:06:52 - mmengine - INFO - Epoch(train) [1][250/487] lr: 2.497495e-04 eta: 3 days, 2:50:52 time: 2.846212 data_time: 0.231822 memory: 10300 loss: 0.001181 loss_kpt: 0.001181 acc_pose: 0.809415 05/28 01:09:19 - mmengine - INFO - Epoch(train) [1][300/487] lr: 2.997996e-04 eta: 3 days, 4:12:30 time: 2.937147 data_time: 0.152995 memory: 10300 loss: 0.001093 loss_kpt: 0.001093 acc_pose: 0.761654 05/28 01:11:45 - mmengine - INFO - Epoch(train) [1][350/487] lr: 3.498497e-04 eta: 3 days, 5:06:45 time: 2.923319 data_time: 0.122444 memory: 10300 loss: 0.001260 loss_kpt: 0.001260 acc_pose: 0.441407 05/28 01:14:07 - mmengine - INFO - Epoch(train) [1][400/487] lr: 3.998998e-04 eta: 3 days, 5:29:32 time: 2.841782 data_time: 0.038016 memory: 10300 loss: 0.001600 loss_kpt: 0.001600 acc_pose: 0.578416 05/28 01:16:30 - mmengine - INFO - Epoch(train) [1][450/487] lr: 4.499499e-04 eta: 3 days, 5:50:05 time: 2.859576 data_time: 0.034563 memory: 10300 loss: 0.001354 loss_kpt: 0.001354 acc_pose: 0.672502 05/28 01:18:06 - mmengine - INFO - Exp name: model1_20240528_005523 05/28 01:19:46 - mmengine - INFO - Epoch(val) [1][ 50/260] eta: 0:06:58 time: 1.993454 data_time: 1.335836 memory: 10300 05/28 01:21:43 - mmengine - INFO - Epoch(val) [1][100/260] eta: 0:05:47 time: 2.351338 data_time: 1.687622 memory: 2921 05/28 01:22:41 - mmengine - INFO - Epoch(val) [1][150/260] eta: 0:03:21 time: 1.156793 data_time: 0.563125 memory: 2921 05/28 01:24:48 - mmengine - INFO - Epoch(val) [1][200/260] eta: 0:02:00 time: 2.538494 data_time: 1.845335 memory: 2921 05/28 01:25:51 - mmengine - INFO - Epoch(val) [1][250/260] eta: 0:00:18 time: 1.263815 data_time: 0.646265 memory: 2921 05/28 01:25:57 - mmengine - WARNING - CocoMetric got empty self.results. Please ensure that the processed results are properly added into self.results in process method.

Traceback (most recent call last): File "tools/train.py", line 162, in main() File "tools/train.py", line 158, in main runner.train() File "C:\Users\Me\miniconda3\envs\openmmlab\lib\site-packages\mmengine\runner\runner.py", line 1777, in train model = self.train_loop.run() # type: ignore File "C:\Users\Me\miniconda3\envs\openmmlab\lib\site-packages\mmengine\runner\loops.py", line 102, in run self.runner.val_loop.run() File "C:\Users\Me\miniconda3\envs\openmmlab\lib\site-packages\mmengine\runner\loops.py", line 374, in run metrics = self.evaluator.evaluate(len(self.dataloader.dataset)) File "C:\Users\Me\miniconda3\envs\openmmlab\lib\site-packages\mmengine\evaluator\evaluator.py", line 79, in evaluate _results = metric.evaluate(size) File "C:\Users\Me\miniconda3\envs\openmmlab\lib\site-packages\mmengine\evaluator\metric.py", line 133, in evaluate _metrics = self.compute_metrics(results) # type: ignore File "c:\users\Me\mmpose\mmpose\evaluation\metrics\coco_metric.py", line 381, in compute_metrics preds, gts = zip(*results) ValueError: not enough values to unpack (expected 2, got 0)

Additional information

Hi,

First of all, thanks for everything you're doing ! It's amazing !

I'm currently using the Hourglass model, I didn't had problem until I try to evaluate across multiples datasets ( In fact, this is not multiple datasets but just some part of HumanArt Dataset) this is made like Coco (points and so on)

I followed the documentation : https://mmpose.readthedocs.io/en/latest/user_guides/train_and_test.html?highlight=Evaluator#evaluating-across-multiple-datasets

And I saw that someone noticed it before me here : https://github.com/open-mmlab/mmpose/pull/2674

And on the forum too : https://github.com/open-mmlab/mmpose/discussions/3024

I would like to know if you could help me to know what's wrong ? It seems like he doesn't have any value or maybe I'm doing something wrong ?

Thanks a lot !

open-mmlab / mmpose

MultiDatasetEvaluator [Bug] #3061