[Bug] The 3d pose is unstable.

unrealMJ commented 1 year ago

Prerequisite

[X] I have searched Issues and Discussions but cannot get the expected help.
[X] The bug has not been fixed in the latest version(https://github.com/open-mmlab/mmpose).

Environment

OrderedDict([('sys.platform', 'linux'), ('Python', '3.10.12 (main, Jul 5 2023, 18:54:27) [GCC 11.2.0]'), ('CUDA available', False), ('numpy_random_seed', 2147483648), ('GCC', 'gcc (GCC) 5.4.0'), ('PyTorch', '2.0.1+cu118'), ('PyTorch compiling details', 'PyTorch built with:\n - GCC 9.3\n - C++ Version: 201703\n - Intel(R) oneAPI Math Kernel Library Version 2022.2-Product Build 20220804 for Intel(R) 64 architecture applications\n - Intel(R) MKL-DNN v2.7.3 (Git Hash 6dbeffbae1f23cbbeae17adb7b5b13f1f37c080e)\n - OpenMP 201511 (a.k.a. OpenMP 4.5)\n - LAPACK is enabled (usually provided by MKL)\n - NNPACK is enabled\n - CPU capability usage: AVX2\n - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.8, CUDNN_VERSION=8.7.0, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=2.0.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, \n'), ('TorchVision', '0.15.2+cu118'), ('OpenCV', '4.8.0'), ('MMEngine', '0.8.2'), ('MMPose', '1.1.0+ec30ee1')])

mmagic 1.0.2.dev0
mmcv 2.0.1 mmdet 3.1.0 mmengine 0.8.2 mmpose 1.1.0
timm 0.9.2

Reproduces the problem - code sample

None

Reproduces the problem - command or script

python  demo/body3d_pose_lifter_demo.py  \
projects/rtmpose/rtmdet/person/rtmdet_m_640-8xb32_coco-person.py \
https://download.openmmlab.com/mmpose/v1/projects/rtmpose/rtmdet_m_8xb32-100e_coco-obj365-person-235e8209.pth \
projects/rtmpose/rtmpose/body_2d_keypoint/rtmpose-x_8xb256-700e_coco-384x288.py \
https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-x_simcc-body7_pt-body7_700e-384x288-71d7b7e9_20230629.pth \
configs/body_3d_keypoint/pose_lift/h36m/pose-lift_motionbert-243frm_8xb32-240e_h36m.py \
https://download.openmmlab.com/mmpose/v1/body_3d_keypoint/pose_lift/h36m/motionbert_ft_h36m-d80af323_20230531.pth \
--input ./demo5.mp4 \
--output-root  vis_results  \

Reproduces the problem - error message

None

Additional information

The 2d pose is stable, but 3d pose is unstable, especially the person-leg.

Hope to get stable 3d pose!

demo4

Tau-J commented 1 year ago

Hi @unrealMJ , the 3d pose model is only trained on h36m, which results in limited capability of generalization. If you want to obtain better performance, you have to train it on larger dataset by yourself.

LareinaM commented 1 year ago

Hi @unrealMJ , this might be due to the limited generality of the model. Have you tried the demo with other models (e.g., VideoPose3d) instead?

unrealMJ commented 1 year ago

Hi @unrealMJ , this might be due to the limited generality of the model. Have you tried the demo with other models (e.g., VideoPose3d) instead?

Hi~, thanks for your advice, I tried all 3d pose methods that mmpose supported including motionbert, videoposed3d, simple baseline, and find the simple baseline performs better. I notice the simple baseline is a image-based 3d pose method in the doc, but outperforms other video-based methods. Here is a result.

ezgif-1-c91a916a7c

aixiaodewugege commented 1 year ago

我觉得是因为 motionbert 是用 Halpe dataset 训练的，从Halpe26 转为 17个点的，你使用的2d pose 格式不一样，转换的时候出了问题

open-mmlab / mmpose