search

openxrlab / xrmocap

OpenXRLab Multi-view Motion Capture Toolbox and Benchmark
https://xrmocap.readthedocs.io/
Other
343 stars 42 forks source link

Reproduction for 4DAG #164

Open Charrrrrlie opened 2 weeks ago

Charrrrrlie commented 2 weeks ago

Hello, I am trying to evaluate 4DAG in its benchmark seq2.

I use the following command to execute in the provided docker.

python tools/mview_mperson_evaluation.py  --enable_log_file  --evaluation_config configs/fourdag/fourdag_config/eval_keypoints3d_seq2.py

Firstly, I got an error:

Traceback (most recent call last):
  File "tools/mview_mperson_evaluation.py", line 45, in <module>
    main(args)
  File "tools/mview_mperson_evaluation.py", line 24, in main
    evaluation = build_evaluation(evaluation_config)
  File "/data/yangyuchen/Projects/xrmocap/xrmocap/core/evaluation/builder.py", line 18, in build_evaluation
    return EVALUATION.build(cfg)
  File "/opt/miniconda/envs/openxrlab/lib/python3.8/site-packages/mmcv/utils/registry.py", line 237, in build
    return self.build_func(*args, **kwargs, registry=self)
  File "/opt/miniconda/envs/openxrlab/lib/python3.8/site-packages/mmcv/utils/registry.py", line 72, in build_from_cfg
    raise type(e)(f'{obj_cls.__name__}: {e}')
TypeError: BottomUpAssociationEvaluation: MviewMpersonDataVisualization: __init__() got an unexpected keyword argument 'resolution'

It seems to come from the out-of-date config in this line. Therefore I comment it out and the whole process can be executed.

However, there is a gap between the final performance and the reported one in the [readme].(https://github.com/openxrlab/xrmocap/blob/main/configs/fourdag/README.md#fourdag)

The log file is

2024-09-09 06:56:28,681 - __main__ - INFO - Left_Shoulder_To_Right_Shoulder is not selected!
2024-09-09 06:56:28,682 - __main__ - INFO - Left_Shoulder_To_Left_Hip_Extra is not selected!
2024-09-09 06:56:28,682 - __main__ - INFO - Right_Shoulder_To_Right_Hip_Extra is not selected!
2024-09-09 06:56:28,682 - __main__ - INFO - Right_Hip_Extra_To_Left_Hip_Extra is not selected!
2024-09-09 06:56:28,970 - __main__ - INFO - Detailed table for PCPMetric
+-----------------+---------+---------+---------+---------+---------+
|    Bone Group   | Actor 0 | Actor 1 | Actor 2 | Actor 3 | Average |
+-----------------+---------+---------+---------+---------+---------+
|  left_lower_leg |  81.65  |  89.76  |   nan   |   nan   |   nan   |
| right_lower_leg |  79.60  |  71.70  |   nan   |   nan   |   nan   |
|  left_upperarm  |  69.18  |  54.33  |   nan   |   nan   |   nan   |
|  right_upperarm |  79.01  |  68.96  |   nan   |   nan   |   nan   |
|   left_forearm  |  67.64  |  26.18  |   nan   |   nan   |   nan   |
|  right_forearm  |  66.29  |  18.35  |   nan   |   nan   |   nan   |
|    left_thigh   |  87.64  |  26.22  |   nan   |   nan   |   nan   |
|   right_thigh   |  79.09  |  38.57  |   nan   |   nan   |   nan   |
|      total      |  76.26  |  49.26  |   nan   |   nan   |   nan   |
+-----------------+---------+---------+---------+---------+---------+
2024-09-09 06:56:29,162 - __main__ - INFO - 
+------------------------------+--------+
|         Metric name          | Value  |
+------------------------------+--------+
|      mpjpe: mpjpe_mean       | 157.21 |
|       mpjpe: mpjpe_std       | 125.83 |
|   pa_mpjpe: pa_mpjpe_mean    | 125.24 |
|    pa_mpjpe: pa_mpjpe_std    | 82.37  |
|     pcp: pcp_total_mean      |  nan   |
|         pck: pck@100         | 46.22  |
|         pck: pck@200         | 86.62  |
| precision_recall: recall@500 | 97.04  |
+------------------------------+--------+
2024-09-09 06:56:29,162 - __main__ - WARNING - Overwriting ./output/fourdag/fourdag_fourdag_19_FourDAGOptimization/ and its files.
2024-09-09 06:56:29,173 - __main__ - INFO - Visualizing predict3d data for scene 0 view 0
2024-09-09 07:03:10,178 - __main__ - INFO - Visualizing predict3d data for scene 0 view 1
2024-09-09 07:09:50,608 - __main__ - INFO - Visualizing predict3d data for scene 0 view 2
2024-09-09 07:16:33,984 - __main__ - INFO - Visualizing predict3d data for scene 0 view 3
2024-09-09 07:23:23,581 - __main__ - INFO - Visualizing predict3d data for scene 0 view 4
2024-09-09 07:30:10,331 - __main__ - INFO - Visualizing predict3d data for scene 0 view 5
2024-09-09 07:36:58,682 - __main__ - INFO - Running "ffmpeg -y -f rawvideo -pix_fmt bgr24 -s 1104x736 -r 30.0 -loglevel error -threads 1 -i - -vcodec libx264 -r 30.0 -an ./output/fourdag/fourdag_fourdag_19_FourDAGOptimization/scene_0/predict3d_project_AIO.mp4"
2024-09-09 07:37:09,425 - __main__ - INFO - Visualizing perception 2D data for scene 0 view 0
2024-09-09 07:44:01,877 - __main__ - INFO - Visualizing perception 2D data for scene 0 view 1

Do you know what went wrong?

Thank you for your time and attention.

github-actions[bot] commented 2 weeks ago

Hi @Charrrrrlie, welcome to commit your first issue! 你好 @Charrrrrlie，非常欢迎首次提交你的问题！

Charrrrrlie commented 2 weeks ago

Besides, I noticed the visualization videos have no joints drawn and the results in 'scene0_associate_keypoints2d.npy' seem in the wrong range (>1e03 while the input image is in (368, 368)).

>>> import numpy as np
>>> data=np.load('scene0_associate_keypoints2d.npy')
>>> data.shape
(2735, 2, 6, 19, 3)
>>> data[0][0][0]
array([[1.11271643e+03, 1.11282910e+03, 7.75416970e-01],
       [1.10170972e+03, 8.67035461e+02, 9.00384009e-01],
       [1.05169531e+03, 1.11828638e+03, 7.92052984e-01],
       [1.17399756e+03, 1.11821265e+03, 7.75319993e-01],
       [1.14057812e+03, 7.33562927e+02, 9.68859971e-01],
       [1.02350208e+03, 8.67381470e+02, 8.72323990e-01],
       [1.18556506e+03, 8.50663574e+02, 8.55180979e-01],
       [1.05166663e+03, 1.30793274e+03, 8.65860999e-01],
       [1.19631995e+03, 1.30230957e+03, 8.17400992e-01],
       [1.05715674e+03, 7.38793030e+02, 9.09249008e-01],
       [0.00000000e+00, 0.00000000e+00, 0.00000000e+00],
       [9.51032104e+02, 9.95484741e+02, 8.78517985e-01],
       [1.27425330e+03, 9.61897583e+02, 9.32591021e-01],
       [1.05167896e+03, 1.48639636e+03, 8.70796025e-01],
       [1.13518225e+03, 1.44758105e+03, 9.48741019e-01],
       [9.67754028e+02, 1.10174255e+03, 8.54382992e-01],
       [1.26316687e+03, 1.05713013e+03, 8.08005989e-01],
       [1.16292114e+03, 1.52544275e+03, 8.27202976e-01],
       [1.05164001e+03, 1.54247791e+03, 8.22476983e-01]])

The results in 'scene0_pred_keypoints3d.npz' seem correct. (And that's why metrics like MPJPE are not very terrible?)

>>> import numpy as np
>>> data = np.load('scene0_pred_keypoints3d.npz')
>>> data.files
['convention', 'keypoints', 'mask']
>>> data['keypoints'].shape
(2735, 2, 19, 4)
>>> data['keypoints'][0,0]
array([[ 0.44035739,  0.81138122, -0.18822792,  1.        ],
       [ 0.44342533,  1.27131438, -0.13195667,  1.        ],
       [ 0.32768041,  0.80619949, -0.17555375,  1.        ],
       [ 0.54508728,  0.81602764, -0.20119405,  1.        ],
       [ 0.49280715,  1.48571718,  0.00272803,  1.        ],
       [ 0.29264435,  1.26126456, -0.11988235,  1.        ],
       [ 0.58975613,  1.28480649, -0.15804569,  1.        ],
       [ 0.34666932,  0.41698083, -0.11620992,  1.        ],
       [ 0.58429468,  0.4399837 , -0.11577299,  1.        ],
       [ 0.35547218,  1.46291053, -0.06224353,  1.        ],
       [ 0.50029427,  1.4707303 , -0.132844  ,  1.        ],
       [ 0.14133491,  1.03275061, -0.13060124,  1.        ],
       [ 0.75000304,  1.06469333, -0.20401694,  1.        ],
       [ 0.34222731,  0.02327326, -0.2378833 ,  1.        ],
       [ 0.5073328 ,  0.05773893, -0.29119289,  1.        ],
       [ 0.17784889,  0.84062225,  0.01113267,  1.        ],
       [ 0.68762159,  0.92540288, -0.08345005,  1.        ],
       [ 0.56086171, -0.02101712, -0.14138444,  1.        ],
       [ 0.34968144, -0.01838641, -0.05645154,  1.        ]])