Pose tracking non-functional due to incorrect code

[VisionaryMind]

Currently, it is not possible to use the pose tracking method (in place of bbox), as your demo.py code is pointing to your Linux machine's home directory. For example, running the following command:

python demo.py --vid_file input/test.mp4 --output_folder output/ --tracking_method pose --display --run_smplify

This yields the following error:

Running "ffmpeg -i input/test.mp4 -f image2 -v error /tmp/test_mp4/%06d.png"
Images saved to "/tmp/test_mp4"
Input video number of frames 3905
Traceback (most recent call last):
  File "demo.py", line 414, in <module>
    main(args)
  File "demo.py", line 86, in main
    tracking_results = run_posetracker(video_file, staf_folder=args.staf_dir, display=args.display)
  File "/content/VIBE/lib/utils/pose_tracker.py", line 92, in run_posetracker
    staf_folder=staf_folder
  File "/content/VIBE/lib/utils/pose_tracker.py", line 33, in run_openpose
    os.chdir(staf_folder)
FileNotFoundError: [Errno 2] No such file or directory: '**/home/mkocabas/developments/openposetrack**

It looks like this pose method requires a valid path to the directory for the STAF pose tracking method, but it is instead pointing by default to /home/mkocabas/developments/openposetrack.

I will attempt to manually download and re-run the method.

[VisionaryMind]

I have installed OpenPose on the test system and pointed the staf_dir argument to it. VIBE is able to run the Multi-Person Tracker without a problem, but when it gets to the tracklet step, it fails with:

  File "demo.py", line 405, in <module>
    main(args)
  File "demo.py", line 155, in main
    norm_joints2d.append(nj2d.numpy().reshape(-1, 21, 3))
ValueError: cannot reshape array of size 1200 into shape (21,3)

Would you be able to advise how the code might be modified to allow OpenPose datasets? I wish to use the SMPLify and smoothing functions, which do not work with bbox.

[VisionaryMind]

I was able to find a solution for this problem by changing the batch processing size for the OpenPose joints from 21 to 25. Here is the change at line 155:

for batch in dataloader:
    if has_keypoints:
        batch, nj2d = batch
        norm_joints2d.append(nj2d.numpy().reshape(-1, 25, 3))

Now, there is another problem, attempting to convert the resulting PKL to FBX:

Input path: D:\mocap\VIBE\output/20sdance_short/vibe_output.pkl
Output path: D:\mocap\VIBE\output/20sdance_short/fbx_output.fbx
Processing: D:\mocap\VIBE\output/20sdance_short/vibe_output.pkl
Traceback (most recent call last):
  File "lib/utils/fbx_output.py", line 320, in <module>
    person_id=args.person_id
  File "lib/utils/fbx_output.py", line 166, in process_poses
    poses = data[person_id]['pose']
KeyError: 1
Error: Not freed memory blocks: 8, total unfreed memory 0.010223 MB

Not sure how the PKL is different now with the OpenPose / SMPLified output. Apparently, it cannot find a person ID.

[VisionaryMind]

It would appear that when using OpenPose, the first person ID begins at -1. The FBX import can be modified either directly to accommodate this problem or the original demo.py and submodules can be changed instead. For now, to get it working, I changed line 154:

def process_poses(
        input_path,
        gender,
        fps_source,
        fps_target,
        start_origin,
        person_id=-1,
):

If there will be time, I will take a look to see where in the processing change this negative ID number is being injected.

[mkocabas]

Per explained in the demo readme, you need to install STAF https://github.com/soulslicer/STAF/tree/staf to enable pose tracking. After installation you need to give the path to your version to run it. OpenPose is a per frame tracking method and it doesn't work when there are multiple people.

[VisionaryMind]

This issue was closed prematurely. All the problems stated above exist with the post tracking installed from the STAF repository, using a single human figure. That said, I have noticed that the OpenPose / STAF output is considerably less accurate than bbox.