Pose Estimation Shape Mismatch

[161siegels]

I have been running into an error with multi person pose estimation. On my example, i keep getting shape mismatch errors. It seems that one of the frames is thrown out when calculating track_ids_last_frame but keypoints has every frame. I am using HALPE_26 and Yolox. Here is where the code errors, followed by the error message. Any help is welcome!

keypoints_filled[pose_tracker.track_ids_last_frame] = keypoints

File "/Users/161siegels/miniconda3/envs/Pose2Sim/lib/python3.9/site-packages/Pose2Sim/poseEstimation.py", line 168, in process_video keypoints_filled[pose_tracker.track_ids_last_frame] = keypoints ValueError: shape mismatch: value array of shape (13,26,2) could not be broadcast to indexing result of shape (12,26,2)

If I edit this line to be: keypoints_filled[pose_tracker.track_ids_last_frame] = keypoints[pose_tracker.track_ids_last_frame] then it would work but I am unclear if this is problematic

[davidpagnon]

Hi, thanks for sharing! Could you try to set tracking to false in your config file ? In the meantime, I'll try to pin the issue and fix it.

And if I can't find it and that's possible for you, could you send your data to contact@david-pagnon.com?

[161siegels]

It works if tracking is false but we want tracking (my understanding is this allows for reidentification of people between frames).

Unfortunately I cannot send my data, but I appreciate you looking into this for me! I tried stepping through the code and I couldn't quite put my finger on it but my hunch is that it is some rounding error.

If this helps, when I reduce the frame rate slightly (from the default 60 fps), i get a different mismatch error

(11,26,2) could not be broadcast to indexing result of shape (12,26,2))

[davidpagnon]

It works if tracking is false but we want tracking (my understanding is this allows for reidentification of people between frames).

The reidentification of people between frames is done at the triangulation stage, so it should not matter. At this stage, tracking is only useful for synchronization if you need it, but I'll try to fix it anyway.

Could you send me the whole error message? (both, actually)?

[161siegels]

Ah thanks for the clarification! Really appreciate the quick responses too!! This repo is awesome. When you say synchronization if needed, do you mean that it will need it for synchronization if the videos are not perfectly aligned? Or is this an additional aid for synchronization that you think is often not necessary.

Just trying to gauge how necessary you think this it is for us to allow tracking (I understand this may be difficult without seeing the data).

Here is the full error message:

Traceback (most recent call last): File "/Users/xxx/miniconda3/envs/Pose2Sim/lib/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/Users/xxx/miniconda3/envs/Pose2Sim/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/Users/xxx/.vscode/extensions/ms-python.debugpy-2024.10.0-darwin-arm64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/main.py", line 39, in cli.main() File "/Users/xxx/.vscode/extensions/ms-python.debugpy-2024.10.0-darwin-arm64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 430, in main run() File "/Users/xxx/.vscode/extensions/ms-python.debugpy-2024.10.0-darwin-arm64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 284, in run_file runpy.run_path(target, run_name="main") File "/Users/xxx/.vscode/extensions/ms-python.debugpy-2024.10.0-darwin-arm64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 321, in run_path return _run_module_code(code, init_globals, run_name, File "/Users/xxx/.vscode/extensions/ms-python.debugpy-2024.10.0-darwin-arm64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 135, in _run_module_code _run_code(code, mod_globals, init_globals, File "/Users/xxx/.vscode/extensions/ms-python.debugpy-2024.10.0-darwin-arm64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 124, in _run_code exec(code, run_globals) File "/Users/xxx/Documents/Trial/main.py", line 3, in Pose2Sim.poseEstimation() File "/Users/xxx/miniconda3/envs/Pose2Sim/lib/python3.9/site-packages/Pose2Sim/Pose2Sim.py", line 237, in poseEstimation rtm_estimator(config_dict) File "/Users/xxx/miniconda3/envs/Pose2Sim/lib/python3.9/site-packages/Pose2Sim/poseEstimation.py", line 433, in rtm_estimator process_video(video_path, pose_tracker, tracking, output_format, save_video, save_images, display_detection, frame_range) File "/Users/xxx/miniconda3/envs/Pose2Sim/lib/python3.9/site-packages/Pose2Sim/poseEstimation.py", line 168, in process_video keypoints_filled[pose_tracker.track_ids_last_frame] = keypoints ValueError: shape mismatch: value array of shape (13,26,2) could not be broadcast to indexing result of shape (12,26,2)

[davidpagnon]

Thanks for the appreciation! And thanks for the error message as well, I'll have a look in the next few days.

It depends on what you are trying to do:

• if your cameras are already synchronized, you do not need to bother.
• If not, and if you'd rather not use a sound clap or a visual flash (or a GPS signal, or anything like that), Pose2Sim synchronization relies on the correlation of vertical speeds to find the best time shift. If the person is alone in the scene, you do not need to bother either
• if there are some people in the background and you can't isolate a moment with a single person / a fast motion / the person around the center of the scene, then you may need to make sure you correlate the right person from camera to camera. In that case, it is good to have the same ID across frame, and that's where tracking may be good to activate.

[161siegels]

Makes sense! I think we would want tracking for our use case then. Thanks again

[davidpagnon]

I'll write a fix and keep you updated as soon as the new version is released then

[davidpagnon]

The update is uploaded, you can upgrade your pose2sim version with pip install pose2sim -U.

Just to keep you informed, I plan to integrate the OpenSim stage within Pose2Sim by the end of the week :)

[davidpagnon]

Sorry, I'm just realizing that I introduced another bug, I'm currently trying to fix it! 😅

[161siegels]

No worries! May I ask what the bug is? I am having extrinsic calibration issues with one of my cameras. Doubt this would affect it but just curious

[davidpagnon]

It was a big on synchronization but I just solved it (although I did not push the release yet). We chose to synchronize on the person with the highest confidence to cope with bad detections, but the person in the background of the Single person demo is detected with a high confidence so that did not work great. Instead, I decided to synchronize on the person with the largest bounding box.

I don't think this has anything to do with your extrinsic calibration problem but tell me if you can't solve it!

[161siegels]

Sounds good! I think I solved the extrinsic calibration when using 2 of our 3 cameras (the reprojection error is low and the picture looks great). However, I am still having issues with the resulting 3d output having nonsensical coordinates. I will provide a bit about my setup + issues... any help would be greatly appreciated!

Setup

• Using 2 cameras/views - we tried a 3rd but the camera was different than the other two and this caused ext calib issues
• Multiperson analysis - There are many people in the frame. Maybe around 20. Some are moving more than others
• The video is are around 5-10 seconds

Issues

• We really only care about a few people but it is very difficult to specify the people of interest. It seems that we can only include height and mass?
• I would be perfectly happy with 3d coordinates for everyone in the frame. Then we could do the filtering on our end, but this does not seem to be an option even when I comment out the height/mass specs in the config
• The synchronization seems to necessitate a specification for time_range_around_maxspeed. However, I think I do not want to specify this since we care about multiple people in the frame throughout its duration
• Lastly, it would be awesome if a future version could save the extrinsic parameters (like we do for intrinsic) so I don't have to click the points every time. Maybe there is a mechanism for this currently of which I am unaware?

Thanks again for all your help! I understand this is a bit difficult without our data but I appreciate your patience and hard work

[davidpagnon]

From what you are telling me, the main difficult points seem to be:

• Calibration; it should not be a problem to use the third camera
• Synchronization: you may need to do it manually after having extracted all the frames, with ffmpeg for example
• Number of people: I have never tried with as many as 20 people

Issues:

• using a third camera, even of a different kind, should not pose any problem since you calibrate their intrinsic properties. Unless you use a camera with large distortions (fisheye for example): I did not find time to correct them as well as I could.
• Participant heights and masses are only used for marker augmentation (that does not always improve the results, so you could just skip it), and inverse kinematics (which is fully automatic and integrated in Pose2Sim since today). But to be honest, I'm not sure I understand this question
• I have never tried with as many as 20 people. This will be tricky, especially with few cameras, and even more if people are entering or exiting the frame. I think it would be best to let the algorithm triangulate everything it can, and then remove the people you are not interested in
• synchronization with multiple persons is very tricky for now, unless you synchronize your videos/images yourself based on a clear event (foot strike, flash, sound...). We are working on the possibility of clicking on the person we want to synchronize on, but it is still in development.
• You can already choose to not calculate the intrinsics if it has already been done (overwrite_intrinsics = false in Config.toml) or to not calculate the extrinsic parameters (calculate_extrinsics = false)

Not sure I correctly answered your comments so feel free to ask again!

[161siegels]

Thank you so much for the detailed response! I think you answered most of my questions but I have a few follow ups.

1. Using the third (different) camera - hmm, i will keep playing around with it. I can't seem to get ext to work for some reason
2. Heights and masses - makes sense, thanks for this
3. 20 people - yes, that is exactly what I want to do. Get coordinates for all and then remove people myself. However, it only seems to give me 3d files for 2 people. I just get P1 files and P2 files. How can I specify otherwise?
4. Synchronization - yes, I realized this morning that this was an issue. I am going to try to synchronize the videos with ffmpeg
5. Extrinsic Calculation - Yes, but if I specify calculate_extrinsics=false, will it use my last calculation? That was not clear to me as it is with overwrite_extrinsics. I want to keep the extrinsic calculation from a previous calibration

[davidpagnon]

1. What is your problem? Do you have high distortions on this third camera?
2. 20 people with 2 cameras is extremely challenging. You'll have to see if a good synchronization helps, otherwise you may need more cameras!
3. It should not overwrite your last extrinsics. If you are afraid of it, just copy your last calibration file and give it the name calibration.toml.old. But How could intrinsic parameters change and not the extrinsic ones?

[161siegels]

1. The problem is the reprojection points are very far from the points I click when i calibrate via 'scene'
2. Sounds good, I will try with better synchronization and go from there
3. Ok so it does save the extrinsic. I wasn't 100% sure. My intrinsic and extrinsic are not changing. I set overwrite_intrinsic to false but did not realize that setting calculate_extrinsic to false would also grab the old ext calibration. My misunderstanding. Thanks!

[davidpagnon]

Is it possible to send me just the folder with your calibration images/videos (intrinsic and extrinsic) so that I can check?

[davidpagnon]

This issue has been handled by email, and is consequently closed.