Starting point of integrating RTMPose into Pose2Sim.

perfanalytics / pose2sim

Markerless kinematics with any cameras — From 2D Pose estimation to 3D OpenSim motion

https://perfanalytics.github.io/pose2sim/

BSD 3-Clause "New" or "Revised" License

270 stars 50 forks source link

Starting point of integrating RTMPose into Pose2Sim. #111

Closed hunminkim98 closed 4 months ago

hunminkim98 commented 5 months ago

Hello, here is a request for PoseEstimator!

The name of the folder that includes raw video is set to 'pose_raw' I haven't tested with images yet, but videos should work. For now, I'll keep testing and improving it!

Thanks!

davidpagnon commented 5 months ago

Awesome, thank you!

It does not pass the tests but it's because the server does not have cuda available, so we should edit device='cpu' in tests.py.
Once we are sure that this will work on any platform, we can remove the json files and just leave the videos so that the user can do run pose estimation directly from the videos
Instead of copy-pasting rtmlib, I was simply thinking of adding these to install_requires in setup.cfg:
```
rtmlib
onnxruntime-gpu
openvino
```
(we also need to remove opencv-python, as rtmlib installs it too and installing it twice sometimes leads to errors)

davidpagnon commented 5 months ago

I'll switch to discord for further discussion :)

hunminkim98 commented 4 months ago

This is the same as a Discord post.

It now supports the following: Automatically download any selected models. mode = 'performance', 'balanced', 'lightweight' skeleton_type = 'body', 'body_and_foot', 'fullbody' install rtmlib as part of Pose2Sim's installation process. use rtmlib's functions in Pose2Sim's PoseEstimation.py. OpenPose style in all models except halpe26(alreay has neck keypoint). However, it is quite slow. Inference with video and images Save the result as jsons Save the result as video Realtime-visualization

I think after some final preparations for integration (cleanup of config.toml and other structural optimizations), raw video to opensim is now ready.

Additionally, rtmlib seems to support webcam, could this be useful for Pose2Sim?

davidpagnon commented 4 months ago

I just merged it into the test branch and I'll test it now!