davidpagnon
commented
10 months ago DETAILED TO-DO LIST:
Studd will be added as needs arise, and removed as they are met.
GUI: ▢ Simple tkinter interface (or Kivy if we want something nice and portable, or Google Colab, maybe Streamlit). Maybe have a look at the Deeplabcut GUI for inspiration. See Sports2D Demo for OpenPose and Python package installation on Google Drive). Simply load, edit, and save Config.toml files. Only display relevant options (e.g., do not display options for calculating calibration if conversion is chosen). ▢ App or webapp (e.g., with Napari ).
Tutorials and Documentation: ▢ Make video tutorials. ▢ Multiple pages for documentation would be clearer that a huge single page. Use Sphinx, MkDocs, or (maybe better) github.io.
Pose: ▢ Implement RTMPoseW3D and monocular 3D kinematics ▢ Support MMPose, SLEAP, etc. ▢ Directly reading from DeepLabCut .csv or .h5 files instead of converting to .json (triangulation, person association, calibration, synchronization...) ▢ GUI help for DeepLabCut model creation.
Calibration: ▢ Currently, clicking the last point confirms the whole selection and switches to the next camera, which can be annoying in case of missclick. Once the reprojection is displayed, show a pop-up message and ask if the user is satisfied. If yes, switch to the next camera; if No, do the clicking again. ▢ Enable calibration with vertical checkerboard. ▢ Calibration of moving cameras: Once object points have been detected or clicked once, track them. Propose to click again when they are lost. ▢ Calibrate cameras by pairs and compute average extrinsic calibration with aniposelib. ▢ Alternatively, self-calibrate with OpenPose keypoints. Set world reference frame in the end. ▢ Calculate calibration with points rather than board. (1) SBA calibration with wand (cf Argus, see converter here). Set world reference frame in the end. ▢ Convert fSpy calibration calibration (interesting and fun approach, but not sure it is accurate enough).
Triangulation: ▢ Solve limb swapping. Try triangulating with opposite side if reprojection error too large (favored approach). Alternatively, ignore right and left sides, use RANSAC or SDS triangulation, and then choose right or left by majority voting. More confidence can be given to cameras whose plane is the most coplanar to the right/left line. See here. ▢ Undistort 2D points before triangulating (and distort them before computing reprojection error). See draft L468 in triangulation.py. ▢ Pre-compile weighted_triangulation and reprojection with @jit(nopython=True, parallel=True) for faster execution. ▢ Track hands and face (won't be taken into account in OpenSim at this stage).
Filtering: ▢ Implement SmoothNet
OpenSim: ▢ Integrate OpenSim in Pose2Sim. ▢ Do not require a separate scaling trial: scale on the 10% slowest frames of the moving trial instead, or taken median scaling value. ▢ OpenSim: Implement optimal fixed-interval Kalman smoothing for inverse kinematics (this OpenSim fork), or Biorbd)
Others: ▢ Real-time: Run Pose estimation, Person association, Triangulation, Kalman filter, IK frame by frame (instead of running each step for all frames) ▢ Conda package ▢ Docker image ▢ Run from command line via click or typer ▢ Utilities: Export other data from c3d files into .mot or .sto files (angles, powers, forces, moments, GRF, EMG...)
Bugs: ▢ common.py, class plotWindow(). Python crashes after a few runs of Pose2Sim.filtering() when display_figures=true. See https://github.com/superjax/plotWindow/issues/7.
hunminkim98
carlosedubarreto
ANaaim
timungereth
Dear all,
I am now doing a relatively unrelated post-doc, and am working on Pose2Sim only on the side. There are tons of features that I think could be useful, and that I cannot find time to implement. If anyone wants to contribute, I'll be more than happy to oblige!
The list is at the end of the Readme file, but I'll put it here for the sake of clarity.
MAIN PROJECTS: ☑ Blender visualizer add-on ☑ Batch processing ☑ Multiple persons triangulation ☑ Synchronization ☑ Integrate pose estimation ▢ Integrate monocular 3D kinematics with RTMPoseW3D ▢ Integrate scaling and inverse kinematics ▢ Graphical User Interface ▢ Self-calibration based on keypoint detection ▢ Video tutorials, documentation on github.io ▢ Calibration of moving cameras
I have a clear idea of how to do each of these, so if you are not sure of the details or are not sure whether you can use or develop the skills, I can definitely tell you more about it! Here is an invite to a Discord server if you are interested in discussing it (no commitment at this stage 😛).
If you want to modify the code, please please follow this guide on how to fork, modify, and push code, and submit a pull request. I would appreciate it if you provided as much useful information as possible about how you modified the code, and a rationale for why you're making this pull request. Please also specify on which operating system and on which Python version you have tested the code.