It implements a mostly functional prototype to automatically generate labels for pose estimation using CoTracker, and works directly as an addition to the plugin with self-sufficient GUI/widgets.
Concept :
Simply by labeling the first frame, you can get up to 150 frames of cleanly labeled data, with keypoints for all user-defined body parts.
When predictions start to worsen, you can directly move to the problematic frame and correct the generated keypoints as usual in napari, and rerun the model starting from that frame in order to quickly generate better quality keypoints for the following frames.
Missing functionalities before it is really feature-complete and usable :
[ ] Fix critical saving/loading issues, mostly when retracking (reshaping errors, likely from inconsistent saving)
[ ] A redesign of the user interface to present newly generated keypoints in a more centralized and easy-to-use manner (no redundant keypoint layer selection, merging results in a single layer, better feedback/metrics, etc)
[ ] Slightly improved logic when yielding results, perhaps updating the results layer as the model goes
[ ] Remove the dependency on CoTracker as a Git submodule : an option would be to use ONNX runtime to run the model on CPU/GPU
This code was written as part of the Lemanic Life Sciences Hackathon at EPFL, the 26th and 27th of April 2024.
It implements a mostly functional prototype to automatically generate labels for pose estimation using CoTracker, and works directly as an addition to the plugin with self-sufficient GUI/widgets.
Concept :
Simply by labeling the first frame, you can get up to 150 frames of cleanly labeled data, with keypoints for all user-defined body parts. When predictions start to worsen, you can directly move to the problematic frame and correct the generated keypoints as usual in napari, and rerun the model starting from that frame in order to quickly generate better quality keypoints for the following frames.
Missing functionalities before it is really feature-complete and usable :
Team members :
Thanks everyone !