feature request: dockable kinematics with updating cursor

[jschoch]

Hello, thanks for this amazing software!

I was wondering if you could dock the kinematics graph below the video playback. Even better if it had a cursor that showed the tracked point values based on the current frame of video. An example would be tracking speed and acceleration in a golf swing. Your hand speed should peak prior to P6 position and you could easily see exactly where hand speed peeked by playing the video and stopping when the cursor on the graph reached it's peak. I can do a mockup if you open to doing this. An example of the cursor would be google finance stock price, as you move the mouse the label updates per the X coordinate mapping to the tick value.

P.S. if this is doable via a script or some other way maybe I can try to make a PR.

[JoanCharmant]

I've been thinking about this. Currently I'm deep into a refactoring of the tracking system and it's going to take some time, but part of this refactoring is going in a similar direction, it's about moving the modal dialog for tracking configuration to the right-hand-side panel (same for the style options).

I'm not sure if kinematic plots should go under or to that same right side panel. Under is better for the timeline aspect. Right is better to give it more room for options. Another idea was to have a tool as an insert onto the main video but that's more complicated in terms of interaction. In any case I agree with the general idea of having these plots directly visible and with a hairline synchronized with the video play head. But need to think about this more. And the current refactoring will change some things for how tracked drawings are providing their kinematics values there.

[jschoch]

since openpose is getting long in the teeth have you considered something like yolo-nas-pose?

I've been playing with hand detection and shaft, clubhead detection and I think it may be good enough to place markers. I don't know how it could integrate into your playback stuff though. Does the existing openpose integration allow for keypoint tracking, it would be great to be able to simply post process with the pose data and have the speed/acceleration calculation/playback display working. Perhaps it is easier to just make my own playback viewer?

[JoanCharmant]

I've added support for the files created by Sports2D: https://github.com/davidpagnon/Sports2D It works better than OpenPose and it's truly open source and focused on sport. If you can try it I'd be curious to know if it works in your case. It's tracking human pose not club.

Unlike for OpenPose this time I properly create a tracked posture object so we can then get the kinematics from the individual points.

At this time I'm not convinced that any of these models are suitable for making measurements though, unless the plane of motion is very well defined and perpendicular to the camera optical axis. I would use them more for qualitative analysis.

[jschoch]

thanks, i jumped right into pose2sim and was able to get some 3d points inferred. Not great but very interesting! I need to try out sports2d next.

[jschoch]

wow, i see you have been busy! Looking really good!

4 questions:

1. can I persist the calibration somehow?
2. can i persist the time calibration? I have to reset this each time to 120fps.
3. does this rely on a specific body configuration or is it generated from the trc file?
4. Since I went through the trouble of calibrating my cameras using caliscope (for pose2sim), can I automagically configure kinovea to use that and to apply the corrected images?

[JoanCharmant]

For 1 & 2 it should be possible by saving a KVA file with just the calibration set up, and loading it first when opening a new video, possibly automatically via Preferences > Playback > General > Default annotation file. It will set the spatial calibration if the current video doesn't already have one set up.

3. The skeleton/armature points are entirely generated from the .trc file, and the kinematics from these points. The colors are a bit random I made the tool quickly as a proof of concept. We could add some angle options later.

4. I'm not familiar with Caliscope, can you export a calibration file and make a new issue and attach it? Ideally with whatever is necessary to verify that it's reloaded correctly like a corresponding video.

[jschoch]

was poking around a bit, code is pretty complex for a WPF newbie. Was trying to see how I might add a Select/Deselect All btn since the default chart view has all the data elements checked by default.

Wondering... Seems like https://github.com/beto-rodriguez/LiveCharts2 might be worth taking a look at. It has an events interface and it is MIT licensed.

Do you think it is worthwhile for me to take a stab at doing a prototype? Any tips for me on how you were thinking about adding this component?

[JoanCharmant]

It's not WPF it's WinForms. The plotting component is OxyPlot: https://github.com/oxyplot/oxyplot It's pretty powerful and I'm currently not even using many of its features and plot types. Also I think most users don't realize you can already drag the plot around, rescale the axes, etc. (By the way this means the vertical hairline corresponding to the current time needs to be an integral part of the plot, not drawn on top of it, because it needs to move with it).

Honestly I'm deep into refactoring the tracking code right now and the next step will have an impact on the plots. Right now trackable objects use their own logic to track their points but later they will be based on the trajectory object. This will change the way the plots are built (and simplify the code a lot).

My intention is to have these plots on a tab in the right hand side panel. They won't be as wide as in your snapshot but I think this is the place that makes the most sense, everything else will be there. The hard part is not necessarily to move the plots there but the synchronization between the main player and the panel (but now that I've moved the drawing style and the tracking this should be easier).

I agree that it needs a select all / deselect all. When all the plots are in a single control it will also need a way to select between the different plot types. Another thing that could be interesting is to show a data table underneath the plot, similar to the "Tracker" software.

Another aspect of this is that the OxyPlot plots can be extracted to a bitmap. This means we could maybe paint them directly on top of the video image somehow, with transparency.

[jschoch]

here's something I hacked together that shows the use case

https://youtu.be/ZNHJcNTGEds