Black-Box Person Tracking

[cstenkamp]

Hey Frankie, You mention you have black-box-person tracking, however in my case the annotation which person it is still jumps around quite a bit - draw_ann (https://github.com/frankier/skelshop/blob/master/skelshop/drawsticks.py#L141) uses the pers_id from bundle, which is what skel_iter contains, which is skel_read, which is AsIfOrdered(UnsegmentedReader(skel_h5f)) (https://github.com/frankier/skelshop/blob/master/skelshop/cmd/playsticks.py#L37-L38), which, as far as I see it, only reads the contents from the hd5-file and doesn't do anything to check for person-consistency, or am I missing something?

[cstenkamp]

I just saw that it goes as deep as ShotReader -> DumpReaderPoseBundle -> PoseBase.from_keypoints() (holy cheese, this is the deepest code-rabbit-hole I've ever seen) but I still don't see any sorting. If one were to implement some kind of sorting (at least in some crude sense like "if there is no scene cut, assume that the skeleton closest to skel1 from the frame before is also skel1 this time"), where would that best be done in the pipeline?

[frankier]

There are unsorted dumps, which are untracked and sorted dumps which are tracked. In retrospect, I should not have introduced the word "sorted" into the mix and instead just referred to tracking. I will try and rectify this soon.

I recently (after the meeting) updated the documentation with a little more info (still needs more work). See https://frankier.github.io/skelshop/pipelines-overview/

The idea is you can dump without tracking and track later. This is useful in a cluster because you can dump on a GPU machine and track on a CPU machine.

It may be a bit over-abstracted. The idea is to allow enough flexibility to add something other than OpenPose in the future and also to deal with different types of pose dumps. All I would say in my defense is there is a lot more over-abstracted out there.

[frankier]

Thanks for bringing this up BTW. It gives me a much clearer idea where to push next on the docs/general usability front.

[frankier]

The actual tracking algorithms are here: https://github.com/frankier/skelshop/blob/master/skelshop/track/confs.py and they are documented here: https://frankier.github.io/skelshop/tracking/

If you wanted to add more tracking you could e.g. add a new metric (e.g. re-identification via face distance) and then you would create a new TrackingSpec in CONFS which took advantage of it. lighttrackish works o.k. for news videos as far as I've seen.

I will work on improving the docs more soon.

[frankier]

By the way, if you're interested in pose tracking here is a shared task/dataset: https://posetrack.net/

Here is state of the art black box tracking (they just use a GCN to classify whether skeletons are from the same person) https://arxiv.org/abs/1912.02323

Here's a simple video/optical flow based approach: https://arxiv.org/abs/1804.06208

However, versus that benchmark, our situation is much easier, at least in the studio, since people stay still mostly and so does the camera. Shots are quite short usually, so even if we lose someone, the shot will end pretty soon after anyway.

As Phil said in the meeting, we don't need to overdo this, since it's quite likely the next pose estimator backend we decide upon after OpenPose will include tracked poses out of the box.

BTW, you can self-assign one of these issues (or any issue not assigned to me) if you'd like to work on one of them: https://github.com/frankier/skelshop/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22

Also do you know why you needed ffprobe for getting the FPS -- I didn't -- it would be good to figure this out: https://github.com/frankier/skelshop/pull/14

[frankier]

P.S: https://github.com/frankier/skelshop/invitations

[cstenkamp]

Oh, I didn't see the invitation so far, but now it seems to be expired - sorry for the annoyance but could you invite me again as collaborator?

[frankier]

Done

[frankier]

Closing as concluded. Feel free to reopen.