search

Guanghan / lighttrack

LightTrack: A Generic Framework for Online Top-Down Human Pose Tracking
MIT License
720 stars 142 forks source link

Validation set size inconsistent with PoseTrack2018 #16

Open taconite opened 5 years ago

taconite commented 5 years ago

Hi,

Thank you for releasing this quality work!

I notices in your paper you mentioned that validation set of PoseTrack18 has 74 sequences, while the official PoseTrack dataset has 170 validation sequences. I assume all the results regarding the validation set here and in the paper correspond to 74 sequences, not the full 170 sequences? Can you provide a way to evaluate your models (detection, pose estimation, matching, etc.) on full validation set?

thanks!

Guanghan commented 5 years ago

Hi @taconite ,

I am confused. While I was participating in the ECCV'18 PoseTrack Challenge, the PoseTrack'18 dataset includes 593 training videos, 74 validation videos and 375 testing videos. So I am indeed evaluating the models on the full validation set. By the time the arxiv paper was uploaded (May 7th, 2019), the validation set was still 74 sequences.

One possibility is that the PoseTrack'18 benchmark is re-opened and more validation sequences are included in the annotations. In fact, while I am writing this, I just checked the official website and see the annotation version is now 0.45 instead of 0.2. So it is highly likely this case.

I might update the validation later. If you are in a hurry, you can just do some minor modification, and I believe you will soon be able to evaluate on the newly released validation set.

taconite commented 5 years ago

Hi @Guanghan,

Thank you for the quick reply. In this case can you share your fine-tuned detection model (specifically Deformable_FPN_RCNN) on PoseTrack dataset?

Guanghan commented 5 years ago

@taconite The Deformable_FPN_RCNN is implemented in MXNet, and was adapted from https://github.com/msracver/Deformable-ConvNets. It is not fine-tuned, I used their pre-trained model deform-RFCN-101 and deform-FPN-101, where the better performance was achieved with deform-RPN-101.

Before releasing the lighttrack code base on Github, I added a demo with YOLO as detector (cleanly and "light-weightedly" implemented in Pytorch). It should be easy to add Deformable-convnet as well to make the processing truly online (not processing in seperate stages), but I decided not to do so because the Lighttrack repository would require 3 deep learning frameworks (tensorflow, pytorch, mxnet) and would scare new users away from using it.

I may add this in another branch, I guess? Maybe make lighttrack-main and lighttrack-slim? Seems dumb... Let me think about it. Thanks! It should not take too long (a week at most).

taconite commented 5 years ago

@Guanghan Thank you for your detailed explanation! I will grab the MXNet version of detector for now.