Evaluating a point-based method

[mchagneux]

Hello,

Is there an easy workaround to use this kit for a tracking method which only detects 2D points ?

From what I see, given bounding boxes coordinates it's easy to rewrite the function computing similarity scores to use Euclidean distance between the centers, but other than that it seems that I will have to specify "fake" bounding box width and heights if I want to comply with any of the dataset formats.

Thanks in advance.

[JonathonLuiten]

Hey! Thanks for the question!

We are actually in the process of supporting official benchmarks that use this kind of evaluation (in progress, can't guarantee when it will be done).

However, I have already had some people ask me about something similar so I have already implemented the Euclidian similarity function, you can find it here: https://github.com/JonathonLuiten/TrackEval/blob/ef561bd5f4a40ca09b50e9f7f38c0568cf250180/trackeval/datasets/_base_dataset.py#L284

You're correct that's it's easy to adapt some current dataset to use it (but not currently dataset currently uses it). I will walk you through an example of how to do this now. You only need to change a few lines I think.

E.g. let's change the mot_challenge_2d_box to the way you want it. First change this line: https://github.com/JonathonLuiten/TrackEval/blob/ef561bd5f4a40ca09b50e9f7f38c0568cf250180/trackeval/datasets/mot_challenge_2d_box.py#L436 To similarity_scores = self._calculate_euclidean_similarity(gt_dets_t, tracker_dets_t, zero_distance=2.0) You probably a want a different value for 'zero_distance' than 2.0 (this is the distance after which the similarity is considered 0). If your values are in pixels then 2.0 is definitely too small. (2.0 is the default here because for 3D tracking it corresponds to 2 meters)

Second let's change this line: https://github.com/JonathonLuiten/TrackEval/blob/ef561bd5f4a40ca09b50e9f7f38c0568cf250180/trackeval/datasets/mot_challenge_2d_box.py#L238 You want this now to only read in 2 value for 2D point tracking (or 3 values for 3D point tracking). e.g. raw_data['dets'][t] = np.atleast_2d(time_data[:, 2:4])

Finally you should change the lines: here: https://github.com/JonathonLuiten/TrackEval/blob/ef561bd5f4a40ca09b50e9f7f38c0568cf250180/trackeval/datasets/mot_challenge_2d_box.py#L264 and here: https://github.com/JonathonLuiten/TrackEval/blob/ef561bd5f4a40ca09b50e9f7f38c0568cf250180/trackeval/datasets/mot_challenge_2d_box.py#L273 such that empty boxes also have the correct size (2 for 2d, 3 for 3d) e.g. raw_data['dets'][t] = np.empty((0, 2)) and raw_data['gt_crowd_ignore_regions'][t] = np.empty((0, 2))

And I think that should be all.

Let me know if that works for you?

Note that you could get away with not doing the last 3 edits if you just made sure that in your files the other values of the bbox were always 0 (but safer to make these edits, then accidently putting different values won't screw up the evaluation.

Jono

(won't close until you let me know if it works)

[mchagneux]

Hi Jonathan,

Thank you so much for the thorough answer, this seems to work well ! I just had to disable preprocessing otherwise I get that error:

"trackeval.utils.TrackEvalException: Attempting to evaluate using invalid gt classes. This warning only triggers if preprocessing is performed, e.g. not for MOT15 or where prepropressing is explicitly disabled. Please either check your gt data, or disable preprocessing. The following invalid classes were found in timestep 0: -1"

I think this makes sense because I have been formatting my gt and trackers files/folders exactly as for MOT15.

Thank again !

Mathis

[JonathonLuiten]

Perfect! Glad to help. Closing this now!