Mismatch Players' Bounding Boxes

[gorayni]

I found that several game matches have a mismatch of the players' bounding boxes and possibly their calibration as well. The games with this problem were the following:

italy_serie-a/2016-2017/2017-03-04 - 17-00 AS Roma 1 - 2 Napoli
italy_serie-a/2015-2016/2015-09-22 - 21-45 Udinese 2 - 3 AC Milan   
italy_serie-a/2015-2016/2015-11-22 - 22-45 Inter 4 - 0 Frosinone
italy_serie-a/2016-2017/2016-08-27 - 21-45 Napoli 4 - 2 AC Milan
italy_serie-a/2016-2017/2016-08-28 - 21-45 Cagliari 2 - 2 AS Roma
italy_serie-a/2016-2017/2016-09-11 - 16-00 AC Milan 0 - 1 Udinese
italy_serie-a/2016-2017/2016-09-16 - 21-45 Sampdoria 0 - 1 AC Milan
italy_serie-a/2016-2017/2016-09-18 - 21-45 Fiorentina 1 - 0 AS Roma
italy_serie-a/2016-2017/2016-09-20 - 21-45 AC Milan 2 - 0 Lazio
italy_serie-a/2016-2017/2016-09-21 - 21-45 AS Roma 4 - 0 Crotone
italy_serie-a/2016-2017/2016-09-25 - 13-30 Torino 3 - 1 AS Roma
italy_serie-a/2016-2017/2016-10-02 - 21-45 AS Roma 2 - 1 Inter
italy_serie-a/2016-2017/2017-02-10 - 22-45 Napoli 2 - 0 Genoa
italy_serie-a/2016-2017/2017-02-25 - 20-00 Napoli 0 - 2 Atalanta
italy_serie-a/2016-2017/2017-04-15 - 21-45 Napoli 3 - 0 Udinese
spain_laliga/2015-2016/2015-09-12 - 17-00 Espanyol 0 - 6 Real Madrid
spain_laliga/2015-2016/2015-09-12 - 21-30 Atl. Madrid 1 - 2 Barcelona   
spain_laliga/2016-2017/2016-11-19 - 18-15 Barcelona 0 - 0 Malaga
spain_laliga/2016-2017/2017-01-08 - 22-45 Villarreal 1 - 1 Barcelona
spain_laliga/2016-2017/2017-04-26 - 22-30 Dep. La Coruna 2 - 6 Real Madrid
spain_laliga/2019-2020/2020-02-09 - 18-00 Osasuna 1 - 4 Real Madrid
spain_laliga/2019-2020/2020-02-16 - 18-00 Real Madrid 2 - 2 Celta Vigo
england_epl/2015-2016/2015-11-07 - 20-30 Stoke City 1 - 0 Chelsea
england_epl/2015-2016/2016-04-09 - 17-00 Swansea 1 - 0 Chelsea
england_epl/2016-2017/2017-01-03 - 18-00 Bournemouth 3 - 3 Arsenal                   
europe_uefa-champions-league/2014-2015/2014-11-04 - 22-45 Arsenal 3 - 3 Anderlecht
europe_uefa-champions-league/2014-2015/2014-11-05 - 22-45 Ajax 0 - 2 Barcelona
europe_uefa-champions-league/2015-2016/2015-11-03 - 22-45 PSV 2 - 0 Wolfsburg   
france_ligue-1/2016-2017/2017-04-18 - 19-30 Metz 2 - 3 Paris SG
germany_bundesliga/2014-2015/2015-05-09 - 16-30 Dortmund 2 - 0 Hertha Berlin 

For the rest of the games the bounding boxes match perfectly. I extract the frames by sampling 2 fps from a half game match by doing the following:

ffmpeg -ss "$start_time" \
       -t "$duration_time" \
       -i "$half_match_HQ_video_path"
       -vf fps=fps=2.:round=down \
       -vsync 1 \
       -q:v 1 \
       "$frames_dest_dir"/%05d.jpg

I initially thought that I was incorrectly sampling frames from the videos, but I also tried with the official frame loader class FrameCV from the DataLoader.py with no success.

I am not quite sure what might be the problem, is there an appropiate way to sample the frames from the videos or are the bounding boxes or is the data wrongly labeled?

[SilvioGiancola]

May I ask which bounding boxes are you refering to? and which video were you using (HQ/LQ/720p/224p) ? note that for the task of ReID, we provide the frames with the bounding boxes.

[gorayni]

Sorry, I forgot to be more specific. I'm using SoccerNet-v2 with its associated files for action spotting. Specifically, I am using the high quality videos 1_HQ.mkv, the bounding boxes that comes from 1_player_boundingbox_maskrcnn.json, and the calibration files 1_field_calib_ccbv.json.

[SilvioGiancola]

Hi @gorayni I see, you are using the data from the paper Camera Calibration and Player Localization in SoccerNet-v2 and Investigation of their Representations for Action Spotting.

I am sorry you are experiencing a mismatch between the frames and the bounding boxes, the latter were extracting automatically from the frames, using frameCV. As far as I know, the video module in OpenCV either use ffmpeg or LibAV/AVcodec, the mismatch might come from a different version of the libraries you used, especially if you are on MacOS. The HQ videos have a large diversity of codecs and frame rate, that might create issues in some platforms. That is why we recently released the videos in a unified format at 720p and 25fps (see soccer-net.org - mySoccerNetDownloader.downloadGames(files=["1_720p.mkv", "2_720p.mkv"], split=["train","valid","test","challenge"])). You might want to try out with this new version of the videos.

On a side note, if the mismatch between frames and bounding box correspond to a small and constant delay of a few frames, the paper mentioned above should still maintain good performance in the spotting tasks, since the metric usually allows for a temporal error of 5 to 60 seconds.

[gorayni]

Thank you for your fast answer! I'll give a try on these standardized videos and check the player detection. It would be great if you could release the versions you are using for your Python environment (preferably an Anaconda environment) or a Docker image.