search

SilvioGiancola / SoccerNetv2-DevKit

Development Kit for the SoccerNet Challenge
MIT License
168 stars 39 forks source link

Question related to camera mIoU metric #18

Closed jvbsoares closed 2 years ago

jvbsoares commented 3 years ago

Greetings,

Thanks for making all this code available, this repo is fantastic!

I wanted to ask about some details regarding the camera mIoU metric. On initially going over the SoccerNet-V2 paper, I got the idea that the camera class "Other" should be treated just as any of the other classes. When I went over the code in more detail, I am second-guessing if that's the case and wanted to ask for your thoughts on what should be the ideal treatment of the "Other" camera class.

In looking at Camera_Type_DICTIONARY at https://github.com/SilvioGiancola/SoccerNetv2-DevKit/blob/f28d40e69a27481a82ee45d8d8cfdad1f05b6d4f/Task2-CameraShotSegmentation/CALF-segmentation/src/config/classes.py#L16 the "Other" camera label is lower-cased as "other", whereas in my JSON files they show up with initial upper-case as "Other", so I wonder if those labels are being ignored when being read here: https://github.com/SilvioGiancola/SoccerNetv2-DevKit/blob/f28d40e69a27481a82ee45d8d8cfdad1f05b6d4f/Task2-CameraShotSegmentation/CALF-segmentation/src/dataset.py#L259 At the same time, I guess the "I don't know" annotation will still get mapped to the last label 12 for "Other"?

Ignoring the "Other" label might be consistent with the weights assigned to the different classes for CALF segmentation, where the last weight (for "Other") is zero: https://github.com/SilvioGiancola/SoccerNetv2-DevKit/blob/f28d40e69a27481a82ee45d8d8cfdad1f05b6d4f/Task2-CameraShotSegmentation/CALF-segmentation/src/loss.py#L20

At the same time, I was confused since the evaluation code seems to take all classes into account when computing mIoU: https://github.com/SilvioGiancola/SoccerNetv2-DevKit/blob/f28d40e69a27481a82ee45d8d8cfdad1f05b6d4f/Task2-CameraShotSegmentation/CALF-segmentation/src/train.py#L366-L390

So I wanted to get your thoughts on what the appropriate definition for mIoU should be and also confirm if "I don't know" should be treated the same as "Other" for evaluation purposes?

Thanks very much! Joao

SilvioGiancola commented 3 years ago

Hi @jvbsoares , That capitalized word might be an issue, you are correct. Have you tried comparing the performances with and without that capitalized letter? I will need the help of @meisamjam or @jagob to support you in this issue :) AFAIR, we do consider "I don't know" and "Other" as the same class for the evaluation, that is why they both point to 12 in Camera_Type_DICTIONARY.

jvbsoares commented 3 years ago

Hi Silvio. Thank you very much for looking into this!

As per your suggestion, I ran evaluation of a model (CALF segmentation model) with and without the capitalized "Other". The model was trained with the original code. There is indeed some difference in the results. Here are the results I got from that single model:

thank you! Joao

SilvioGiancola commented 3 years ago

Hi @jvbsoares , thank you for reporting this discrepancy, I am glad the difference between both setups is within such a small range :) Cheers!