evaluation on VIS Eval server issue

[dmn-sjk]

Hi, I have an issue with results evaluation on VIS Eval server. I uploaded segmentation masks generated with TENT on validation split (submission file) and this is the result I get:

Starting Evaluation.....
Evaluation environments:
 - Python version: 3.8.17
 - NumPy version: 1.21.0
Start unzipping...
Unzipping completed.
Evaluation phase: Dev
>> Evaluating semantic segmentation estimation...
>> Semantic segmentation results:
 {'mIoU': 0.12130181939646796, 'mAcc': 0.1382010607215447, 'start_mIoU': 0.11743503124325975, 'end_mIoU': 0.265219426411771, 'loop_back_mIoU': 0.17676837107687926, 'mIoU_drop': -0.14778439516851125}
Completed evaluation for Dev Phase
{'mIoU': 0.12130181939646796, 'mIoU_drop': -0.14778439516851125, 'mIoU_source': 0.11743503124325975, 'mIoU_target': 0.265219426411771, 'mIoU_loop_back': 0.17676837107687926, 'overall': 0.41687060973349044}

The values clearly differ from the local evaluation. Results of TENT on test split (submission_file) also seem too low:

Starting Evaluation.....
Evaluation environments:
 - Python version: 3.8.17
 - NumPy version: 1.21.0
Start unzipping...
Unzipping completed.
Evaluation phase: Test
>> Evaluating semantic segmentation estimation...
>> Semantic segmentation results:
 {'mIoU': 0.2837294566129351, 'mAcc': 0.3823126694620284, 'start_mIoU': 0.09630856237282565, 'end_mIoU': 0.5338620767039061, 'loop_back_mIoU': 0.09739022159143337, 'mIoU_drop': -0.4375535143310804}
Completed evaluation for Test Phase

I followed the instructions of uploading only every 10 frames and coding semantic ID in R channel of RGB images. Do you know what am I doing wrong?

[btwardow]

Hi, @zwbx @suniique any news or info about this issue?

[zwbx]

sorry for late reply. first. have you remap the labels to original ID for each category?

[dmn-sjk]

Oh, I didn't realize I have to do that. I will try it

[dmn-sjk]

I tried uploading the submission with remapped labels but now I got the following error from the server:

Traceback (most recent call last):
  File "/code/scripts/workers/submission_worker.py", line 514, in run_submission
    submission_output = EVALUATION_SCRIPTS[challenge_id].evaluate(
AttributeError: module 'challenge_data.challenge_4' has no attribute 'evaluate'

[zwbx]

Please strictly follow the submission rules. If it still doesn't work, try it several times soon later, sometimes the server might be experiencing issues.