search

open-compass / MMBench

Official Repo of "MMBench: Is Your Multi-modal Model an All-around Player?"
Apache License 2.0
163 stars 10 forks source link

[BUG] A potential bug on the evaluation server in handling submissions with the same file name but different contents #16

Closed MengLcool closed 10 months ago

MengLcool commented 12 months ago

Thanks for your effort on the great job.

When using the mmbench server to evaluate our predictions, we encountered a potential bug related to handling submissions with the same file name but different contents.

For instance, if we have two files: 'mmb/7b-reimplement_bug.xlsx' for mmb and 'mmbcn/7b-reimplement_bug.xlsx' for mmbcn, and we submit them to the server one by one, the bug occurs.

The procedure is as follows:

  1. Submit ‘mmb/7b-reimplement_bug.xlsx’  to server,  and return true result

  2. Submit 'mmbcn/7b-reimplement_bug.xlsx' to the server, which has the same file name as the previous 'mmb/7b-reimplement_bug.xlsx' but is for the 'mmbcn' dataset, and return a false result.

  3. Rename 'mmbcn/7b-reimplement_bug.xlsx' to 'mmbcn/7b-reimplement_bug_rename.xlsx' (a different name from the previous submission) and return a true result.

  4. Rename 'mmbcn/7b-reimplement_bug_rename.xlsx' back to 'mmbcn/7b-reimplement_bug.xlsx' (the same name as the previous submission) and return a false result.

image

Looking forward to your response!

kennymckormick commented 11 months ago

Hi, @MengLcool , thanks for this report, and we will try to reproduce this bug asap.

kennymckormick commented 10 months ago

Hi, @MengLcool , it seems that when two submission files has the same name, there is possibility for some errors to occur. This problem is already fixed now.