As a mentor/member of a participating team in the competition and as a previous organizer for multiple AI competitions, I want to make a comment/discuss the latest submission upload failures that multiple teams have been experiencing.
For upload failure cases, the quota count is increased although the code was not evaluated and, therefore, not reflected in the leaderboard. For teams experiencing this case, the default answer by the organizers has been:
"In the spirit of fairness, we will not be able to refund these submissions without evidence of a technical issue on our end, and encourage you to try again."
I think that the upload failure cases are actually hurting the fairness of the AISG Online Safety Challenge. While I understand how challenging it is to create a GPU infrastructure system for an AI competition, as an organizer, the goal for fairness is to make sure that each submission is run and it is run just one time, the evaluation is correct within the rules, and that the dataset is not leaked.
From our part as a team, we need to make sure that our Docker image runs in the environment respecting the size and time limit constraints set by the organizer. If these are not respected, the team should be rightfully penalized losing one submission from their quota.
At this moment, for the cases in which upload failures occur, a component of luck and circumstances that cannot be predicted is happening. This clearly hurts the fairness of the competition as there are teams that, for the same amount of upload attempts, have their strategy evaluated more than other teams. Subtracting a submission quota in this case seems unreasonable because the team's code was not evaluated in the competition system.
I understand that the issue can be caused by an external factor that the organizers cannot control (internet instabilities, system instabilities, etc.) but I kindly ask for re-evaluation to keep the same submission quota count in the case of upload failures.
Dear AISG Online Safety Challenge organizers,
As a mentor/member of a participating team in the competition and as a previous organizer for multiple AI competitions, I want to make a comment/discuss the latest submission upload failures that multiple teams have been experiencing.
For upload failure cases, the quota count is increased although the code was not evaluated and, therefore, not reflected in the leaderboard. For teams experiencing this case, the default answer by the organizers has been:
"In the spirit of fairness, we will not be able to refund these submissions without evidence of a technical issue on our end, and encourage you to try again."
I think that the upload failure cases are actually hurting the fairness of the AISG Online Safety Challenge. While I understand how challenging it is to create a GPU infrastructure system for an AI competition, as an organizer, the goal for fairness is to make sure that each submission is run and it is run just one time, the evaluation is correct within the rules, and that the dataset is not leaked.
From our part as a team, we need to make sure that our Docker image runs in the environment respecting the size and time limit constraints set by the organizer. If these are not respected, the team should be rightfully penalized losing one submission from their quota.
At this moment, for the cases in which upload failures occur, a component of luck and circumstances that cannot be predicted is happening. This clearly hurts the fairness of the competition as there are teams that, for the same amount of upload attempts, have their strategy evaluated more than other teams. Subtracting a submission quota in this case seems unreasonable because the team's code was not evaluated in the competition system.
I understand that the issue can be caused by an external factor that the organizers cannot control (internet instabilities, system instabilities, etc.) but I kindly ask for re-evaluation to keep the same submission quota count in the case of upload failures.
Best regards, Felipe.