Cloud-CV / EvalAI

:cloud: :rocket: :bar_chart: :chart_with_upwards_trend: Evaluating state of the art in AI
https://eval.ai
Other
1.77k stars 787 forks source link

​ Problem with remote evaluation worker #4424

Open JannikSchneider12 opened 1 month ago

JannikSchneider12 commented 1 month ago

Hello everyone,

I am trying to set up a remote evaluation challenge and facing issues. When I submit a file in the web interface, the fetching as well as the evaluation script are working but at the end, I am getting this error:

Error: 400 Client Error: Bad Request for url: https://eval.ai/api/jobs/challenge/2389/update_submission/

I also printed out the data that is passed to the update_submission_data function:

{‘challenge_phase’: 4723, ‘submission’: 477497, ‘stdout’: ‘’, ‘stderr’: ‘’, ‘submission_status’: ‘FINISHED’, ‘result’: [{‘train_split’: {‘final_train_loss’: 0.00010908210573358123, ‘final_val_loss’: 0.0009243950532566357, ‘Metric3’: 123, ‘Total’: 123}}, {‘test_split’: {‘final_train_loss’: 0.00010908210573358123, ‘final_val_loss’: 0.0009243950532566357, ‘Metric3’: 123, ‘Total’: 123}}], ‘metadata’: ‘’}

Can someone help me whats wrong?

Thanks for your time and help

PredictiveManish commented 3 weeks ago

Does anyone working on this? If not interested to work on this issue. Please assign it to me.

JannikSchneider12 commented 3 weeks ago

Just to give an update. So I changed the structure of my results dictionary and now it runs without errors. However while the result files for the submissions are now present, the leaderboard remains empty, so I am not sure whether something with my format is still wrong. One result file in the web interface looks e.g. like this:

[{'final_train_loss': 6.025793465431145e-05, 'final_val_loss': 0.00013760417953432937, 'Metric3': 123, 'Total': 123}]

And this is my leaderboard setting in the config:

leaderboard: