search

argoverse / av2-api

Argoverse 2: Next generation datasets for self-driving perception and forecasting.
https://argoverse.github.io/user-guide/
MIT License
306 stars 71 forks source link

EvalAI Submission Blocked #268

Closed ziqipang closed 2 weeks ago

ziqipang commented 4 weeks ago

Hi @wqi @neeharperi

I am trying to submit the multi-world forecasting challenge before the deadline, but it seems the submission is blocked since last evening. Could you please help me check if there is any issue? (FYI The single-world forecasting challenge looks good to me, not sure if the issue is related to the file size) I found on GitHub that you guys have helped a lot of people addressing this problem. Thank you so much!

Best,

Ziqi

wqi commented 4 weeks ago

Hey @ziqipang! I just checked the evaluation server and everything looks good on our end (for both single and multi-agent evaluation).

Looks like it might be an issue specific to your submission file unfortunately.

What are you seeing on your end?

ziqipang commented 4 weeks ago

@wqi Thanks for your follow-up! I am trying to submit my submission.parquet (~270MB) to evalai server, using the "upload file" option. After I click the "Submit" button, the page freezes without output. I waited for ~5mins but still not response. In comparison, the submission to single-agent forecasting is completed within 10s, so it is confusing. Please let me know what needs to be done from my end.

wqi commented 4 weeks ago

I suspect that this may be an EvalAI issue related to the web interface.

Do you see your submission added to the "My submissions" page on EvalAI after waiting 5 minutes?

If you do see a submission being created, but it's stuck in the scoring/processing stage, then I can take a look on our end.

ziqipang commented 4 weeks ago

@wqi Thanks for the reply! I cannot see my submission added to EvalAI after waiting a long period. Do you suggest any workarounds for this issue?

I also tried the CLI submission but encountered the following issue. It would be great if I could also submit via CLI with your help. Thank you!

image
wqi commented 4 weeks ago

Have you tried submitting from a different computer or on a different network?

This looks like an issue with the EvalAI web interface, which we unfortunately don't have any control over.

Re: CLI submission, I think that looks like an authentication error? You may have to consult the EvalAI CLI documentation to set the correct tokens for authentication.

ziqipang commented 4 weeks ago

@wqi Thanks for the answer! I have tried a different network but not a new computer yet. I understand that you have limited control on this. Let me try again.

bharilya commented 2 weeks ago

Hey @wqi , The submission period for Argoverse2 forecasting has ended, so we cannot submit our model's test results. Is there any way to still do it?

ashok-arora commented 2 weeks ago

Hey @wqi @neeharperi, any updates on how to submit the files for evaluation now that the competition has ended?

wqi commented 2 weeks ago

Should be open again now, thanks for your patience!

bharilya commented 2 weeks ago

@wqi Thank you very much. Moreover, I am currently in the second year of my PhD and have already submitted several papers using this dataset. Could you please ensure that the submission leaderboard remains available for at least a few more years?

wqi commented 2 weeks ago

No problem! We plan to keep the leaderboard open for the forseeable future.

We just had submissions closed for the last week while we finalized the results of this year's CVPR forecasting challenge. :-)