search

waymo-research / waymo-open-dataset

Waymo Open Dataset
https://www.waymo.com/open
Other
2.73k stars 617 forks source link

Unknown error in WOSAC submission? #875

Open pengzhenghao opened 1 month ago

pengzhenghao commented 1 month ago

Hi I just reused my WOSAC submission generation pipeline half year ago and made a submission. But it seems an unknown error happens:

(original post has a wrong link. Here is the real one:)

https://waymo.com/open/challenges/sim-agents/results/92d57ec4-798b/1727496916523000/

Is there anyway I can know what's the error? Thanks!

pengzhenghao commented 1 month ago

I believe this might be the same bug we previously encountered: https://github.com/waymo-research/waymo-open-dataset/issues/807#issuecomment-2034939254

Close this for now.

pengzhenghao commented 1 month ago

Sorry after reusing a previously working pipeline I still get unknown error. Can someone help me take a look? Thanks!

https://waymo.com/open/challenges/sim-agents/results/92d57ec4-798b/1727718185742000/

pengzhenghao commented 1 month ago

Hello?

jlambert-waymo commented 1 month ago

Hi @pengzhenghao, it looks like you had a previously successful submission on April 10/11: 1712813129292000.tar.gz.

Please see the April 11, 2024 update: https://github.com/waymo-research/waymo-open-dataset?tab=readme-ov-file#april-2024-update

This commit contains the changes: https://github.com/waymo-research/waymo-open-dataset/commit/22e51efa143667f7e27442e06f6443f1093ea0a9

New required metadata fields are added to this proto (describing your submission): src/waymo_open_dataset/protos/sim_agents_submission.proto

Have you added those?

pengzhenghao commented 3 weeks ago

I have added the required metadata fields and was able to make a submission with those new fields in.

I just made another attempt and there is no error printed out:

https://waymo.com/open/challenges/sim-agents/results/92d57ec4-798b/1729809749033000/

Could you help me take a look? Thank you!

zhejz commented 2 weeks ago

I'm facing exactly the same problem. Here is my submission to validation split: https://waymo.com/open/challenges/sim-agents/results/5ea7a3eb-7337/1730040554700000/

However, the leaderboard works fine when I use the same code but pack the ground-truth future into the submission file. So it's related to the trajectories, not the metadata.

zhejz commented 2 weeks ago

Seems to be the same as #864 ? @jlambert-waymo @nicomon24

zhejz commented 2 weeks ago

I uploaded exactly the same submission file again, now this time it works. Somehow the evaluation server fails randomly from time to time.

pengzhenghao commented 2 weeks ago

@jlambert-waymo Could you please take a look? It's even better if you can update the server side error messsage. Thanks!