Is the sim agents benchmark still running?

waymo-research / waymo-open-dataset

Waymo Open Dataset

https://www.waymo.com/open

Other

2.6k stars 604 forks source link

Is the sim agents benchmark still running? #749

Open jonahthelion opened 7 months ago

jonahthelion commented 7 months ago

Hello,

I made a submission to the validation eval for the sim agents benchmark about 2 days ago and it hasn't finished evaluating yet. I wanted to check if the evaluation server is down?

I see this message when I view the submission:

Thank you!

alexgorban commented 7 months ago

Hi,

thanks for reaching out. Our evaluation server has some stability issues, I've just triggered a reprocessing of your submission. The results should be ready within ~1 day. Apologize for the inconvenience.

jonahthelion commented 7 months ago

Thank you very much for the prompt response!

I've just uploaded a new submission to the sim agents test server. I was wondering if it would be possible to verify that the evaluation is running on that submission now? I'm guessing it is, but I want to be absolutely sure there isn't a problem given there are conference deadlines coming very soon.

Thank you!

Update: I see it has already finished evaluating! We're all good now.

jonahthelion commented 5 months ago

Hello,

I made a submission to the sim agents test server 12 hours ago but it still says it's processing. Is the server down again?

jonahthelion commented 3 months ago

Hello, I submitted to the sim agents test server 26 hours ago but it says it's still processing (I was struggling to upload results and ended up submitting 2 that are the same by accident, sorry about that). I wanted to check if the server is down?