The evalai evaluation takes too long, and we can't submit our new container

[piqiuni]

we want to know if the evaluation is running automatically, and why it takes so long time?

Our last submition takes 15hs and return "Unable to start container. Consider rebuilding your image. Please make sure you are submitting the submission image." And the runing one now has toke more than 24hs and we can't cancle it to run a new one.

Hope to get some advice.

Thanks!

[piqiuni]

And if we submit one to the final(1/3) and it runs too slow then we don't have chance to submit a new one(2/3)？As the evalai limits that you can can't submit a new when when last is still running.

[patk-motional]

Hi @piqiuni,

The evaluation will run automatically. If the container fails to start we need to manually flag it, unfortunately. Therefore it could take us a day before seeing that it is still "Running". We can make submitting concurrent submissions for the test phase possible.

[piqiuni]

Hi @piqiuni,

The evaluation will run automatically. If the container fails to start we need to manually flag it, unfortunately. Therefore it could take us a day before seeing that it is still "Running". We can make submitting concurrent submissions for the test phase possible.

can you help us to stop the running of { "submitted_image_uri": "937891341272.dkr.ecr.us-east-1.amazonaws.com/nuplan-planning-challenge-1856-participant-team-21340:939f8ee2-3c7b-4ddb-9677-20ffc54481dc" }

[patk-motional]

Done. For reference, a nominal warmup phase submission should take about an hour to complete. Anything beyond 2 hours is an anomaly.

[piqiuni]

Done. For reference, a nominal warmup phase submission should take about an hour to complete. Anything beyond 2 hours is an anomaly.

Got it, thanks!