We have received questions from several people on if the nuPlan submission server will be reopened in the future. Here is a quick Q/A:
Will the nuPlan submission server be reopened?
There are a few factors that have to be addressed:
Cost. It is relatively costly to keep the submission server running. We must optimize some variables to reopen the server. This includes but is not limited to submission per week/month, available compute, and concurrent submissions.
Error reporting. As a security measure, we've been manually vetting the error messages and sending them to each participant. We will have to automate this for this to be viable in the long term. The priority is to ensure no sensitive information can be leaked.
Manpower. Running the server requires a lot of maintenance and supervision. Hence, it depends on if we have the bandwidth to support the server in the long run.
Are there plans to release the challenge scenarios and set up to run the benchmark locally?
Unfortunately not. We cannot release the scenarios because they are a part of a private set - not available to the public. This is to ensure the integrity of the benchmark and any future competitions. This is to ensure that training data cannot be contaminated by the test data.
One of nuPlan's missions is to provide the academic community with a high-quality standardized benchmark to evaluate and compare new and novel motion planning algorithms. We will continue to support the devkit. Whether we reopen the submission server will have to be a team decision, backed by Motional. Thank you for your continued interest and support. Please stay tuned for further updates.
Hi all,
We have received questions from several people on if the nuPlan submission server will be reopened in the future. Here is a quick Q/A:
Will the nuPlan submission server be reopened?
There are a few factors that have to be addressed:
Are there plans to release the challenge scenarios and set up to run the benchmark locally?
Unfortunately not. We cannot release the scenarios because they are a part of a private set - not available to the public. This is to ensure the integrity of the benchmark and any future competitions. This is to ensure that training data cannot be contaminated by the test data.
One of nuPlan's missions is to provide the academic community with a high-quality standardized benchmark to evaluate and compare new and novel motion planning algorithms. We will continue to support the devkit. Whether we reopen the submission server will have to be a team decision, backed by Motional. Thank you for your continued interest and support. Please stay tuned for further updates.