Question for how to write ml_planner.yaml in submitting process

[YangYangLeo]

Hello, Could you please tell me what should I write for "model_config“ if I want to submit my own ml_planner?

I just type to "go_model" like the picture above.

But I got "validation_challenge99.log 2023-05-13 14:15:39,072 : ERROR : Planner initialization failed! 2023-05-13 14:15:44,421 : ERROR : Planner initialization failed!" issue after submitting image.

I am wondering if this issue is related to the"model_config" ? If so, could you please tell me a suggestion?Thanks a lot.

BTW here is my id: {"submitted_image_uri": "937891341272.dkr.ecr.us-east-1.amazonaws.com/nuplan-planning-challenge-1856-participant-team-20954:83cee015-17ba-4524-9e75-0ebcfedeb3a2"}

[patk-motional]

Hi @YangYangLeo,

Try without the ' around go_model.

Refer to the comment below.

[YangYangLeo]

Hi @patk-motional ,

when I submit as you said, the new issue comes out.Just like follow, do you have any ideas about that?

I am really thankful.

Traceback (most recent call last): File "/opt/conda/envs/nuplan/lib/python3.9/site-packages/grpc/_server.py", line 443, in _call_behavior response_or_iterator = behavior(argument, context) File "/nuplan_devkit/nuplan/submission/challenge_servicers.py", line 98, in InitializePlanner planners = build_planners(self._planner_config, None) File "/nuplan_devkit/nuplan/planning/script/builders/planner_builder.py", line 58, in build_planners return [_build_planner(planner, scenario) for planner in planner_cfg.values()] File "/nuplan_devkit/nuplan/planning/script/builders/planner_builder.py", line 58, in return [_build_planner(planner, scenario) for planner in planner_cfg.values()] File "/nuplan_devkit/nuplan/planning/script/builders/planner_builder.py", line 25, in _build_planner torch_module_wrapper = build_torch_module_wrapper(planner_cfg.model_config) File "/nuplan_devkit/nuplan/planning/script/builders/model_builder.py", line 19, in build_torch_module_wrapper model = instantiate(cfg) File "/opt/conda/envs/nuplan/lib/python3.9/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 182, in instantiate raise InstantiationException( hydra.errors.InstantiationException: Top level config has to be OmegaConf DictConfig, plain dict, or a Structured Config class or instance

[patk-motional]

You can try a few more methods:

1.

Revert ml_planner.yaml to the original. It should look like this:

ml_planner:
  _target_: nuplan.planning.simulation.planner.ml_planner.ml_planner.MLPlanner
  _convert_: 'all'

  model_config: ???  # Dictionary key from existing planner model config (e.g. raster_model)
  checkpoint_path: ???  # Path to trained model checkpoint

In entrypoint_submission.py

conda run -n nuplan --no-capture-output python -u nuplan/planning/script/run_submission_planner.py output_dir=/tmp/ model=YOUR_MODEL planner=ml_planner planner.ml_planner.model_config=\${model} planner.ml_planner.checkpoint_path="YOUR_CHECKPOINT"

replace YOUR_PLANNER and YOUR_CHECKPOINT.

2.

Modify ml_planner.yaml. It should look like this:

ml_planner:
  _target_: nuplan.planning.simulation.planner.ml_planner.ml_planner.MLPlanner
  _convert_: 'all'

  model_config: ???  # Dictionary key from existing planner model config (e.g. raster_model)
  checkpoint_path: "YOUR_CHECKPOINT"  # Path to trained model checkpoint

In entrypoint_submission.py

conda run -n nuplan --no-capture-output python -u nuplan/planning/script/run_submission_planner.py output_dir=/tmp/ model=YOUR_MODEL planner=ml_planner planner.ml_planner.model_config=\${model}

replace YOUR_PLANNER and YOUR_CHECKPOINT.

3.

Modify default_common.py

# Default common configs

defaults:
  # Add common items
  - scenario_builder: nuplan_mini
  - scenario_filter: one_continuous_log
  - model: YOUR_MODEL
  # Worker that is used to run simulations
  - worker: ray_distributed

Modify ml_planner.yaml to the original. It should look like this:

ml_planner:
  _target_: nuplan.planning.simulation.planner.ml_planner.ml_planner.MLPlanner
  _convert_: 'all'

  model_config: ${model}  # Dictionary key from existing planner model config (e.g. raster_model)
  checkpoint_path: "YOUR_CHECKPOINT"  # Path to trained model checkpoint

In entrypoint_submission.py

conda run -n nuplan --no-capture-output python -u nuplan/planning/script/run_submission_planner.py output_dir=/tmp/ planner=ml_planner

replace YOUR_PLANNER and YOUR_CHECKPOINT.

[YangYangLeo]

HI @patk-motional, I have tried all 3 ways you have mentioned above. Way 1, I got the error "hydra.errors.InstantiationException: Top level config has to be OmegaConf DictConfig, plain dict, or a Structured Config class or instance"

Way2 and Way3, I got the error "FileNotFoundError: [Errno 2] No such file or directory: '/nuplan-devkit/best.ckpt'" However, I have double checked that my model(best.ckpt) is under /nuplan-devkit for the container test-submission.

Do you have any more ideas?

Thanks a lot.

[YangYangLeo]

@patk-motional , all the docker images (3 ways above) run locally present as follow,

Is that run successfull in local?

But when it is submitted, it comes out issue.

[patk-motional]

No, you should see this at the end

nuplan-devkit-simulation-1  | 2023-05-23 05:26:18,306 INFO {/nuplan_devkit/nuplan/planning/simulation/runner/executor.py:127}  Number of successful simulations: 1
nuplan-devkit-simulation-1  | 2023-05-23 05:26:18,306 INFO {/nuplan_devkit/nuplan/planning/simulation/runner/executor.py:128}  Number of failed simulations: 0
nuplan-devkit-simulation-1  | 2023-05-23 05:26:18,306 INFO {/nuplan_devkit/nuplan/planning/script/utils.py:147}  Finished executing runners!
nuplan-devkit-simulation-1  | 2023-05-23 05:26:18,330 INFO {/nuplan_devkit/nuplan/planning/script/utils.py:74}  Saved runner reports to /data/exp/nuplan/exp/simulation/open_loop_boxes/2023.05.23.05.26.09/runner_report.parquet
nuplan-devkit-simulation-1  | 2023-05-23 05:26:18,330 INFO {/nuplan_devkit/nuplan/planning/simulation/main_callback/time_callback.py:27}  Simulation duration: 00:00:06 [HH:MM:SS]
nuplan-devkit-simulation-1  | 2023-05-23 05:26:18,357 INFO {/nuplan_devkit/nuplan/planning/simulation/main_callback/metric_file_callback.py:79}  Metric files integration: 00:00:00 [HH:MM:SS]
nuplan-devkit-simulation-1  | 2023-05-23 05:26:18,392 INFO {/nuplan_devkit/nuplan/planning/simulation/main_callback/metric_aggregator_callback.py:58}  Running metric aggregator: open_loop_boxes_weighted_average
nuplan-devkit-simulation-1  | 2023-05-23 05:26:18,398 INFO {/nuplan_devkit/nuplan/planning/simulation/main_callback/metric_aggregator_callback.py:69}  Metric aggregator: 00:00:00 [HH:MM:SS]
Rendering histograms: 100%|██████████| 6/6 [00:01<00:00,  5.41it/s]
nuplan-devkit-simulation-1  | 2023-05-23 05:26:20,220 INFO {/nuplan_devkit/nuplan/planning/simulation/main_callback/metric_summary_callback.py:344}  Metric summary: 00:00:01 [HH:MM:SS]
nuplan-devkit-simulation-1  | 2023-05-23 05:26:20,220 INFO {/nuplan_devkit/nuplan/planning/script/run_simulation.py:80}  Finished running simulation!

[sindhu-pr]

In https://github.com/motional/nuplan-devkit/issues/302#issuecomment-1549319730 what is YOUR_MODEL, where should this be set and where should model be set so that ${model} is replaced properly?

[patk-motional]

YOUR_MODEL is a yaml file. You should only need to replace YOUR_MODEL. There is no need to modify anything else. Note this is assuming you are using ml_planner.yaml

[sindhu-pr]

YOUR_MODEL is a yaml file. You should only need to replace YOUR_MODEL. There is no need to modify anything else. Note this is assuming you are using ml_planner.yaml

Thanks.

[sindhu-pr]

As suggested in https://github.com/motional/nuplan-devkit/issues/302#issuecomment-1549319730 none of the three options work. requesting the developers to look into this. We are unable to run simulation planner on local copies.

[patk-motional]

Hi @sindhu-pr, it is hard for us to help if you are unable to run it locally. Have you tried with the example planner yet?