search

motional / nuplan-devkit

The devkit of the nuPlan dataset.
https://www.nuplan.org
Other
673 stars 129 forks source link

Submission failed without stderr #284

Closed Shi-Qi-Li closed 1 year ago

Shi-Qi-Li commented 1 year ago

Hi team, I have a submission which is in Failed status on evalai but the corresponding stderr file is empty, could you provide more information that I can debug my docker, Thanks! my submission id {"submitted_image_uri": "937891341272.dkr.ecr.us-east-1.amazonaws.com/nuplan-planning-challenge-1856-participant-team-16486:5866a88f-cb67-428b-9e82-d922b87c89c0"}

patk-motional commented 1 year ago

Hi @Shi-Qi-Li,

Here is your error:

Traceback (most recent call last):
--
File "/opt/conda/envs/nuplan/lib/python3.9/site-packages/grpc/_server.py", line 443, in _call_behavior
response_or_iterator = behavior(argument, context)
File "/nuplan_devkit/nuplan/submission/challenge_servicers.py", line 98, in InitializePlanner
planners = build_planners(self._planner_config, None)
File "/nuplan_devkit/nuplan/planning/script/builders/planner_builder.py", line 70, in build_planners
planner = cache.get(name, _build_planner(config, scenario))
File "/nuplan_devkit/nuplan/planning/script/builders/planner_builder.py", line 26, in _build_planner
torch_module_wrapper = build_torch_module_wrapper(planner_cfg.model_config)
File "/nuplan_devkit/nuplan/planning/script/builders/model_builder.py", line 19, in build_torch_module_wrapper
model = instantiate(cfg)
File "/opt/conda/envs/nuplan/lib/python3.9/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 182, in instantiate
raise InstantiationException(
hydra.errors.InstantiationException: Top level config has to be OmegaConf DictConfig, plain dict, or a Structured Config class or instance
patk-motional commented 1 year ago

You can join our slack workspace: https://join.slack.com/t/opendrivelab/shared_invite/zt-1uhny7uci-T5~otGGdwUtGo8L1j0~NUA

Shi-Qi-Li commented 1 year ago

Hi @Shi-Qi-Li,

Here is your error:

Traceback (most recent call last):
--
File "/opt/conda/envs/nuplan/lib/python3.9/site-packages/grpc/_server.py", line 443, in _call_behavior
response_or_iterator = behavior(argument, context)
File "/nuplan_devkit/nuplan/submission/challenge_servicers.py", line 98, in InitializePlanner
planners = build_planners(self._planner_config, None)
File "/nuplan_devkit/nuplan/planning/script/builders/planner_builder.py", line 70, in build_planners
planner = cache.get(name, _build_planner(config, scenario))
File "/nuplan_devkit/nuplan/planning/script/builders/planner_builder.py", line 26, in _build_planner
torch_module_wrapper = build_torch_module_wrapper(planner_cfg.model_config)
File "/nuplan_devkit/nuplan/planning/script/builders/model_builder.py", line 19, in build_torch_module_wrapper
model = instantiate(cfg)
File "/opt/conda/envs/nuplan/lib/python3.9/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 182, in instantiate
raise InstantiationException(
hydra.errors.InstantiationException: Top level config has to be OmegaConf DictConfig, plain dict, or a Structured Config class or instance

Thanks for your reply! I have one more question about the submission. I have a ml planner which I can successfully simulate locally, based on the submission tutorial if I want to submit it, I need to

  1. set planner=ml_planner in entrypoint_submission.sh
  2. set model_config and checkpoint_path path in ml_planner.yaml

Follow these two steps I got the upper error log so I want to double check if I miss any steps?

patk-motional commented 1 year ago

Hi @Shi-Qi-Li,

That seems about right. One step that might be missing is step three from this tutorial. You need to specify your checkpoint file in Docker.submission. All paths here are relative to nuplan_devkit/. The simplest thing you can do is to place your checkpoint in nuplan_devkit/ and replace my_fabulous_model.ckpt with yours. In ml_planner.yaml, give the relative path to your config.

Please verify by running docker-compose locally first.

Hope that helps!

jkewang commented 1 year ago

what about this submission, also with no output. @patk-motional {"submitted_image_uri": "937891341272.dkr.ecr.us-east-1.amazonaws.com/nuplan-planning-challenge-1856-participant-team-8540:928923d0-798c-4bdf-bfba-852ce5b3328b"}

patk-motional commented 1 year ago

Hi @jkewang,

What is your team name?

jkewang commented 1 year ago

Hi @jkewang,

What is your team name?

xg_test

patk-motional commented 1 year ago

Your submission exceeded the time limit:

StatusCode.DEADLINE_EXCEEDED: The submission container timed out during the trajectory computation.

Shi-Qi-Li commented 1 year ago

Hi @patk-motional , I have another submission failed with empty stderr, I'm not sure it's caused by the time limit or other issue, could you please provide the stderr file?

patk-motional commented 1 year ago

Hi @Shi-Qi-Li

Your latest submission (May 9, 2023 6:41:40 PM) exceeded the time limit:

StatusCode.DEADLINE_EXCEEDED: The submission container timed out during the trajectory computation.

Shi-Qi-Li commented 1 year ago

Hi @Shi-Qi-Li

Your latest submission (May 9, 2023 6:41:40 PM) exceeded the time limit:

StatusCode.DEADLINE_EXCEEDED: The submission container timed out during the trajectory computation.

Got it! Thank you!

YangYangLeo commented 1 year ago

HI @Shi-Qi-Li, I got the same issue like you when submission. Traceback (most recent call last):

File "/opt/conda/envs/nuplan/lib/python3.9/site-packages/grpc/_server.py", line 443, in _call_behavior response_or_iterator = behavior(argument, context) File "/nuplan_devkit/nuplan/submission/challenge_servicers.py", line 98, in InitializePlanner planners = build_planners(self._planner_config, None) File "/nuplan_devkit/nuplan/planning/script/builders/planner_builder.py", line 70, in build_planners planner = cache.get(name, _build_planner(config, scenario)) File "/nuplan_devkit/nuplan/planning/script/builders/planner_builder.py", line 26, in _build_planner torch_module_wrapper = build_torch_module_wrapper(planner_cfg.model_config) File "/nuplan_devkit/nuplan/planning/script/builders/model_builder.py", line 19, in build_torch_module_wrapper model = instantiate(cfg) File "/opt/conda/envs/nuplan/lib/python3.9/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 182, in instantiate raise InstantiationException( hydra.errors.InstantiationException: Top level config has to be OmegaConf DictConfig, plain dict, or a Structured Config class or instance

Would you like to share your idea with me how to solve it?

I am really thankful.

Shi-Qi-Li commented 1 year ago

HI @Shi-Qi-Li,

I got the same issue like you when submission. Traceback (most recent call last): File "/opt/conda/envs/nuplan/lib/python3.9/site-packages/grpc/_server.py", line 443, in _call_behavior response_or_iterator = behavior(argument, context) File "/nuplan_devkit/nuplan/submission/challenge_servicers.py", line 98, in InitializePlanner planners = build_planners(self._planner_config, None) File "/nuplan_devkit/nuplan/planning/script/builders/planner_builder.py", line 70, in build_planners planner = cache.get(name, _build_planner(config, scenario)) File "/nuplan_devkit/nuplan/planning/script/builders/planner_builder.py", line 26, in _build_planner torch_module_wrapper = build_torch_module_wrapper(planner_cfg.model_config) File "/nuplan_devkit/nuplan/planning/script/builders/model_builder.py", line 19, in build_torch_module_wrapper model = instantiate(cfg) File "/opt/conda/envs/nuplan/lib/python3.9/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 182, in instantiate raise InstantiationException( hydra.errors.InstantiationException: Top level config has to be OmegaConf DictConfig, plain dict, or a Structured Config class or instance

Would you like to share your idea with me how to solve it?

I am really thankful.

Hi @YangYangLeo , You can try directly write the model_config in ml_planner.yaml, for example

ml_planner:
  _target_: nuplan.planning.simulation.planner.ml_planner.ml_planner.MLPlanner
  _convert_: 'all'

  model_config: 
    _target_: nuplan.planning.training.modeling.models.simple_vector_map_model.VectorMapSimpleMLP
    _convert_: 'all'

    # Model Setup
    hidden_size: 128
    num_output_features: 48

    # VectorMapFeatureBuilder
    vector_map_feature_radius: 20    # [m] The query radius scope relative to the current ego-pose.

    # Parameters for past trajectory
    past_trajectory_sampling:
      _target_: nuplan.planning.simulation.trajectory.trajectory_sampling.TrajectorySampling
      _convert_: 'all'
      num_poses: 4      # target future poses
      time_horizon: 1.5  # [s] time horizon of future poses

    # Parameters for predicted trajectory
    future_trajectory_sampling:
      _target_: nuplan.planning.simulation.trajectory.trajectory_sampling.TrajectorySampling
      _convert_: 'all'
      num_poses: 16      # target future poses
      time_horizon: 8.0  # [s] time horizon of future poses
  checkpoint_path: your_ckpt_path