grpc error: Deadline Exceeded

[jkewang]

hello, when I test submission with "docker-compose up --build" in my local machine, some senarios will output:

nuplan-devkit-simulation-1  | >
nuplan-devkit-simulation-1  | '
nuplan-devkit-simulation-1  | 2023-04-21 09:02:22,962 WARNING {/nuplan_devkit/nuplan/planning/simulation/runner/executor.py:123}  Failed Simulation.
nuplan-devkit-simulation-1  |  'Traceback (most recent call last):
nuplan-devkit-simulation-1  |   File "/nuplan_devkit/nuplan/planning/simulation/runner/executor.py", line 27, in run_simulation
nuplan-devkit-simulation-1  |     return sim_runner.run()
nuplan-devkit-simulation-1  |   File "/nuplan_devkit/nuplan/planning/simulation/runner/simulations_runner.py", line 113, in run
nuplan-devkit-simulation-1  |     trajectory = self.planner.compute_trajectory(planner_input)
nuplan-devkit-simulation-1  |   File "/nuplan_devkit/nuplan/planning/simulation/planner/abstract_planner.py", line 106, in compute_trajectory
nuplan-devkit-simulation-1  |     raise e
nuplan-devkit-simulation-1  |   File "/nuplan_devkit/nuplan/planning/simulation/planner/abstract_planner.py", line 103, in compute_trajectory
nuplan-devkit-simulation-1  |     trajectory = self.compute_planner_trajectory(current_input)
nuplan-devkit-simulation-1  |   File "/nuplan_devkit/nuplan/planning/simulation/planner/remote_planner.py", line 162, in compute_planner_trajectory
nuplan-devkit-simulation-1  |     trajectory = self._compute_trajectory(self._stub, current_input=current_input)
nuplan-devkit-simulation-1  |   File "/nuplan_devkit/nuplan/planning/simulation/planner/remote_planner.py", line 205, in _compute_trajectory
nuplan-devkit-simulation-1  |     trajectory_message = stub.ComputeTrajectory(planner_input, timeout=self._compute_trajectory_timeout)
nuplan-devkit-simulation-1  |   File "/opt/conda/envs/nuplan/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
nuplan-devkit-simulation-1  |     return _end_unary_response_blocking(state, call, False, None)
nuplan-devkit-simulation-1  |   File "/opt/conda/envs/nuplan/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
nuplan-devkit-simulation-1  |     raise _InactiveRpcError(state)
nuplan-devkit-simulation-1  | grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
nuplan-devkit-simulation-1  |   status = StatusCode.DEADLINE_EXCEEDED
nuplan-devkit-simulation-1  |   details = "Deadline Exceeded"
nuplan-devkit-simulation-1  |   debug_error_string = "{"created":"@1682067307.592177835","description":"Error received from peer ipv4:127.0.0.1:50051","file":"src/core/lib/surface/call.cc","file_line":1074,"grpc_message":"Deadline Exceeded","grpc_status":4}"

But some other scenarios work fine. I'm not familiar with GRPC. Has anyone had the same problem or can offer some advice?

PS: By the way, is it normal to stay in the "submission" state for a long time(3days) after submitting docker?

[JingyuQian]

@jkewang the organizer is triggering the run of submitted dockers one at a time, so it's normal to be in a "submitted" state for that long. Just FYI.

[patk-motional]

Hi @jkewang,

There is a time limit between each simulation step. It looks like that your planner is taking too long to produce a new trajectory.