search

motional / nuplan-devkit

The devkit of the nuPlan dataset.
https://www.nuplan.org
Other
674 stars 129 forks source link

Question about Dashboard_agent.log location #172

Closed IvanChen777 closed 1 year ago

IvanChen777 commented 1 year ago

Hi, I have a question about dashboard_agent.log in nuplan_framework.ipynb. When I run the training or run the Simulate a trained ML planner for comparison, I met the following issues: 2022-10-28 14:44:50,199 WARNING worker.py:1257 -- (ip=128.178.17.178) The agent on node hossein-System-Product-Name failed to be restarted 5 times. There are 3 possible problems if you see this error.

  1. The dashboard might not display correct information on this node.
  2. Metrics on this node won't be reported.
  3. runtime_env APIs won't work. Check out the dashboard_agent.log to see the detailed failure messages.

Then I want to check this dashboard_agent.log, but I cannot find it. Does anyone know this issue? If so, please let me know. Thank you in advance:)

michael-motional commented 1 year ago

Hey @IvanChen777, this error is generated by the ray library that backs the default worker type that runs simulations. Often if the job succeeds you don't have to worry about it, but you can also use worker=single_machine_thread_pool which seems to give fewer problems.

I'll look into the notebook and share any findings I have as well!