search

michigan-traffic-lab / Dense-Deep-Reinforcement-Learning

This repo contains the code for paper "Dense reinforcement learning for safety validation of autonomous vehicles"
Other
317 stars 49 forks source link

d2rl_train.py gives error "trails did not complete" after 35 mins in all three experiments with the pre-collected data. #7

Open remoteblue opened 3 months ago

remoteblue commented 3 months ago

error.txt

Hello, following the README_D2RL_Train.md, the goal is to go through step 2 and step 3. The pre-collected data was downloaded and extracts to the subfolder per the requirements. Note: one deviation is the raylib version. in the requirements_d2rl_train.txt, raylib == 1.11. However, the closest raylib version is 1.13. therefore, requirements_d2rl_train.txt was updated to accept 1.13. Other than that, all are satisfied.

In step 2, do $python d2rl_train.py --yaml_conf ./d2rl_training/d2rl_train.yaml

And it runs for 20-35 mins in all 3 experiments. All experiments ends up with the following error msg. The error.txt is attached in the ticket too.

My computer is Linux machine. Please advise.

Current time: 2024-06-10 23:33:04 (running for 00:35:05.94) Memory usage on this node: 3.7/31.1 GiB Using FIFO scheduling algorithm. Resources requested: 0/20 CPUs, 0/1 GPUs, 0.0/17.09 GiB heap, 0.0/8.54 GiB objects (0.0/1.0 accelerator_type:G) Result logdir: /home/honghaotan/Desktop/github/Dense-Deep-Reinforcement-Learning/ray_results/2lane_400m_D2RL_Training Number of trials: 1/1 (1 ERROR) +------------------------+----------+----------------------+--------+------------------+---------+----------+----------------------+----------------------+--------------------+ | Trial name | status | loc | iter | total time (s) | ts | reward | episode_reward_max | episode_reward_min | episode_len_mean | |------------------------+----------+----------------------+--------+------------------+---------+----------+----------------------+----------------------+--------------------| | PPO_my_env_68750_00000 | ERROR | 192.168.86.30:127930 | 356 | 2068.43 | 2840880 | 74.0043 | 99.9485 | -100 | 10.6109 | +------------------------+----------+----------------------+--------+------------------+---------+----------+----------------------+----------------------+--------------------+ Number of errored trials: 1 +------------------------+--------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Trial name | # failures | error file | |------------------------+--------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------| | PPO_my_env_68750_00000 | 1 | /home/honghaotan/Desktop/github/Dense-Deep-Reinforcement-Learning/ray_results/2lane_400m_D2RL_Training/PPO_my_env_68750_00000_0_2024-06-10_22-57-58/error.txt | +------------------------+--------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

Traceback (most recent call last): File "d2rl_train.py", line 86, in tune.run( File "/home/honghaotan/Desktop/github/Dense-Deep-Reinforcement-Learning/d2rl/lib/python3.8/site-packages/ray/tune/tune.py", line 741, in run raise TuneError("Trials did not complete", incomplete_trials) ray.tune.error.TuneError: ('Trials did not complete', [PPO_my_env_68750_00000])