trust-ai / SafeBench

A Benchmark for Evaluating Autonomous Vehicles in Safety-critical Scenarios
https://safebench.github.io
MIT License
96 stars 21 forks source link

Will RL-Agent checkpoint be released? #6

Open Morphlng opened 1 year ago

Morphlng commented 1 year ago

Hi! Thank you for developing the SafeBench platform. which has been incredibly inspirational for me.

However, I am currently facing a challenge in training a valid RL-agent. When I use the command: python scripts/run.py --agent_cfg ppo.yaml --scenario_cfg standard.yaml --mode train_agent, the log indicates that no checkpoint was found, leading to training from scratch. Unfortunately, even after half a day of training, it appears that the agent has only learned to remain stationary in order to avoid crashes. Additionally, I've encountered some crashes during training, seem to be a loss of connection between the client and Carla. I am currently investigating whether this issue is specific to my setup.

Anyway, while reading the SafeBench paper, I noticed that a pretrained model from Town03 was used. I'm curious to know if there are any plans to release this model in the future. It would be incredibly valuable to have access to such a pretrained model.

Thank you once again for your work on SafeBench, and I appreciate any assistance you can provide in addressing my questions and concerns.