search

zhejz / carla-roach

Roach: End-to-End Urban Driving by Imitating a Reinforcement Learning Coach. ICCV 2021.
https://zhejz.github.io/roach
Other
274 stars 50 forks source link

About training RL expert #32

Closed Lesliewsq closed 12 months ago

Lesliewsq commented 12 months ago

Hello, I'm running run/train_rl.sh and meet this error. I don't know what the problem is.

CarlaUE4-Linux: 未找到进程
[2023-09-05 15:34:29,212][utils.server_utils][INFO] - Kill Carla Servers!
CarlaUE4-Linux: 未找到进程
[2023-09-05 15:34:30,225][utils.server_utils][INFO] - Kill Carla Servers!
[2023-09-05 15:34:30,225][utils.server_utils][INFO] - CUDA_VISIBLE_DEVICES=0 bash /home/tech/carla/CarlaUE4.sh -fps=10 -quality-level=Epic -carla-rpc-port=2000
[2023-09-05 15:34:30,232][utils.server_utils][INFO] - CUDA_VISIBLE_DEVICES=0 bash /home/tech/carla/CarlaUE4.sh -fps=10 -quality-level=Epic -carla-rpc-port=2005
[2023-09-05 15:34:30,238][utils.server_utils][INFO] - CUDA_VISIBLE_DEVICES=0 bash /home/tech/carla/CarlaUE4.sh -fps=10 -quality-level=Epic -carla-rpc-port=2010
[2023-09-05 15:34:30,242][utils.server_utils][INFO] - CUDA_VISIBLE_DEVICES=0 bash /home/tech/carla/CarlaUE4.sh -fps=10 -quality-level=Epic -carla-rpc-port=2015
[2023-09-05 15:34:30,246][utils.server_utils][INFO] - CUDA_VISIBLE_DEVICES=0 bash /home/tech/carla/CarlaUE4.sh -fps=10 -quality-level=Epic -carla-rpc-port=2020
[2023-09-05 15:34:30,250][utils.server_utils][INFO] - CUDA_VISIBLE_DEVICES=0 bash /home/tech/carla/CarlaUE4.sh -fps=10 -quality-level=Epic -carla-rpc-port=2025
4.24.3-0+++UE4+Release-4.24 518 0
Disabling core dumps.
4.24.3-0+++UE4+Release-4.24 518 0
Disabling core dumps.
4.24.3-0+++UE4+Release-4.24 518 0
Disabling core dumps.
4.24.3-0+++UE4+Release-4.24 518 0
Disabling core dumps.
4.24.3-0+++UE4+Release-4.24 518 0
Disabling core dumps.
4.24.3-0+++UE4+Release-4.24 518 0
Disabling core dumps.
Process ForkServerProcess-6:
Traceback (most recent call last):
  File "/home/tech/anaconda3/envs/carla/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/home/tech/anaconda3/envs/carla/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/home/tech/anaconda3/envs/carla/lib/python3.7/site-packages/stable_baselines3/common/vec_env/subproc_vec_env.py", line 13, in _worker
    env = env_fn_wrapper.var()
  File "train_rl.py", line 64, in <lambda>
    env = SubprocVecEnv([lambda config=config: env_maker(config) for config in server_manager.env_configs])
  File "train_rl.py", line 57, in env_maker
    seed=cfg.seed, no_rendering=True, **config['env_configs'])
  File "/home/tech/anaconda3/envs/carla/lib/python3.7/site-packages/gym/envs/registration.py", line 145, in make
    return registry.make(id, **kwargs)
  File "/home/tech/anaconda3/envs/carla/lib/python3.7/site-packages/gym/envs/registration.py", line 90, in make
    env = spec.make(**kwargs)
  File "/home/tech/anaconda3/envs/carla/lib/python3.7/site-packages/gym/envs/registration.py", line 60, in make
    env = cls(**_kwargs)
  File "/home/tech/carla-roach/carla_gym/envs/suites/endless_env.py", line 9, in __init__
    obs_configs, reward_configs, terminal_configs, all_tasks)
  File "/home/tech/carla-roach/carla_gym/carla_multi_agent_env.py", line 28, in __init__
    self._init_client(carla_map, host, port, seed=seed, no_rendering=no_rendering)
  File "/home/tech/carla-roach/carla_gym/carla_multi_agent_env.py", line 151, in _init_client
    self._world = client.load_world(carla_map)
RuntimeError: map not found
/home/tech/anaconda3/envs/carla/lib/python3.7/site-packages/gym/logger.py:30: UserWarning: WARN: Box bound precision lowered by casting to float32
  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))
/home/tech/anaconda3/envs/carla/lib/python3.7/site-packages/gym/logger.py:30: UserWarning: WARN: Box bound precision lowered by casting to float32
  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))
wandb: Currently logged in as: greatest-of-all-time (use `wandb login --relogin` to force relogin)
/home/tech/anaconda3/envs/carla/lib/python3.7/site-packages/gym/logger.py:30: UserWarning: WARN: Box bound precision lowered by casting to float32
  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))
/home/tech/anaconda3/envs/carla/lib/python3.7/site-packages/gym/logger.py:30: UserWarning: WARN: Box bound precision lowered by casting to float32
  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))
wandb: Tracking run with wandb version 0.10.12
wandb: Syncing run roach
wandb: ⭐️ View project at https://wandb.ai/greatest-of-all-time/train_rl_experts
wandb: 🚀 View run at https://wandb.ai/greatest-of-all-time/train_rl_experts/runs/22x9vtjs
wandb: Run data is saved locally in /home/tech/carla-roach/outputs/2023-09-05/15-34-28/wandb/run-20230905_153439-22x9vtjs
wandb: Run `wandb offline` to turn off syncing.

wandb: WARNING Symlinked 3 files into the W&B run directory, call wandb.save again to sync new files.
trainable parameters: 1.53M
Traceback (most recent call last):
  File "train_rl.py", line 74, in main
    agent.learn(env, total_timesteps=int(cfg.total_timesteps), callback=callback, seed=cfg.seed)
  File "/home/tech/carla-roach/agents/rl_birdview/rl_birdview_agent.py", line 107, in learn
    model.learn(total_timesteps, callback=callback, seed=seed)
  File "/home/tech/carla-roach/agents/rl_birdview/models/ppo.py", line 216, in learn
    self.env.seed(seed)
  File "/home/tech/anaconda3/envs/carla/lib/python3.7/site-packages/stable_baselines3/common/vec_env/subproc_vec_env.py", line 114, in seed
    remote.send(("seed", seed + idx))
  File "/home/tech/anaconda3/envs/carla/lib/python3.7/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/home/tech/anaconda3/envs/carla/lib/python3.7/multiprocessing/connection.py", line 404, in _send_bytes
    self._send(header + buf)
  File "/home/tech/anaconda3/envs/carla/lib/python3.7/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

wandb: Waiting for W&B process to finish, PID 943313
wandb: Program failed with code 1.  Press ctrl-c to abort syncing.
Process ForkServerProcess-5:loaded (0.00MB deduped)
Traceback (most recent call last):
  File "/home/tech/anaconda3/envs/carla/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/home/tech/anaconda3/envs/carla/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/home/tech/anaconda3/envs/carla/lib/python3.7/site-packages/stable_baselines3/common/vec_env/subproc_vec_env.py", line 13, in _worker
    env = env_fn_wrapper.var()
  File "train_rl.py", line 64, in <lambda>
    env = SubprocVecEnv([lambda config=config: env_maker(config) for config in server_manager.env_configs])
  File "train_rl.py", line 57, in env_maker
    seed=cfg.seed, no_rendering=True, **config['env_configs'])
  File "/home/tech/anaconda3/envs/carla/lib/python3.7/site-packages/gym/envs/registration.py", line 145, in make
    return registry.make(id, **kwargs)
  File "/home/tech/anaconda3/envs/carla/lib/python3.7/site-packages/gym/envs/registration.py", line 90, in make
    env = spec.make(**kwargs)
  File "/home/tech/anaconda3/envs/carla/lib/python3.7/site-packages/gym/envs/registration.py", line 60, in make
    env = cls(**_kwargs)
  File "/home/tech/carla-roach/carla_gym/envs/suites/endless_env.py", line 9, in __init__
    obs_configs, reward_configs, terminal_configs, all_tasks)
  File "/home/tech/carla-roach/carla_gym/carla_multi_agent_env.py", line 28, in __init__
    self._init_client(carla_map, host, port, seed=seed, no_rendering=no_rendering)
  File "/home/tech/carla-roach/carla_gym/carla_multi_agent_env.py", line 152, in _init_client
    self._tm = client.get_trafficmanager(port+6000)
RuntimeError: trying to create rpc server for traffic manager; but the system failed to create because of bind error.
wandb:                                                                                
wandb: Find user logs for this run at: /home/tech/carla-roach/outputs/2023-09-05/15-34-28/wandb/run-20230905_153439-22x9vtjs/logs/debug.log
wandb: Find internal logs for this run at: /home/tech/carla-roach/outputs/2023-09-05/15-34-28/wandb/run-20230905_153439-22x9vtjs/logs/debug-internal.log
wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 4 other file(s)
wandb: 
wandb: Synced roach: https://wandb.ai/greatest-of-all-time/train_rl_experts/runs/22x9vtjs
 PYTHON_RETURN=0!!! Start Over!!!
Bash script done.