Some errors encounted when running 'feature_reinforcement_learning' version of OpenCDA

[flammingRaven]

First thanks for open-sourcing OpenCDA framework, and the code of main version is significantly clear to read through! I am so interested in applying multi-agent deep reinforcement learning with co-simulation mode on it, so I turn to the 'feature_reinforcement_learning' version. When I run the command python opencda.py -t single_rldqn_town06_carla -rl train, I encounter the following problems:

• Using port 9000 cannot connect to my CARLA server, so I just set the client port to 2000, and it works.

• The CarlaRLEnv may lack of the definitions of 'observation_space', 'action_space', and 'reward_space' which cannot be passed to openDI engine, so I simply assign empty dict spaces.Dict({}) to these variables within the CarlaRLEnv and the errors are gone.

• Then I re-run the program and encounter the following problem, but I don't know how to solve it:

  Traceback (most recent call last):
  File "/home/ghz/PycharmProjects/OpenCDA-feature-reinforcement_learning/opencda.py", line 65, in <module>
  main()
  File "/home/ghz/PycharmProjects/OpenCDA-feature-reinforcement_learning/opencda.py", line 60, in main
  scenario_runner(opt, config_yaml)
  File "/home/ghz/PycharmProjects/OpenCDA-feature-reinforcement_learning/opencda/scenario_testing/single_rldqn_town06_carla.py", line 16, in run_scenario
  rl_train(opt, config_yaml)
  File "/home/ghz/PycharmProjects/OpenCDA-feature-reinforcement_learning/opencda/core/ml_libs/rl/rl_api.py", line 176, in rl_train
  tb_logger, exp_name=rl_cfg.exp_name)
  File "/home/ghz/anaconda3/envs/opencda/lib/python3.7/site-packages/ding/worker/replay_buffer/naive_buffer.py", line 81, in __init__
  self._instance_name, EasyDict(seconds=self._cfg.periodic_thruput_seconds), self._logger, self._tb_logger
  AttributeError: 'EasyDict' object has no attribute 'periodic_thruput_seconds'
  Exception ignored in: <function NaiveReplayBuffer.__del__ at 0x7f1f6c5df0e0>
  Traceback (most recent call last):
  File "/home/ghz/anaconda3/envs/opencda/lib/python3.7/site-packages/ding/worker/replay_buffer/naive_buffer.py", line 277, in __del__
  self.close()
  File "/home/ghz/anaconda3/envs/opencda/lib/python3.7/site-packages/ding/worker/replay_buffer/naive_buffer.py", line 97, in close
  self.clear()
  File "/home/ghz/anaconda3/envs/opencda/lib/python3.7/site-packages/ding/worker/replay_buffer/naive_buffer.py", line 268, in clear
  self._periodic_thruput_monitor.valid_count = self._valid_count
  AttributeError: 'NaiveReplayBuffer' object has no attribute '_periodic_thruput_monitor'
  terminate called without an active exception

• Plus, I also want to ask about the version of openDI-engine (ding) used in this RL version of OpenCDA, I download the latest version of '0.4.3'.

Thanks a lot!

[DerrickXuNu]

Thanks for using OpenCDA! The RL branch is not finished yet. We are still developing it, so there maybe some bugs.

[flammingRaven]

Thanks for your reply, and I will wait for the final RL branch. For the last problem, I just add a line in rl_api.py, and the previous error does not show again.

rl_cfg.policy.other.replay_buffer.periodic_thruput_seconds = 60 # it may be the default value in ding, I am not sure.
replay_buffer = NaiveReplayBuffer(rl_cfg.policy.other.replay_buffer,
                                  tb_logger, exp_name=rl_cfg.exp_name)

but unfortunately I have encountered another problem:

Traceback (most recent call last):
  File "/home/ghz/PycharmProjects/OpenCDA-feature-reinforcement_learning/opencda.py", line 65, in <module>
    main()
  File "/home/ghz/PycharmProjects/OpenCDA-feature-reinforcement_learning/opencda.py", line 60, in main
    scenario_runner(opt, config_yaml)
  File "/home/ghz/PycharmProjects/OpenCDA-feature-reinforcement_learning/opencda/scenario_testing/single_rldqn_town06_carla.py", line 16, in run_scenario
    rl_train(opt, config_yaml)
  File "/home/ghz/PycharmProjects/OpenCDA-feature-reinforcement_learning/opencda/core/ml_libs/rl/rl_api.py", line 193, in rl_train
    policy_kwargs={'eps': eps})
  File "/home/ghz/anaconda3/envs/opencda/lib/python3.7/site-packages/ding/worker/collector/sample_serial_collector.py", line 246, in collect
    timesteps = self._env.step(actions)
  File "/home/ghz/anaconda3/envs/opencda/lib/python3.7/site-packages/ding/envs/env_manager/base_env_manager.py", line 346, in step
    timesteps[env_id] = self._step(env_id, act)
  File "/home/ghz/anaconda3/envs/opencda/lib/python3.7/site-packages/ding/envs/env_manager/base_env_manager.py", line 380, in _step
    raise runtime_error
  File "/home/ghz/anaconda3/envs/opencda/lib/python3.7/site-packages/ding/envs/env_manager/base_env_manager.py", line 369, in _step
    return step_fn()
  File "/home/ghz/anaconda3/envs/opencda/lib/python3.7/site-packages/ding/envs/env_manager/base_env_manager.py", line 56, in wrapper
    raise e
  File "/home/ghz/anaconda3/envs/opencda/lib/python3.7/site-packages/ding/envs/env_manager/base_env_manager.py", line 54, in wrapper
    return func(*args, **kwargs)
  File "/home/ghz/anaconda3/envs/opencda/lib/python3.7/site-packages/ding/envs/env_manager/base_env_manager.py", line 364, in step_fn
    return self._envs[env_id].step(act)
  File "/home/ghz/PycharmProjects/OpenCDA-feature-reinforcement_learning/opencda/core/ml_libs/rl/envs/simple_carla_env_scenario_manager.py", line 527, in step
    control = self._hero_vehicle_manager.run_step()
  File "/home/ghz/PycharmProjects/OpenCDA-feature-reinforcement_learning/opencda/core/common/vehicle_manager.py", line 205, in run_step
    target_speed, target_pos = self.agent.run_rl_action_step()
  File "/home/ghz/PycharmProjects/OpenCDA-feature-reinforcement_learning/opencda/core/plan/rl_behavior_agent.py", line 408, in run_rl_action_step
    target_acc)        # a_end
  File "/home/ghz/PycharmProjects/OpenCDA-feature-reinforcement_learning/opencda/core/plan/rl_local_planner_behavior.py", line 158, in generate_path
    v_end, a_end, plan_time)
  File "/home/ghz/anaconda3/envs/opencda/lib/python3.7/site-packages/mistgen/mist.py", line 297, in mist_2d_gen
    self.polys_x,self.polys_y = self.minimum_snap_2d(waypts,self.ts,v0,a0,ve,ae)
  File "/home/ghz/anaconda3/envs/opencda/lib/python3.7/site-packages/mistgen/mist.py", line 234, in minimum_snap_2d
    polys_x = self.minimum_snap_single(waypts[0],ts,v0[0],a0[0],ve[0],ae[0])
  File "/home/ghz/anaconda3/envs/opencda/lib/python3.7/site-packages/mistgen/mist.py", line 148, in minimum_snap_single
    x = quadprog(Q_all,b_all,Aieq,bieq,Aeq,beq)
  File "/home/ghz/anaconda3/envs/opencda/lib/python3.7/site-packages/mistgen/utils/cvxopt_qp.py", line 41, in quadprog
    sol = cvxopt.solvers.qp(P, q, L, k, Aeq, beq)
  File "/home/ghz/anaconda3/envs/opencda/lib/python3.7/site-packages/cvxopt/coneprog.py", line 4485, in qp
    return coneqp(P, q, G, h, None, A,  b, initvals, kktsolver = kktsolver, options = options)
  File "/home/ghz/anaconda3/envs/opencda/lib/python3.7/site-packages/cvxopt/coneprog.py", line 2013, in coneqp
    raise ValueError("Rank(A) < p or Rank([P; A; G]) < n")
RuntimeError: Env 0 step has exceeded max retries(1), and the latest exception is: ValueError('Rank(A) < p or Rank([P; A; G]) < n')
ERROR: failed to destroy actor 281 : unable to destroy actor: not found 
WARNING: sensor object went out of the scope but the sensor is still alive in the simulation: Actor 279 (sensor.other.gnss) 
WARNING: sensor object went out of the scope but the sensor is still alive in the simulation: Actor 280 (sensor.other.imu) 
terminate called without an active exception

I would be very grateful If you could give me some guides. Thanks a lot!