search

huawei-noah / SMARTS

Scalable Multi-Agent RL Training School for Autonomous Driving
MIT License
952 stars 190 forks source link

Using all social agents and initialize their original position #1870

Closed yuant95 closed 1 year ago

yuant95 commented 1 year ago

Hi there!

I'm playing around with the bubble feature and social agents. I am able to create my own social agents and set up bubble to transfer the control from sumo to my own social agent in a map.

However I did notice that sometimes the social agents seem to lose control and get transferred back to sumo before they exit the bubble. Also my social agent behaivor is not so stable if it starts in the middle of the map (entering the bubble) without knowing the previous trajectory of its own.

Therefore I'm wondering whether there is a way to have only social agents in the map (without defining traffic and bubbles)? In addition, is there a way for me to initilize the location of the agents other than from the boundary of the map (like just directly place them in certain location with speed at the beginning of the simulation)?

Any help is appreicated!

Gamenot commented 1 year ago

Hello @yuant95, thank you for the questions.

Hi there!

I'm playing around with the bubble feature and social agents. I am able to create my own social agents and set up bubble to transfer the control from sumo to my own social agent in a map.

However I did notice that sometimes the social agents seem to lose control and get transferred back to sumo before they exit the bubble.

This is intended but perhaps confusing, the bubble features are only intended to keep control of a social agent until they exit the bubble if the agent interface given tells the simulation that it is "done". The done_criteria and max_episode_steps need to be configured.

Also my social agent behaivor is not so stable if it starts in the middle of the map (entering the bubble) without knowing the previous trajectory of its own.

We provide a margin around the bubble that creates an airlock that should allow for agents to receive historic information before being passed over to the agent. Observations are passed to the agent but all agent actions when in the airlock are ineffective. We also have an observation field under_this_agent_control which can be used to check while the agent is in the bubble margin/airlock.

Therefore I'm wondering whether there is a way to have only social agents in the map (without defining traffic and bubbles)? In addition, is there a way for me to initilize the location of the agents other than from the boundary of the map (like just directly place them in certain location with speed at the beginning of the simulation)?

Any help is appreicated!

This task is a bit more complicated. It is possible by specifying social agents to specific places in scenario studio, I think the better option would be to mass convert all of the vehicles in the traffic providers to agents.

I don't have time right at the moment to go into full detail. I will have a better reply later today.

yuant95 commented 1 year ago

Thanks so much for your reply!

Sorry I also have two more questions.

  1. When I use my customic social agents, sometimes they just dissappears in the bubble or right after they exit the bubble? I don't think they collide or off road. It would be really helpful if you have some clue about what happened.

  2. I also get assertion error when using my customer social agents after roughly 10 episodes like below. I assume something went wrong during the control hand over part between sumo and social agent? Is there anything I did wrong in setting up my social agents that causes this?

ERROR:root:TraCI has disconnected with: connection closed by SUMO
WARNING:SumoTrafficSimulation:attempting to transfer SUMO vehicles to other providers...
ERROR:SMARTS:Simulation crashed with exception. Attempting to cleanly shutdown.
ERROR:SMARTS:{'car-flow-route--E1_1_0--E4_1_max-248899497872519462--7266489842092764092--2-0.1', 'Agent_0-0c72fe13', 'car-flow-route-E4_1_0-E1_1_max--2352894271058968195--7266489842092764092--0-0.1'} in [VehicleState(actor_id='car-flow-route--E1_1_0--E4_1_max-248899497872519462--7266489842092764092--2-0.1', actor_type=None, source='LocalTrafficProvider', role=<ActorRole.Social: 1>, updated=False, vehicle_config_type='passenger', pose=Pose(position=array([32.19410324, 38.40062581,  0.        ]), orientation=array([ 0.        ,  0.        , -0.70705117,  0.70716239]), heading_=Heading(-1.570639044790708)), dimensions=Dimensions(length=3.68, width=1.47, height=1.4), speed=15.019651331922937, steering=None, yaw_rate=None, linear_velocity=None, angular_velocity=None, linear_acceleration=array([-9.17117717e-02, -1.44246114e-05]), angular_acceleration=None), VehicleState(actor_id='car-flow-route--E1_0_0--E4_0_max--2345656016055477228--7266489842092764092--1-0.1', actor_type=None, source='LocalTrafficProvider', role=<ActorRole.Social: 1>, updated=False, vehicle_config_type='passenger', pose=Pose(position=array([21.54514159, 35.19850092,  0.        ]), orientation=array([ 0.        ,  0.        , -0.70723654,  0.706977  ]), heading_=Heading(-1.5711633772185118)), dimensions=Dimensions(length=3.68, width=1.47, height=1.4), speed=13.53303759248366, steering=None, yaw_rate=None, linear_velocity=None, angular_velocity=None, linear_acceleration=array([-5.08045579e-02,  1.86478354e-05]), angular_acceleration=None)]
Traceback (most recent call last):
  File "/home/yuant426/miniconda3/envs/smartsEnvTest/lib/python3.8/site-packages/smarts/core/smarts.py", line 228, in step
    return self._step(agent_actions, time_delta_since_last_step)
  File "/home/yuant426/miniconda3/envs/smartsEnvTest/lib/python3.8/site-packages/smarts/core/smarts.py", line 283, in _step
    provider_state = self._step_providers(all_agent_actions)
  File "/home/yuant426/miniconda3/envs/smartsEnvTest/lib/python3.8/site-packages/smarts/core/smarts.py", line 1238, in _step_providers
    assert not isinstance(
AssertionError: {'car-flow-route--E1_1_0--E4_1_max-248899497872519462--7266489842092764092--2-0.1', 'Agent_0-0c72fe13', 'car-flow-route-E4_1_0-E1_1_max--2352894271058968195--7266489842092764092--0-0.1'} in [VehicleState(actor_id='car-flow-route--E1_1_0--E4_1_max-248899497872519462--7266489842092764092--2-0.1', actor_type=None, source='LocalTrafficProvider', role=<ActorRole.Social: 1>, updated=False, vehicle_config_type='passenger', pose=Pose(position=array([32.19410324, 38.40062581,  0.        ]), orientation=array([ 0.        ,  0.        , -0.70705117,  0.70716239]), heading_=Heading(-1.570639044790708)), dimensions=Dimensions(length=3.68, width=1.47, height=1.4), speed=15.019651331922937, steering=None, yaw_rate=None, linear_velocity=None, angular_velocity=None, linear_acceleration=array([-9.17117717e-02, -1.44246114e-05]), angular_acceleration=None), VehicleState(actor_id='car-flow-route--E1_0_0--E4_0_max--2345656016055477228--7266489842092764092--1-0.1', actor_type=None, source='LocalTrafficProvider', role=<ActorRole.Social: 1>, updated=False, vehicle_config_type='passenger', pose=Pose(position=array([21.54514159, 35.19850092,  0.        ]), orientation=array([ 0.        ,  0.        , -0.70723654,  0.706977  ]), heading_=Heading(-1.5711633772185118)), dimensions=Dimensions(length=3.68, width=1.47, height=1.4), speed=13.53303759248366, steering=None, yaw_rate=None, linear_velocity=None, angular_velocity=None, linear_acceleration=array([-5.08045579e-02,  1.86478354e-05]), angular_acceleration=None)]
Traceback (most recent call last):
  File "train/run.py", line 292, in <module>
    main(args)
  File "train/run.py", line 99, in main
    run(envs_train=envs_train, envs_eval=envs_eval, config=config, wandb_run = wandb_run)
  File "train/run.py", line 151, in run
    model.learn(
  File "/home/yuant426/.local/lib/python3.8/site-packages/stable_baselines3/ppo/ppo.py", line 299, in learn
    return super(PPO, self).learn(
  File "/home/yuant426/.local/lib/python3.8/site-packages/stable_baselines3/common/on_policy_algorithm.py", line 250, in learn
    continue_training = self.collect_rollouts(self.env, callback, self.rollout_buffer, n_rollout_steps=self.n_steps)
  File "/home/yuant426/.local/lib/python3.8/site-packages/stable_baselines3/common/on_policy_algorithm.py", line 178, in collect_rollouts
    new_obs, rewards, dones, infos = env.step(clipped_actions)
  File "/home/yuant426/.local/lib/python3.8/site-packages/stable_baselines3/common/vec_env/base_vec_env.py", line 162, in step
    return self.step_wait()
  File "/home/yuant426/.local/lib/python3.8/site-packages/stable_baselines3/common/vec_env/vec_monitor.py", line 76, in step_wait
    obs, rewards, dones, infos = self.venv.step_wait()
  File "/home/yuant426/.local/lib/python3.8/site-packages/stable_baselines3/common/vec_env/dummy_vec_env.py", line 43, in step_wait
    obs, self.buf_rews[env_idx], self.buf_dones[env_idx], self.buf_infos[env_idx] = self.envs[env_idx].step(
  File "/home/yuant426/miniconda3/envs/smartsEnvTest/lib/python3.8/site-packages/smarts/env/wrappers/single_agent.py", line 60, in step
    obs, reward, done, info = self.env.step({self._agent_id: action})
  File "/home/yuant426/.local/lib/python3.8/site-packages/gym/core.py", line 282, in step
    observation, reward, done, info = self.env.step(action)
  File "/home/yuant426/miniconda3/envs/smartsEnvTest/lib/python3.8/site-packages/smarts/env/wrappers/frame_stack.py", line 90, in step
    env_observations, rewards, dones, infos = super(FrameStack, self).step(
  File "/home/yuant426/.local/lib/python3.8/site-packages/gym/core.py", line 248, in step
    return self.env.step(action)
  File "/home/yuant426/.local/lib/python3.8/site-packages/gym/core.py", line 282, in step
    observation, reward, done, info = self.env.step(action)
  File "/home/yuant426/.local/lib/python3.8/site-packages/gym/core.py", line 308, in step
    return self.env.step(self.action(action))
  File "/home/yuant426/.local/lib/python3.8/site-packages/gym/core.py", line 282, in step
    observation, reward, done, info = self.env.step(action)
  File "/home/yuant426/Desktop/SMARTS_track1/competition/track1/train/train/reward.py", line 27, in step
    obs, reward, done, info = self.env.step(action)
  File "/home/yuant426/Desktop/SMARTS_track1/competition/track1/train/train/info.py", line 28, in step
    obs, reward, done, info = self.env.step(action)
  File "/home/yuant426/Desktop/SMARTS_track1/competition/track1/train/train/history.py", line 83, in step
    env_observations, rewards, dones, infos = super(HistoryStack, self).step(
  File "/home/yuant426/.local/lib/python3.8/site-packages/gym/core.py", line 248, in step
    return self.env.step(action)
  File "/home/yuant426/.local/lib/python3.8/site-packages/gym/core.py", line 308, in step
    return self.env.step(self.action(action))
  File "/home/yuant426/.local/lib/python3.8/site-packages/gym/core.py", line 282, in step
    observation, reward, done, info = self.env.step(action)
  File "/home/yuant426/miniconda3/envs/smartsEnvTest/lib/python3.8/site-packages/smarts/env/multi_scenario_env.py", line 498, in step
    obs, reward, done, info = self.env.step(action)
  File "/home/yuant426/miniconda3/envs/smartsEnvTest/lib/python3.8/site-packages/smarts/env/multi_scenario_env.py", line 402, in step
    out = self.env.step(limited_actions)
  File "/home/yuant426/miniconda3/envs/smartsEnvTest/lib/python3.8/site-packages/smarts/env/hiway_env.py", line 233, in step
    observations, rewards, dones, extras = self._smarts.step(agent_actions)
  File "/home/yuant426/miniconda3/envs/smartsEnvTest/lib/python3.8/site-packages/smarts/core/smarts.py", line 228, in step
    return self._step(agent_actions, time_delta_since_last_step)
  File "/home/yuant426/miniconda3/envs/smartsEnvTest/lib/python3.8/site-packages/smarts/core/smarts.py", line 283, in _step
    provider_state = self._step_providers(all_agent_actions)
  File "/home/yuant426/miniconda3/envs/smartsEnvTest/lib/python3.8/site-packages/smarts/core/smarts.py", line 1238, in _step_providers
    assert not isinstance(
AssertionError: {'car-flow-route--E1_1_0--E4_1_max-248899497872519462--7266489842092764092--2-0.1', 'Agent_0-0c72fe13', 'car-flow-route-E4_1_0-E1_1_max--2352894271058968195--7266489842092764092--0-0.1'} in [VehicleState(actor_id='car-flow-route--E1_1_0--E4_1_max-248899497872519462--7266489842092764092--2-0.1', actor_type=None, source='LocalTrafficProvider', role=<ActorRole.Social: 1>, updated=False, vehicle_config_type='passenger', pose=Pose(position=array([32.19410324, 38.40062581,  0.        ]), orientation=array([ 0.        ,  0.        , -0.70705117,  0.70716239]), heading_=Heading(-1.570639044790708)), dimensions=Dimensions(length=3.68, width=1.47, height=1.4), speed=15.019651331922937, steering=None, yaw_rate=None, linear_velocity=None, angular_velocity=None, linear_acceleration=array([-9.17117717e-02, -1.44246114e-05]), angular_acceleration=None), VehicleState(actor_id='car-flow-route--E1_0_0--E4_0_max--2345656016055477228--7266489842092764092--1-0.1', actor_type=None, source='LocalTrafficProvider', role=<ActorRole.Social: 1>, updated=False, vehicle_config_type='passenger', pose=Pose(position=array([21.54514159, 35.19850092,  0.        ]), orientation=array([ 0.        ,  0.        , -0.70723654,  0.706977  ]), heading_=Heading(-1.5711633772185118)), dimensions=Dimensions(length=3.68, width=1.47, height=1.4), speed=13.53303759248366, steering=None, yaw_rate=None, linear_velocity=None, angular_velocity=None, linear_acceleration=array([-5.08045579e-02,  1.86478354e-05]), angular_acceleration=None)]
Gamenot commented 1 year ago

Hello @yuant95, I appologise that I have not yet been able to follow up properly on your question about large scale vehicle-replacement/agent-generation.

Thanks so much for your reply!

Sorry I also have two more questions.

  1. When I use my customic social agents, sometimes they just dissappears in the bubble or right after they exit the bubble? I don't think they collide or off road. It would be really helpful if you have some clue about what happened.

Could you provide some additional information so I can determine what the issue might be? The more I get the easier it will be for me to solve:

  1. I also get assertion error when using my customer social agents after roughly 10 episodes like below. I assume something went wrong during the control hand over part between sumo and social agent? Is there anything I did wrong in setting up my social agents that causes this?
logs...

This appears to be a bug, I am adding test cases to try and weed this problem out.

Gamenot commented 1 year ago

Therefore I'm wondering whether there is a way to have only social agents in the map (without defining traffic and bubbles)? In addition, is there a way for me to initilize the location of the agents other than from the boundary of the map (like just directly place them in certain location with speed at the beginning of the simulation)?

Modifying Ego Agents

It is possible to modify the agents. This has the downside of changing the environment action space.

Observing from a vehicle

If the vehicle has sensors already you should be able to do this:

from smarts.core.smarts import SMARTS

smarts: SMARTS = env._smarts # `env.smarts` in the newest version
vehicle_ids = { "vehicle_id_1", "vehicle_id_2", ... }
## The rewards and scores here are volatile and may be changed in future
observations, rewards, scores, dones = smarts.observe_from(
    vehicle_ids=vehicle_ids
)

If the vehicle needs sensors you can do the following:

from smarts.core.smarts import SMARTS

smarts: SMARTS = env._smarts # `env.smarts` in the newest version

vehicle_ids = { "vehicle_id_1", "vehicle_id_2", ... }
smarts.attach_sensors_to_vehicles(agent_interface, vehicle_ids)
...

## not currently an interface through smarts to detach sensors
agent_manager: AgentManager = smarts.agent_manager
vehicle_index: VehicleIndex = smarts.vehicle_index
for vehicle_id in vehicle_ids:
    # Be careful that you don't detach sensors from a vehicle
    # an agent is attempting to use.
    if (
        self.vehicle_index.vehicle_is_hijacked(vehicle_id)
        or vehicle_id in self.vehicle_index.agent_vehicle_ids()
    ):
        continue
    agent_manager.detach_sensors_from_vehicle(vehicle_id)

add_agent_and_switch_control

The first option lets you take over an existing vehicle immediately with an agent.

from smarts.core.smarts import SMARTS
from smarts.core.plan import Mission
from smarts.core.vehicle_index import VehicleIndex

smarts: SMARTS = env._smarts # `env.smarts` in the newest version
vehicle_index: VehicleIndex = smarts.vehicle_index
vehicle = vehicle_index.vehicle_by_id(vehicle_id, default=None)
mission: Mission = Mission.endless_mission(vehicle.pose)
AGENT_ID = "new_agent_id"
smarts.add_agent_and_switch_control(vehicle.id, AGENT_ID, agent_interface, mission)
...
# actions will want you to provide an action for AGENT_ID
obs, ... = env.step({AGENT_ID: action , ...})

add_agent_with_mission

This one attempts to grab vehicles from a location depending on the mission configuration but it is more involved. It also modifies the ego agents.

from smarts.core.plan import Mission
from smarts.core.road_map import RoadMap

smarts: SMARTS = env.smarts
road_map: RoadMap = smarts.scenario.road_map
mission = Mission(
    start=Start.from_pose(pose),
    route_vias={}, # a set of road ids between start and goal for routing
    goal=PositionalGoal.from_road(end_road_id, road_map),
    start_time=smarts.elapsed_sim_time + 2, # 2 simulation seconds after the current sim time
    entry_tactic=TrapEntryTactic(
        wait_to_hijack_limit_s=0.2, # attempts to find existing vehicle for 0.2s
        exclusion_prefixes=("agent_",), # ignores vehicles with the given vehicle_id prefixes
        default_entry_speed=20, # default mps speed if a default vehicle is emitted.
    ), 
)
AGENT_ID = "new_agent_id"

smarts.add_agent_with_mission(AGENT_ID, agent_interface, mission)

You can use the road map to discover the information necessary.

Scenario Studio

TODO

Future interfaces

I will try adding these interfaces next week since they can simplify some issues

Finding vehicles from a specific provider (Added)

This is an interface I am going to add in shortly for getting actor ids from a specific provider.

from smarts.core.smarts import SMARTS
from smarts.core.provider import Provider
from smarts.core.sumo_traffic_simulation import SumoTrafficSimulation
# from smarts.core.local_traffic_provider import LocalTrafficProvider

smarts: SMARTS = env.smarts # in the new version
provider: Provider = smarts.get_provider_by_type(SumoTrafficSimulation)
actor_ids = provider.actor_ids # These will be vehicle ids

Adding a social agent

I am considering some of the following:

# agent_ids = smarts.add_social_agents(agent_locator=locator, count=4)
agent_ids = smarts.add_social_agents(agent_spec=agent_spec, count=4)
...
# set social agent to receive observations but not control vehicle
smarts.shadow_vehicle(vehicle_id, agent_id)
...
# set social agent to take control
smarts.switch_control_to_agent(vehicle_id, agent_id, mission, false, false)

or

agent_ids, rejected_vehicle_ids = smarts.control_actors_with_social_agents(agent_locator, vehicle_ids)
Gamenot commented 1 year ago

Closed by mistake.

yuant95 commented 1 year ago

Thanks for you reply! I've been trying with the bubble and my customatic social agents now. I defined my social agents as a boid agent as below. 

class invertedAiBoidAgent(Agent):
    def __init__(self):
        self.location = ITRA_MAP_LOCATION
        self.recurrent_states = {}
        self.offset = [50, 20]
        super().__init__()

    def act(self, obs):
        if len(obs) > 0 :
            agent_ids, agent_states, agent_attributes = self.get_iai_agents(obs)
            # This is hack to provide initial recurrent state to ITRA
            if len(self.recurrent_states) == 0:
                recurrent_states = [RecurrentState() for _ in range(len(agent_states))]
                for recurrent, st in zip(recurrent_states, agent_states):
                    recurrent.packed[-4:] = [st.center.x, st.center.y, st.orientation, st.speed]

            else:
                recurrent_states = []
                for index, agent_id in enumerate(agent_ids):
                    if agent_id in self.recurrent_states:
                        recurrent_states.append(self.recurrent_states[agent_id])
                    else:
                        recurrent_state = RecurrentState()
                        recurrent_state.packed[-4:] = [agent_states[index].center.x, agent_states[index].center.y, agent_states[index].orientation, agent_states[index].speed]
                        recurrent_states.append(recurrent_state)

            tries = 50
            for i in range(0,tries):
                try:
                    res = iai.api.drive(
                        location=self.location, 
                        agent_states=agent_states, 
                        agent_attributes=agent_attributes, 
                        recurrent_states=recurrent_states,
                        get_birdview=False)
                except Exception as e:
                    if i < tries - 1: # i is zero indexed
                        print("Exception raised from iai.api.drive： {}".format(str(e)))
                        print("Retrying sending the request {} times...".format(str(i)))
                        continue

                    else:
                        raise e
                else:
                    break

            # Code for export birdview for debugging

            # birdview = res.birdview.decode()
            # fig, ax = plt.subplots(constrained_layout=True, figsize=(5, 5))
            # ax.set_axis_off(), ax.imshow(birdview)

            for index, agent_id in enumerate(obs):
                self.recurrent_states[agent_id] = res.recurrent_states[index]

            actions = {vehicle_id: self.get_action(res.agent_states[agent_ids.index(vehicle_id)]) for index, vehicle_id in enumerate(obs)}
            return actions
        else:
            return {}

    def get_iai_agents(self, obs):
        agent_states = []
        agent_attributes = []
        agent_ids = []
        for vehicle_id, obs_ in obs.items():
            agent_ids.append(vehicle_id)

            ego_center = iai.common.Point(x=float(obs_.ego_vehicle_state.position[0]-self.offset[0]), y=float(obs_.ego_vehicle_state.position[1]-self.offset[1]))
            ego_state = iai.common.AgentState(center=ego_center, orientation=float(obs_.ego_vehicle_state.heading+np.pi/2.0), speed=float(obs_.ego_vehicle_state.speed))
            agent_states.append(ego_state)

            ego_length = float(obs_.ego_vehicle_state.bounding_box.length)
            ego_width = float(obs_.ego_vehicle_state.bounding_box.width)
            rear_axis_offset = ego_length * 0.4
            ego_attri = iai.common.AgentAttributes(length=ego_length, width=ego_width, rear_axis_offset=rear_axis_offset)

            agent_attributes.append(ego_attri)

        for vehicle_id, obs_ in obs.items():
            if (obs_.neighborhood_vehicle_states):
                neighbors = obs_.neighborhood_vehicle_states
                for i in range(len(neighbors)):
                    if neighbors[i].id not in agent_ids:
                        agent_ids.append(neighbors[i].id)

                        center = iai.common.Point(x=float(neighbors[i].position[0]-self.offset[0]), y=float(neighbors[i].position[1]-self.offset[1]))
                        orientation = float(neighbors[i].heading+np.pi/2.0)
                        speed = float(neighbors[i].speed)

                        length = float(neighbors[i].bounding_box.length)
                        width = float(neighbors[i].bounding_box.width)
                        rear_axis_offset = length * 0.4

                        state = iai.common.AgentState(center=center, orientation=orientation, speed=speed)
                        attri = iai.common.AgentAttributes(length=length, width=width, rear_axis_offset=rear_axis_offset)

                        agent_states.append(state)
                        agent_attributes.append(attri)

        return agent_ids, agent_states, agent_attributes

    def get_action(self, agent_state):

        # Currently only use the prediction of the ego car as action
        # Could controll all agents in the neighbors as well in the future. 
        action = [agent_state.center.x+self.offset[0], agent_state.center.y+self.offset[1], agent_state.orientation-np.pi/2.0]
        time_delta = 0.1
        action = np.array(
                        [action[0], action[1], action[2], time_delta], dtype=np.float32
                    )

        return action

and in the scenario I defined

invertedai_boid_agent = t.BoidAgentActor(
    name="invertedai-boid-agent",
    agent_locator=f"{agent_prefabs}:inverted-boid-agent-v0",
)

However, after successfully running one episode, I always run into the error below. 

ERROR:MotionPlannerProvider:actions specified for an agent without an actor: {'BUBBLE-AGENT-invertedai-boid-agent'}. Cleaning up social agents.
ERROR:SMARTS:Simulation crashed with exception. Attempting to cleanly shutdown.
ERROR:SMARTS:'BUBBLE-AGENT-invertedai-boid-agent'
Traceback (most recent call last):
  File "/home/yuant426/miniconda3/envs/smartsEnvTest/lib/python3.8/site-packages/smarts/core/smarts.py", line 228, in step
    return self._step(agent_actions, time_delta_since_last_step)
  File "/home/yuant426/miniconda3/envs/smartsEnvTest/lib/python3.8/site-packages/smarts/core/smarts.py", line 283, in _step
    provider_state = self._step_providers(all_agent_actions)
  File "/home/yuant426/miniconda3/envs/smartsEnvTest/lib/python3.8/site-packages/smarts/core/smarts.py", line 1214, in _step_providers
    agent_actions, vehicle_actions = self._provider_actions(provider, actions)
  File "/home/yuant426/miniconda3/envs/smartsEnvTest/lib/python3.8/site-packages/smarts/core/smarts.py", line 1192, in _provider_actions
    interface = self._agent_manager.agent_interface_for_agent_id(agent_id)
  File "/home/yuant426/miniconda3/envs/smartsEnvTest/lib/python3.8/site-packages/smarts/core/agent_manager.py", line 113, in agent_interface_for_agent_id
    return self._agent_interfaces[agent_id]
KeyError: 'BUBBLE-AGENT-invertedai-boid-agent'

This only happens when I try to use my customatic boid agent, when I use just a customatic social agent (similar implementation as above but without boid). The exception doesn't happen.

Another usual seen error is as below. It doesn't kill the training, but I'm a bit concerned that whether this will influence the training quality. 

ERROR:root:TraCI has disconnected with: Route replacement failed for car-flow-route--E1_1_0--E4_1_max-248899497872519462--7266489842092764092--2-0.0
Gamenot commented 1 year ago

@yuant95 Sorry for the issues. Boid agents (and bubbles) are useful for your work but you may need to wait for the attached PRs while I work out the bugs.

I also just realized that you are attempting to drive the simulation using a separate engine. The provider interface is made for that.

smarts.add_provider(custom_provider)

SMARTS will handle calls to the following provider methods:

Provider needs to handle the following:

ExternalProvider

This is a special provider that allows injection of vehicle state into the simulator. This state then takes precedence over internal SMARTS simulation state. 

from smarts.core.external_provider import ExternalProvider

external_provider = smarts.get_provider_by_type(ExternalProvider) # or smarts.external_provider
external_provider.state_update(vehicle_states)

Additional context

Vehicle states that are not marked external are available to SMARTS to take over with social agents so ActorState.External may be required if that behaviour is not desired.

Vehicle state is defined: https://github.com/huawei-noah/SMARTS/blob/15909df059f84f414290450c45f1b58705e04861/smarts/core/vehicle.py#L59-L72

And vehicle config: https://github.com/huawei-noah/SMARTS/blob/15909df059f84f414290450c45f1b58705e04861/smarts/core/vehicle.py#L106-L149

Traffic provider

Traffic provider is a provider subtype that has additional features. It has to be added using the SMARTS constructor:

SMARTS(
    ...,
    traffic_sims=[SumoTrafficSimulation(), CustomTrafficProvider()],
)

https://github.com/huawei-noah/SMARTS/blob/15909df059f84f414290450c45f1b58705e04861/smarts/core/traffic_provider.py#L30-L66

Gamenot commented 1 year ago

I am hoping to wrap up the bubble related stability issues late next week.

Gamenot commented 1 year ago

Thanks for you reply! I've been trying with the bubble and my customatic social agents now. I defined my social agents as a boid agent as below.

code...

However, after successfully running one episode, I always run into the error below.

ERROR:MotionPlannerProvider:actions specified for an agent without an actor: {'BUBBLE-AGENT-invertedai-boid-agent'}. Cleaning up social agents.
ERROR:SMARTS:Simulation crashed with exception. Attempting to cleanly shutdown.
ERROR:SMARTS:'BUBBLE-AGENT-invertedai-boid-agent'
Traceback (most recent call last):
  File "/home/yuant426/miniconda3/envs/smartsEnvTest/lib/python3.8/site-packages/smarts/core/smarts.py", line 228, in step
    return self._step(agent_actions, time_delta_since_last_step)
  File "/home/yuant426/miniconda3/envs/smartsEnvTest/lib/python3.8/site-packages/smarts/core/smarts.py", line 283, in _step
    provider_state = self._step_providers(all_agent_actions)
  File "/home/yuant426/miniconda3/envs/smartsEnvTest/lib/python3.8/site-packages/smarts/core/smarts.py", line 1214, in _step_providers
    agent_actions, vehicle_actions = self._provider_actions(provider, actions)
  File "/home/yuant426/miniconda3/envs/smartsEnvTest/lib/python3.8/site-packages/smarts/core/smarts.py", line 1192, in _provider_actions
    interface = self._agent_manager.agent_interface_for_agent_id(agent_id)
  File "/home/yuant426/miniconda3/envs/smartsEnvTest/lib/python3.8/site-packages/smarts/core/agent_manager.py", line 113, in agent_interface_for_agent_id
    return self._agent_interfaces[agent_id]
KeyError: 'BUBBLE-AGENT-invertedai-boid-agent'

This only happens when I try to use my customatic boid agent, when I use just a customatic social agent (similar implementation as above but without boid). The exception doesn't happen.

This appears to have been due to overeagerly removing the agents. There is a patch #1886 going in for it.

Another usual seen error is as below. It doesn't kill the training, but I'm a bit concerned that whether this will influence the training quality.

ERROR:root:TraCI has disconnected with: Route replacement failed for car-flow-route--E1_1_0--E4_1_max-248899497872519462--7266489842092764092--2-0.0

We are unable to to much about that because it has to do with terminal SUMO error, it should occur with low frequency but it could cause some issues with training since it may release all vehicles. It will, however, still reset and use sumo again in the next episode.

If you want to invalidate you can check if the SUMO provider is still connected by calling smarts.get_provider_by_type(smarts.core.SumoTrafficSimulation).connected.

yuant95 commented 1 year ago

Thanks so much! I think I get rid of the removing agents error once I pull the patch! I'm taking a look at the provider interface right now. However, there is another error popping up after 3hr ish of training using the boid agent I defined before. 

  File "train/run.py", line 293, in <module>
    main(args)
  File "train/run.py", line 100, in main
    run(envs_train=envs_train, envs_eval=envs_eval, config=config, wandb_run = wandb_run)
  File "train/run.py", line 152, in run
    model.learn(
  File "/home/yuant426/.local/lib/python3.8/site-packages/stable_baselines3/ppo/ppo.py", line 299, in learn
    return super(PPO, self).learn(
  File "/home/yuant426/.local/lib/python3.8/site-packages/stable_baselines3/common/on_policy_algorithm.py", line 250, in learn
    continue_training = self.collect_rollouts(self.env, callback, self.rollout_buffer, n_rollout_steps=self.n_steps)
  File "/home/yuant426/.local/lib/python3.8/site-packages/stable_baselines3/common/on_policy_algorithm.py", line 178, in collect_rollouts
    new_obs, rewards, dones, infos = env.step(clipped_actions)
  File "/home/yuant426/.local/lib/python3.8/site-packages/stable_baselines3/common/vec_env/base_vec_env.py", line 162, in step
    return self.step_wait()
  File "/home/yuant426/.local/lib/python3.8/site-packages/stable_baselines3/common/vec_env/vec_monitor.py", line 76, in step_wait
    obs, rewards, dones, infos = self.venv.step_wait()
  File "/home/yuant426/.local/lib/python3.8/site-packages/stable_baselines3/common/vec_env/dummy_vec_env.py", line 43, in step_wait
    obs, self.buf_rews[env_idx], self.buf_dones[env_idx], self.buf_infos[env_idx] = self.envs[env_idx].step(
  File "/home/yuant426/miniconda3/envs/smartsEnvTest/lib/python3.8/site-packages/smarts/env/wrappers/single_agent.py", line 60, in step
    obs, reward, done, info = self.env.step({self._agent_id: action})
  File "/home/yuant426/.local/lib/python3.8/site-packages/gym/core.py", line 282, in step
    observation, reward, done, info = self.env.step(action)
  File "/home/yuant426/miniconda3/envs/smartsEnvTest/lib/python3.8/site-packages/smarts/env/wrappers/frame_stack.py", line 90, in step
    env_observations, rewards, dones, infos = super(FrameStack, self).step(
  File "/home/yuant426/.local/lib/python3.8/site-packages/gym/core.py", line 248, in step
    return self.env.step(action)
  File "/home/yuant426/.local/lib/python3.8/site-packages/gym/core.py", line 282, in step
    observation, reward, done, info = self.env.step(action)
  File "/home/yuant426/.local/lib/python3.8/site-packages/gym/core.py", line 308, in step
    return self.env.step(self.action(action))
  File "/home/yuant426/.local/lib/python3.8/site-packages/gym/core.py", line 282, in step
    observation, reward, done, info = self.env.step(action)
  File "/home/yuant426/Desktop/SMARTS_track1/competition/track1/train/train/reward.py", line 27, in step
    obs, reward, done, info = self.env.step(action)
  File "/home/yuant426/Desktop/SMARTS_track1/competition/track1/train/train/info.py", line 28, in step
    obs, reward, done, info = self.env.step(action)
  File "/home/yuant426/Desktop/SMARTS_track1/competition/track1/train/train/history.py", line 83, in step
    env_observations, rewards, dones, infos = super(HistoryStack, self).step(
  File "/home/yuant426/.local/lib/python3.8/site-packages/gym/core.py", line 248, in step
    return self.env.step(action)
  File "/home/yuant426/.local/lib/python3.8/site-packages/gym/core.py", line 308, in step
    return self.env.step(self.action(action))
  File "/home/yuant426/.local/lib/python3.8/site-packages/gym/core.py", line 282, in step
    observation, reward, done, info = self.env.step(action)
  File "/home/yuant426/miniconda3/envs/smartsEnvTest/lib/python3.8/site-packages/smarts/env/multi_scenario_env.py", line 498, in step
    obs, reward, done, info = self.env.step(action)
  File "/home/yuant426/miniconda3/envs/smartsEnvTest/lib/python3.8/site-packages/smarts/env/multi_scenario_env.py", line 402, in step
    out = self.env.step(limited_actions)
  File "/home/yuant426/miniconda3/envs/smartsEnvTest/lib/python3.8/site-packages/smarts/env/hiway_env.py", line 235, in step
    observations, rewards, dones, extras = self._smarts.step(agent_actions)
  File "/home/yuant426/miniconda3/envs/smartsEnvTest/lib/python3.8/site-packages/smarts/core/smarts.py", line 228, in step
    return self._step(agent_actions, time_delta_since_last_step)
  File "/home/yuant426/miniconda3/envs/smartsEnvTest/lib/python3.8/site-packages/smarts/core/smarts.py", line 292, in _step
    self._bubble_manager.step(self)
  File "/home/yuant426/miniconda3/envs/smartsEnvTest/lib/python3.8/site-packages/smarts/core/bubble_manager.py", line 497, in step
    self._handle_transitions(sim, self._cursors)
  File "/home/yuant426/miniconda3/envs/smartsEnvTest/lib/python3.8/site-packages/smarts/core/bubble_manager.py", line 574, in _handle_transitions
    sim.vehicle_exited_bubble(cursor.vehicle_id, teardown)
  File "/home/yuant426/miniconda3/envs/smartsEnvTest/lib/python3.8/site-packages/smarts/core/smarts.py", line 671, in vehicle_exited_bubble
    self._agent_manager.detach_sensors_from_vehicle(vehicle_id)
  File "/home/yuant426/miniconda3/envs/smartsEnvTest/lib/python3.8/site-packages/smarts/core/agent_manager.py", line 671, in detach_sensors_from_vehicle
    self._vehicle_index.stop_agent_observation(vehicle_id)
  File "/home/yuant426/miniconda3/envs/smartsEnvTest/lib/python3.8/site-packages/smarts/core/utils/cache.py", line 136, in wrapper
    return func(self, *args, **kwargs)
  File "/home/yuant426/miniconda3/envs/smartsEnvTest/lib/python3.8/site-packages/smarts/core/vehicle_index.py", line 461, in stop_agent_observation
    vehicle = self._vehicles[vehicle_id]
KeyError: b'00000000000000000000000000000000000000000000000$car-flow-route--E1_1_0--E4_1_max-248899497872519462--7266489842092764092--2-0.0'

It seems like the vehicle is removed due to cannot find provider for it, but was later referred again? Is there any quick fix for this I can use so I can continue testing the training while figuring out the provider interface?

Thanks!

Gamenot commented 1 year ago

Hello @yuant95, I am pushing a fix for the new error.

Gamenot commented 1 year ago

https://github.com/huawei-noah/SMARTS/pull/1894 fixed the former bug.

Gamenot commented 1 year ago

I should note that it looks like removing vehicles during the provider update easily causes problems in the simulator. We may need to lock out or quarantine direct vehicle index access in provider classes.

yuant95 commented 1 year ago

Thanks! I just poulled the fix and it seems to resolve the exception I ran into before. However, I did notice that a lot of social agents were removed frequently, which means in the ego car's birdview, these social agents just freezed on the lane. I'm a bit concerned this would impact the RL training as the ego car doesn't get to interact with these social agents enough. Is there a way no reduce the frequency of this condition?

yuant95 commented 1 year ago

Hi @Gamenot! Thanks so much for all the timely reponses for my questions! I'm wondering wether we could set up some sort of zoom meeting? I feel like I have some confusion about how to set up social agents (or maybe I should use provider?) with third party engine. I wonder whether this is the reason that I ran into so many unstabilities in the simulation. I would really appreciate it if you could meet up with me for a Q&A session. I'm available for Monday afternoon, Tuesday, Wednesday, Thursday night, Friday night. Just let me know if this is possible. Thanks so much!

Gamenot commented 1 year ago

Hello @yuant95, I can help you out next week Tuesday or Wednesday.

yuant95 commented 1 year ago

Great! What time on Tuesday(March 14th) would work for you? Would 11 am PST work? Here is the google meet link https://meet.google.com/ysa-yivb-nab. Thanks!

Gamenot commented 1 year ago

11 PST works for me.

Gamenot commented 1 year ago

Notes

TODO

Problems

Adaickalavan commented 1 year ago
  • Fork errors with parallel training
    • @Adaickalavan Have you seen any success with parallel training recently?
yuant95 commented 1 year ago

  • Fork errors with parallel training

    • @Adaickalavan Have you seen any success with parallel training recently?
  • I did not perform parallel training using SMARTS environment recently.
  • Did this happen with SMARTS/examples/control/parallel_environment.py ?

This happens when I try to set up SubprocVecEnv from stable baseline library, which is the provided RL training script example in the competition branch. The error msg I'm recieving is AssertionError: daemonic processes are not allowed to have children. I'm wondering whether this is due to smarts itself start subprocess? Let me know if you need any more information for this error.

Adaickalavan commented 1 year ago

Hi @yuant95,

Gamenot commented 1 year ago

@yuant95 I looked into the stopping vehicles and cannot seem to reproduce the issue with vehicles stopping. It might be isolated to changes related to your branch.

The most I saw was sudden braking watching the images through visdom. There is some possibility that is the cause so I am going to check to see if there is any way where a vehicle could get locked at standstill.

yuant95 commented 1 year ago

@yuant95 I looked into the stopping vehicles and cannot seem to reproduce the issue with vehicles stopping. It might be isolated to changes related to your branch.

The most I saw was sudden braking watching the images through visdom. There is some possibility that is the cause so I am going to check to see if there is any way where a vehicle could get locked at standstill.

@Gamenot Let me try to see whether I could have some reproducable code for it. I feel like the frozen car happens when it exit the bubble, and the control were trying to be hand over from iai (my custom social agent) to either sumo or other provider. And then none of the provider can take it, so the car is teared down. However the car seems to be still existed in ego car's observation space or other social agents' observation space.

Besides this, could you send me some instruction on how to have a custom provider, and only use the custom provider in the scenarios? Right now, I think all of my scenarios define a car flow (which use sumo provider and local traffic provider?) How do I remove sumo/local traffic provider from the scenarios and only use a custom one?

Thanks!

Gamenot commented 1 year ago

...

@Gamenot Let me try to see whether I could have some reproducable code for it. I feel like the frozen car happens when it exit the bubble, and the control were trying to be hand over from iai (my custom social agent) to either sumo or other provider. And then none of the provider can take it, so the car is teared down. However the car seems to be still existed in ego car's observation space or other social agents' observation space.

Sure, I spent several hours testing the bubble transfer and found that the vehicles were properly transferred to the providers. The provider systems are set up such that if the vehicle is not in a provider it no longer exists (and will not show up in observations.)

Besides this, could you send me some instruction on how to have a custom provider, and only use the custom provider in the scenarios? Right now, I think all of my scenarios define a car flow (which use sumo provider and local traffic provider?) How do I remove sumo/local traffic provider from the scenarios and only use a custom one?

Thanks!

I am writing up a guide today.

Gamenot commented 1 year ago

Implement a Traffic Provider

The Provider interface

Providers represent the interface for external simulators to integrate with the SMARTS simulation.

Accessing SMARTS from a SMARTS gym environment

SMARTS environments provide access to a SMARTS instance through the _smarts attribute (or smarts in newer environments)

env = gym.make("hiway-v0") # "hiway-v1", "multi_env-v0", ...
smarts = env._smarts

Adding Providers

smarts.add_provider(CustomTrafficProvider(...))

Removing providers

This will be added with #1917

smarts.remove_provider(SumoTrafficSimulation)
# or
sts = smarts.get_provider_by_type(SumoTrafficSimulation)
smarts.remove_provider(sts)

smarts.core.provider.Provider

The provider interface allows

https://github.com/huawei-noah/SMARTS/blob/15909df059f84f414290450c45f1b58705e04861/smarts/core/provider.py#L120-L122

The following should be implemented:

set_manager # The provider manager reference is passed in here (usually a SMARTS instance)
actions # This should return empty set for general use cases
setup # called when the scenario changes
step # called each simulation step
sync # provides all current actor states to this provider
can_accept_actor # necessary if you want the provider to take over existing vehicles given up by agents
add_actor # same as `can_accept_actor`
reset # reset to base state
teardown # close all resources and connections
connected # the status of connection to a remote (like a server)
source_str # the identity of this provider
actor_ids # the ids of actors that this provider manages
manages_actor # if this provider manages the given actor id (or vehicle id)
stop_managing # a command to give up management of a vehicle

I would suggest looking at the smarts.core.external_provider.ExternalProvider and smarts.core.agents_provider.AgentsProvider for an idea on how implement providers.

smarts.core.external_provider.ExternalProvider

This is a special provider that allows injection of vehicle state into the simulator. This state then takes precedence over internal SMARTS simulation state.

from smarts.core.external_provider import ExternalProvider

external_provider = smarts.get_provider_by_type(ExternalProvider) # or smarts.external_provider
external_provider.state_update(vehicle_states)

The ExternalProvider demonstrates a primitive way to implement a provider: https://github.com/huawei-noah/SMARTS/blob/master/smarts/core/external_provider.py

smarts.core.traffic_provider.TrafficProvider

Providers of this sub-type are specific to generating traffic for the simulation. This was previously required to be added through the SMARTS constructor but now can be added the same way as other providers:

smarts.add_provider(CustomTrafficProvider(...))

It provides a few more interfaces to implement:

https://github.com/huawei-noah/SMARTS/blob/15909df059f84f414290450c45f1b58705e04861/smarts/core/traffic_provider.py#L30-L66

smarts.core.vehicle_state.VehicleState

Vehicle state is what SMARTS will use to generate a vehicle within the simulation. The more complete the state, the more accurate the information provided to the agent.

@dataclass
class VehicleState(ActorState):
    """Vehicle state information."""

    ## actor state inherited fields
    # actor_id: str  # unique identifier within the simulation
    # actor_type: Optional[str] = None # non-exaustive but commonly ["social_vehicle", "social_agent", "agent", "signal"]
    # source: Optional[str] = None  # used to identify the source provider. Should match Provider.source_str
    # role: ActorRole = ActorRole.Unknown # ActorRole.External is privilaged state
    # updated: bool = False # not necessary to change

    # vehicle state specific fields that should be filled out as much as possible
    vehicle_config_type: Optional[str] = None  # key into VEHICLE_CONFIGS ["passenger", "truck", "pedestrian", "bus", "coach", "trailer", "motorcycle"]
    pose: Optional[Pose] = None # required
    dimensions: Optional[Dimensions] = None # will default to vehicle_config if not given
    speed: float = 0.0 # not required but will mess with observations
    steering: Optional[float] = None # not required but will mess with observations
    yaw_rate: Optional[float] = None # not required but will mess with observations
    linear_velocity: Optional[np.ndarray] = None # not required
    angular_velocity: Optional[np.ndarray] = None # not required
    linear_acceleration: Optional[np.ndarray] = None # not required
    angular_acceleration: Optional[np.ndarray] = None # not required
yuant95 commented 1 year ago

@Gamenot Thanks so much! I think I found the cause of the "frozen" car issue. I didn't set keep_alive true for the bubble where I use a boid agent. I think what happens is that when the first actor exit the bubble, it also tear down the boid agent, and then the rest actor controlled by the agent also get teared down in the sumo simulation?(at least in the sumo gui). However, they somehow still exist in all other observation space. This issue seems to be resolved after I set keep_alive true for the bubble.

Thanks for the instruction on how to create a custom provider. I'll give it a try soon.

yuant95 commented 1 year ago

@Gamenot I'm going to try the parallel env and provider interface now I have the training with bubble running. Could you please update me on the access to evaluation sceanrios? Could I have access to the scenarios? Or at least a subset of it? I would like to modify some of it using different social agents to test my resulting policy. Thanks!

smarts-project commented 1 year ago

Hello @yuant95, we will be releasing the scenarios within a week. There is some restrictions due to the Waymo license on how we can release them publicly. I may just provide you access to the private repository tomorrow to help you along.

Gamenot commented 1 year ago

@yuant95 See: https://github.com/smarts-project/driving-smarts-1.eval_scenarios for the evaluation scenarios.

yuant95 commented 1 year ago

Hi @Gamenot, thanks for sharing the evaluation scenarios with me. I have one more question about generating the visualization of the episode. What is the procedure I need to follow to generate gif looks like the ones you post on the website as shown below.

image

Adaickalavan commented 1 year ago

Hi @yuant95,

These gif were captured as follows: 1) Run the Envision server on a terminal.

    $ scl envision start

2) Run the desired scenario in another terminal with headless=False option. 3) Perform a screen capture of http://localhost:8081/ and save it into gif. See ScreenToGif tool.

yuant95 commented 1 year ago

Thanks for your instructions! @Gamenot One more question about the visualization. I followed the step there, but in my envision webpage, the road is not shown. Below is the screen captured. Any settings I need to change? Screenshot from 2023-06-08 14-15-23

Adaickalavan commented 1 year ago

The above (i.e., missing roadmap) generally happens when the Envision server is unable to access or find the underlying map file used by the scenario.

For example, when a scenario is executed inside a docker container while the Envision server is running in the local computer, the Envision server would be unable to access the map file located inside the docker container. This results in a visualisation with missing road map.

Both the Envision server and the scenario (with the map file) should be run in the same disk space so that Envision server could access the map file used.

Consider running the Envision server in the background and executing your script (with head enabled) in the same terminal.

$ scl envision start &
$ python3.8 <script.py>
Gamenot commented 1 year ago

Hello @yuant95, I have a slight update on the previous instruction. The scenario rendering in envision depends somewhat on what version of smarts you use.

Earlier version (<v1.0.2)

If it is an earlier version, you will need to supply the scenarios root directory in the envision arguments:

$ scl envision start -s /path/to/scenarios &
$ python3.8 <script.py>

If scl envision start --help does not show a --scenarios option, it is likely you are working with a later version.

Modern version (>v1.0.2)

If using a newer version of SMARTS and you have your own environment you will need to note the changes in smarts/env/hiway_env.py and supply a preamble to envision which can tell envision where the scenarios are:

https://github.com/huawei-noah/SMARTS/pull/1806/files#diff-45db2043100916303501756505b7b37728f90084b06929c2e01092b92f8ab328R160-R161

Replay

You may also want to save the replay and run it back so that the frame-rate is more even. In that case please look at https://smarts.readthedocs.io/en/latest/sim/visualization.html#data-recording-and-replay. This relates to the arguments for the envision client as noted here:

https://github.com/huawei-noah/SMARTS/blob/3384c72cda74f82d971ea4260b8d50098a0a670e/smarts/env/hiway_env.py#L156-L165