Test & Support ALKS OpenSCENARIO examples

fabianoboril commented 3 years ago

Some additional OSC examples were published here: https://github.com/arauschert/OSC-ALKS-scenarios

Test and find out what we need to support.

[x] OSC allows to have no environment setup in the Init part.
[x] Support ObjectController (with PR #713)
[x] Support ActivateController Action (with PR #713)
[x] Support for multiple stories within one scenario (with PR #739)
[x] Support for LaneOffsetAction (with PR #702)
[ ] Support for FollowTrajectoryAction

Scenarios:

[x] ALKS_Scenario_4.1_1_FreeDriving (if an ALKS controller is implemented)
[ ] ALKS_Scenario_4.1_2_SwervingLeadVehicle
[ ] ALKS_Scenario_4.1_3_SwervingSideVehicle
[x] ALKS_Scenario_4.2_1_FullyBlockingTarget (if an ALKS controller is implemented)
[x] ALKS_Scenario_4.2_2_PartiallyBlockingTarget (if an ALKS controller is implemented)
[ ] ALKS_Scenario_4.2_3_CrossingPedestrian
[ ] ALKS_Scenario_4.2_4_MultipleBlockingTargets
[x] ALKS_Scenario_4.3_1_FollowLeadVehicleComfortable
[ ] ALKS_Scenario_4.3_2_FollowLeadVehicleEmergencyBrake
[ ] ALKS_Scenario_4.4_1_CutInNoCollision
[ ] ALKS_Scenario_4.4_2_CutInUnavoidableCollision
[ ] ALKS_Scenario_4.5_1_CutOutFullyBlocking
[ ] ALKS_Scenario_4.5_2_CutOutMultipleBlockingTargets
[ ] ALKS_Scenario_4.6_1_ForwardDetectionRange
[ ] ALKS_Scenario_4.6_2_LateralDetectionRange

arauschert commented 3 years ago

Hi,

great that you already created an issue for support of the OSC-ALKS-scenarios!

I'm also trying to run them with CARLA now and here are my additional findings:

[x] error on loading OpenDRIVE map. see here
[x] support for pedestrian catalog needed
[ ] 4.2_3: if Orientation is given in LanePosition, then the offset attribute is applied after the rotation and the entity coordinate system is used for the translation instead of the road coordinate system (pedestrian is turned 90° to the left and then moved 5 meters along the entity x coordinate which then equals to the road's s coordinate instead of the road's t coordinate). The offset attribute in the LanePosition is meant as t offset.
[ ] 4.2_3: LanePosition - the offset direction is inverted. the positive t axis of the road coordinate system goes to the left, negative to the right
[ ] 4.4_1: RelativeLanePosition: dLane direction needs to be inverted. positive lane numbering is to the left (offset probably also inverted like in LanePosition)
[ ] 4.1_3: RelativeLanePosition: if ds is <= 0.0, the scenario_runner throws an error. Actually these are valid values. E.g. a vehicle can also be placed behind another vehicle.
[x] 4.3_2: The speed action doesn't do anything. Don't know if it is not triggered (SimulationTimeCondition should work) or one of the attributes cannot be interpreted
[x] 4.3_1: Doesn't start. Error see comment below. Since the scenario is almost the same as 4.3_2, only with multiple consecutive SpeedActions, I assume it's either related to the StartTrigger (StoryboardElementStateCondition) or to the multiple events in one maneuver. The error message looks like the actor isn't set for the second event.
[ ] RelativeTargetSpeed seems not supported (in SpeedAction). When exchanging to AbsoluteTargetSpeed it works, but for the CutIn a relative init speed is needed due to the parameterization of this value.

I will fix an issue with the start position of the ego vehicle. It starts at s=0, so in theory the rear axles are exactly on the road, but it's better if the whole vehicle is on a road on startup. So I will move it forward a few meters and adjust the scenarios.

One further remark: In general more debug output of the scenario runner would be desirable. This would make it easier to pin down the root cause of the issues.

fabianoboril commented 3 years ago

Hi Andreas,

thanks for your input. Let us try to get the things hunted down.

Regarding the debug out: Do you know about the --debug option? This will also provide you the complete behavior tree while the scenario is executing to get an understanding which part is not working as desired.

fabianoboril commented 3 years ago

Hi,

great that you already created an issue for support of the OSC-ALKS-scenarios!

I'm also trying to run them with CARLA now and here are my additional findings:

error on loading OpenDRIVE map. see here

support for pedestrian catalog needed

Could you please elaborate on the pedestrian catalog support. We have this supported and I tested your catalog and it seems to work just fine.

I also just updated the loading of the OpenDRIVE map, to avoid that the map is reloaded on every call (PR #752)

arauschert commented 3 years ago

Could you please elaborate on the pedestrian catalog support. We have this supported and I tested your catalog and it seems to work just fine.

Sure, thanks for investigating. Here's the output I get when running the scenario 4.2_1, where a pedestrian from the catalog is used as blocking target:

q496134@cmucl872167:~/CARLA/scenario_runner-0.9.11$ python scenario_runner.py --openscenario ../../OSC-ALKS-scenarios/Scenarios/ALKS_Scenario_4.2_1_FullyBlockingTarget_TEMPLATE.xosc --debug WARNING:[SR:OpenScenarioConfiguration]: Wrong OpenDRIVE map in use. Forcing reload of CARLA world WARNING: cannot parse georeference: ''. Using default values. WARNING: cannot parse georeference: ''. Using default values. WARNING: cannot parse georeference: ''. Using default values. Preparing scenario: ALKS Scenario 4.2_1 FullyBlockingTarget Template WARNING: Actor model car_ego not available. Using instead vehicle.tesla.model3 WARNING: Actor model Adult not available. Using instead walker.pedestrian.0001 The scenario cannot be loaded Traceback (most recent call last): File "scenario_runner.py", line 374, in _load_and_run_scenario scenario = OpenScenario(world=self.world, File "/home/q496134/CARLA/scenario_runner-0.9.11/srunner/scenarios/open_scenario.py", line 187, in init super(OpenScenario, self).init("OpenScenario", ego_vehicles=ego_vehicles, config=config, File "/home/q496134/CARLA/scenario_runner-0.9.11/srunner/scenarios/basic_scenario.py", line 53, in init self._initialize_actors(config) File "/home/q496134/CARLA/scenario_runner-0.9.11/srunner/scenarios/basic_scenario.py", line 116, in _initialize_actors raise Exception("Error: Unable to add actors") Exception: Error: Unable to add actors Error: Unable to add actors Destroying ego vehicle 87 ERROR: failed to destroy actor 87 : unable to destroy actor: not found No more scenarios .... Exiting

Another observation is that if you run scenario 4.2_4, where also a pedestrian from a catalog is used, it runs. But I didn't see the pedestrian. Here's the relevant debug output:

q496134@cmucl872167:~/CARLA/scenario_runner-0.9.11$ python scenario_runner.py --openscenario ../../OSC-ALKS-scenarios/Scenarios/ALKS_Scenario_4.2_4_MultipleBlockingTargets_TEMPLATE.xosc --debug WARNING: cannot parse georeference: ''. Using default values. WARNING: cannot parse georeference: ''. Using default values. WARNING:[SR:OpenScenarioConfiguration]: Wrong OpenDRIVE map in use. Forcing reload of CARLA world WARNING: cannot parse georeference: ''. Using default values. WARNING: cannot parse georeference: ''. Using default values. WARNING: cannot parse georeference: ''. Using default values. Preparing scenario: ALKS Scenario 4.2_4 MultipleBlockingTargets Template WARNING: Actor model car_ego not available. Using instead vehicle.tesla.model3 WARNING: Actor model Adult not available. Using instead walker.pedestrian.0001 WARNING: Actor model bus not available. Using instead vehicle.volkswagen.t2 WARNING: cannot parse georeference: ''. Using default values. WARNING: cannot parse georeference: ''. Using default values. ScenarioManager: Running scenario OpenScenario

--------- Tick ---------

...

   (-) InitBehaviour [✓]
        --> InitialActorSettings Done? [✕] -- 'InitialActorSettings' comparison failed
        [-] OneShot [✓]
            (o) InitBehaviour [✓]
                [-] InitActorTargetBlocking2 [✓]
                    --> ChangeActorControl [✓]
                [-] InitActorEgo [✓]
                    --> ChangeActorControl [✓]
                    --> ChangeActorTargetSpeed [✓]
            --> Mark Done [✓] -- success

Seems like the pedestrian (TargetBlocking) is parsed from the scenario file but not added to the simulation. Only the exception from above does not occur.

fabianoboril commented 3 years ago

Which commit hash are you testing with?

arauschert commented 3 years ago

I'm using release 0.9.11

fabianoboril commented 3 years ago

Could you try testing #713? It is not yet completed but should resolve some of the issues you might see.

arauschert commented 3 years ago

Could you try testing #713? It is not yet completed but should resolve some of the issues you might see.

Hi, I checked out the branch and now the pedestrian scenarios run. Great! Seems these things are already fixed. Then let's concentrate on the other stuff.

arauschert commented 3 years ago

Here's the output for scenario 4.3_1(tested with branch feat/OscControllerUpdates):

q496134@cmucl872167:~/CARLA/scenario_runner$ ./scenario_runner.py --openscenario ../../OSC-ALKS-scenarios/Scenarios/ALKS_Scenario_4.3_1_FollowLeadVehicleComfortable_TEMPLATE.xosc WARNING: cannot parse georeference: ''. Using default values. WARNING: cannot parse georeference: ''. Using default values. WARNING: cannot parse georeference: ''. Using default values. WARNING: cannot parse georeference: ''. Using default values. Preparing scenario: ALKS Scenario 4.3_1 FollowLeadVehicleComfortable Template WARNING: Actor model car_ego not available. Using instead vehicle.tesla.model3 WARNING: Actor model car not available. Using instead vehicle.tesla.model3 The scenario cannot be loaded Traceback (most recent call last): File "./scenario_runner.py", line 375, in _load_and_run_scenario scenario = OpenScenario(world=self.world, File "/home/q496134/CARLA/scenario_runner/srunner/scenarios/open_scenario.py", line 187, in init super(OpenScenario, self).init(self.config.name, ego_vehicles=ego_vehicles, config=config, File "/home/q496134/CARLA/scenario_runner/srunner/scenarios/basic_scenario.py", line 63, in init behavior = self._create_behavior() File "/home/q496134/CARLA/scenario_runner/srunner/scenarios/open_scenario.py", line 325, in _create_behavior maneuver_behavior = OpenScenarioParser.convert_maneuver_to_atomic( File "/home/q496134/CARLA/scenario_runner/srunner/tools/openscenario_parser.py", line 1039, in convert_maneuver_to_atomic if 'role_name' in traffic_actor.attributes and traffic_actor.attributes['role_name'] == obj: AttributeError: 'NoneType' object has no attribute 'attributes' 'NoneType' object has no attribute 'attributes' Destroying ego vehicle 124 ERROR: failed to destroy actor 124 : unable to destroy actor: not found No more scenarios .... Exiting

fabianoboril commented 3 years ago

A fix is integrated into the branch of #713.

arauschert commented 3 years ago

A fix is integrated into the branch of #713.

Yes, looks good. Now the scenario 4.3_1 starts. I'll mark this issue as done in the comment above.

arauschert commented 3 years ago

The issue with the ego position at the start of the road (s=0) is now fixed in the newest version of the ALKS scenarios. They were also moved to the ASAM organization GitHub account: https://github.com/asam-oss/OSC-ALKS-scenarios

arauschert commented 3 years ago

One of the next topics could be the inversion of the direction of the lateral axis for positions in teleport actions according to the coordinate system definitions of OpenSCENARIO: https://www.asam.net/index.php?eID=dumpFile&t=f&f=4092&token=d3b6a55e911b22179e3c0895fe2caae8f5492467#_coordinate_systems I assume that then also other existing example scenarios in CARLA need to be adjusted?

ralwing commented 2 years ago

I tried to test those scenarios tag/0.3.2 in Carla 0.9.13 + Scenario Runner master with merged #713 without luck:

carla@GiGis:~/scenario_runner$ python3 scenario_runner.py --openscenario /home/carla/OSC-ALKS-scenarios/Scenarios/ALKS_Scenario_4.2_1_FullyBlockingTarget_TEMPLATE.xosc 
WARNING:[SR:OpenScenarioConfiguration]: Wrong OpenDRIVE map in use. Forcing reload of CARLA world
WARNING: cannot parse georeference: ''. Using default values. 
Traceback (most recent call last):
  File "scenario_runner.py", line 607, in main
    result = scenario_runner.run()
  File "scenario_runner.py", line 507, in run
    result = self._run_openscenario()
  File "scenario_runner.py", line 495, in _run_openscenario
    config = OpenScenarioConfiguration(self._args.openscenario, self.client, openscenario_params)
  File "/home/carla/scenario_runner/srunner/scenarioconfigs/openscenario_configuration.py", line 61, in __init__
    self._parse_openscenario_configuration()
  File "/home/carla/scenario_runner/srunner/scenarioconfigs/openscenario_configuration.py", line 93, in _parse_openscenario_configuration
    self._set_actor_information()
  File "/home/carla/scenario_runner/srunner/scenarioconfigs/openscenario_configuration.py", line 251, in _set_actor_information
    entry = OpenScenarioParser.get_catalog_entry(self.catalogs, catalog_reference)
  File "/home/carla/scenario_runner/srunner/tools/openscenario_parser.py", line 365, in get_catalog_entry
    entry = catalogs[catalog_reference.attrib.get("catalogName")][entry_name]
KeyError: '$TargetBlocking_Catalog'
carla@GiGis:~/scenario_runner$

I've also tried the current master, still no luck:

carla@GiGis:~/scenario_runner$ python3 scenario_runner.py --openscenario /home/carla/OSC-ALKS-scenarios/Scenarios/ALKS_Scenario_4.2_1_FullyBlockingTarget_TEMPLATE.xosc 
Traceback (most recent call last):
  File "scenario_runner.py", line 607, in main
    result = scenario_runner.run()
  File "scenario_runner.py", line 507, in run
    result = self._run_openscenario()
  File "scenario_runner.py", line 495, in _run_openscenario
    config = OpenScenarioConfiguration(self._args.openscenario, self.client, openscenario_params)
  File "/home/carla/scenario_runner/srunner/scenarioconfigs/openscenario_configuration.py", line 61, in __init__
    self._parse_openscenario_configuration()
  File "/home/carla/scenario_runner/srunner/scenarioconfigs/openscenario_configuration.py", line 89, in _parse_openscenario_configuration
    self._check_version()
  File "/home/carla/scenario_runner/srunner/scenarioconfigs/openscenario_configuration.py", line 103, in _check_version
    raise AttributeError("Only OpenSCENARIO 1.0 is supported")
AttributeError: Only OpenSCENARIO 1.0 is supported

what's intended...

What changes do i need to make it work?

arauschert commented 2 years ago

Hi! Great that you try to make CARLA compatible to OpenSCENARIO! The upper error looks like the runner couldn't find a catalog with the given name. The parameter is defined in the parameter declaration section with the value "PedestrianCatalog". Was the parameter reference replaced by it's value, so the runner looks for the key "PedestrianCatalog" and not for "$TargetBlocking_Catalog"?

ralwing commented 2 years ago

Thank You @arauschert. A one small change for me, one giant leap for mankind ;-)

christinaSak commented 2 years ago

Hi @arauschert, I have the same issue when I try to run scenario 4.3_2 as you mentioned above: "4.3_2: The speed action doesn't do anything. Don't know if it is not triggered (SimulationTimeCondition should work) or one of the attributes cannot be interpreted". Did you manage to make any progress?

arauschert commented 2 years ago

Hi @arauschert, I have the same issue when I try to run scenario 4.3_2 as you mentioned above: "4.3_2: The speed action doesn't do anything. Don't know if it is not triggered (SimulationTimeCondition should work) or one of the attributes cannot be interpreted". Did you manage to make any progress?

Hi @christinasak ! Probably the acceleration/deceleration feature is contained in the branch of this PR https://github.com/carla-simulator/scenario_runner/pull/713 . Can you please try with this scenario_runner version?

christinaSak commented 2 years ago

Hi @arauschert, I have the same issue when I try to run scenario 4.3_2 as you mentioned above: "4.3_2: The speed action doesn't do anything. Don't know if it is not triggered (SimulationTimeCondition should work) or one of the attributes cannot be interpreted". Did you manage to make any progress?

Hi @christinaSak ! Probably the acceleration/deceleration feature is contained in the branch of this PR #713 . Can you please try with this scenario_runner version?

Thank you now it works fine, probably the AbsoluteTargetSpeed in SpeedActionTarget was not supported in the version I was using.

rupeshreddy97 commented 1 year ago

Hi @arauschert. I have been running the ALKS scenarios with Carla and scenario runner but during the simulation I don't think the ALKS feature is being implemented in the scenario. For example in ALKS_Scenario_4.2_1_FullyBlockingTarget_TEMPLATE, the vehicle doesn't stop for the Pedestrian. I tried in multiple versions of Carla but that didn't help. Is there something I'm missing regarding the Controller part? Please suggest!

arauschert commented 1 year ago

Hi @rupeshreddy97 ! Yes, your observation is correct. The ALKS as system under test is not modelled in the scenario, since it is a behavioural model, which needs to be implemented by a simulation-external controller (e.g. an actual ALKS by an OEM). esmini provides an exemplary simulation-internal ALKS controller, with which you can see, how a vehicle should roughly behave in these scenarios.

rupeshreddy97 commented 1 year ago

Thank you @arauschert for your input! This would really help us moving forward.

fatemehprhm commented 11 months ago

Hi, Can you please add support for SpeedProfileAction? It really makes scenarios more simple to generate.

Kai-tang-Hu commented 9 months ago

Hi! Great that you try to make CARLA compatible to OpenSCENARIO! The upper error looks like the runner couldn't find a catalog with the given name. The parameter is defined in the parameter declaration section with the value "PedestrianCatalog". Was the parameter reference replaced by it's value, so the runner looks for the key "PedestrianCatalog" and not for "$TargetBlocking_Catalog"?

HI,@arauschert. I have the same issue. In the parameter declaration section, there isn't the value "PedestrianCatalog", so I added one.

But It still doesn't work for me.

Kai-tang-Hu commented 9 months ago

Thank You @arauschert. A one small change for me, one giant leap for mankind ;-)

HI,@ralwing . I got the same issue as you. Did you get the code to work? What did you change for this code?

carla-simulator / scenario_runner

Test & Support ALKS OpenSCENARIO examples #655