RGB Camera on an ego vehiche in scenario_runner

[Giovygio97]

Hello everybody, I want to ask what is the best way to instantiate and use an RGB camera on an ego vehicle on Scenario_Runner, and use it, with an ML model, to recognize traffic signs and change ego vehicle behaviour based on these signs. I think it's more convenient to initialize the actor and, then, load the behaviors in parallel, with the camera and his ML recognition in the criteria. Is it possible?

• OS: Ubuntu 20.04.2 LTS
• CARLA Version: 0.9.11
• Python version: 3.7
• Version: 0.9.11

[glopezdiest]

Hey @Giovygio97. Personally, I'd recommend directly using an agent to control the ego, instead of using the atomic behaviors to do so. You can create your own class and then using a similar loop to this one, you can synchronize your agent with the ego vehicle. From there, just create your own RGB camera on top of the vehicle. Then, just initialize the agent along with the scenario and add the agent run step to the main loop, here.

I think it's more convenient to initialize the actor and, then, load the behaviors in parallel, with the camera and his ML recognition in the criteria.

I'd recommend against this. Criteria don't affect the behavior of the actor, they are used to either stop the simulation or have some feedback about the simulation.

[fabianoboril]

The easiest way is to use OpenSCENARIO with the simple_vehicle_control Controller. It has a flag to attach a camera.

                    <PrivateAction>
                        <ControllerAction>
                            <AssignControllerAction>
                                <Controller name="HeroAgent_1">
                                    <Properties>
                                        <Property name="module" value="simple_vehicle_control"/>
                                        <Property name="attach_camera" value="True"/>
                                    </Properties>
                                </Controller>
                            </AssignControllerAction>
                            <OverrideControllerValueAction>
                                <Throttle value="0" active="false"/>
                                <Brake value="0" active="false"/>
                                <Clutch value="0" active="false"/>
                                <ParkingBrake value="0" active="false"/>
                                <SteeringWheel value="0" active="false"/>
                                <Gear number="0" active="false"/>
                            </OverrideControllerValueAction>
                        </ControllerAction>
                    </PrivateAction>