DReyeVR Util Sensor does not work with Additional Maps

[laurayuzheng]

Hi, I'm trying to collect sensor data from additional maps such as Town 6. I imported the tarball here: https://carla-releases.s3.eu-west-3.amazonaws.com/Linux/AdditionalMaps_0.9.13.tar.gz

into the ./Import folder of the built-from-source CARLA folder, then ran ImportAssets.sh to install them.

It seems that CARLA itself loads the world fine (and it is possible to drive in VR to explore these towns), but the find_ego_sensor() from DReyeVR_utils.py is unable to find the DReyeVR sensor in Town 6. Here is the full output in my implementation:

initialized DReyeVRSensor PythonAPI client
Starting scenario runner
Recording on file: D:\VRDRiving\data\test_09_30_2022_15_41_13\replay.rec
Saving sensor data to: D:\VRDRiving\data\test_09_30_2022_15_41_13\sensors.csv
Unable to find DReyeVR ego vehicle in world!
initialized DReyeVRSensor PythonAPI client
INFO:  Found the required file in cache!  Carla/Maps/TM/Town06.bin 
Traceback (most recent call last):
  File "D:\carla\PythonAPI\examples\VRDriving_study.py", line 216, in <module>
    main()
  File "D:\carla\PythonAPI\examples\VRDriving_study.py", line 208, in main
    start_scenario_runner(scenario_runner_instance)
  File "D:\carla\PythonAPI\examples\VRDriving_study.py", line 41, in start_scenario_runner
    result = scenario_runner_instance.run()
  File "D:\VRDriving\scenario_runner\scenario_runner.py", line 529, in run
    result = self._run_route()
  File "D:\VRDriving\scenario_runner\scenario_runner.py", line 494, in _run_route
    result = self._load_and_run_scenario(config)
  File "D:\VRDriving\scenario_runner\scenario_runner.py", line 370, in _load_and_run_scenario
    if not self._load_and_wait_for_world(config.town, config.ego_vehicles):
  File "D:\VRDriving\scenario_runner\scenario_runner.py", line 346, in _load_and_wait_for_world
    self.setup_sensor()
  File "D:\VRDriving\scenario_runner\scenario_runner.py", line 123, in setup_sensor
    self.sensor.ego_sensor.listen(self.callback)
AttributeError: 'NoneType' object has no attribute 'listen'

self.sensor is created via:

def setup_sensor(self):
        from DReyeVR_utils import DReyeVRSensor
        self.sensor = DReyeVRSensor(self.world)
        self.sensor.ego_sensor.listen(self.callback)

Is this expected (I saw there were some quirks mentioned with Additional Maps not being included)? Does this mean that DReyeVR cannot work with Towns 6, 7, and 11?

[GustavoSilvera]

Hi, this is a problem that the sensor.dreyevr.dreyevrsensor is not being found in your self.world client.

May I ask why you are importing the Carla maps from the AdditionalMaps link? We already provide all the maps here. This is important because the maps include the EgoVehicle (and a custom DReyeVRLevel script) embedded in the blueprints, so you can't use the vanilla Carla maps.

This is probably why there is no EgoVehicle in your new maps, so the EgoSensor cannot be found.

You should be able to use our maps in both editor and package mode (though you might need to copy them over manually since the additional maps aren't included in Carla's default build pipeline).

[laurayuzheng]

Thanks! I'm trying to use it with the package mode. I saw an error that Town 6 was not found, and I had used CARLA before for a separate project and assumed I could go ahead and import the usual way. Maps in DReyeVR being different from vanilla maps makes a lot of sense to me now. Where would I copy them over manually?

Would it be in WindowsNoEditor\CarlaUE4\Content\Carla\Maps ?

[GustavoSilvera]

Yes! You should be able to manually copy the Maps we provide to that folder in the Build directory

[laurayuzheng]

I just tried to copy/paste it into the location of the packaged build which I mentioned above, and I got an error here:

Experiment ID:  test
initialized DReyeVRSensor PythonAPI client
Starting scenario runner
Recording on file: D:\VRDRiving\data\test_09_30_2022_16_28_28\replay.rec
Saving sensor data to: D:\VRDRiving\data\test_09_30_2022_16_28_28\sensors.csv
initialized DReyeVRSensor PythonAPI client
The CARLA server uses the wrong map: Town03
This scenario requires to use map: Town06

And it looks for a .bin file with the currently loaded town afterwards:

INFO:  Found the required file in cache!  Carla/Maps/TM/Town03.bin

I am actually not familiar with these maps in CARLA; are the .umap files all I need to load the towns?

[GustavoSilvera]

How are you running these experiments? I think you need to copy both the .umap and the .uexp files.

[laurayuzheng]

Where would** the .uexp files be? Also the experiments are just run using ScenarioRunner. I modified ScenarioRunner so that a DReyeVR sensor is being spawned after the world is restarted. The town is being read from the ScenarioRunner routes xml file.

[GustavoSilvera]

You should also look into our ScenarioRunner changes (specifically run_experiment.py) which we use to run custom routes with ScenarioRunner.

My bad, I thought the .uexp files were in the editor mode, they are temporary files that are build when you package the game. You should copy all the maps into Unreal/CarlaUE4/Content/Carla/Maps and then rebuild make package from there to see the updated maps in your WindowsNoEditor\CarlaUE4\Content\Carla\Maps dir

[laurayuzheng]

I will try this and get back to you! Thanks for quick and helpful responses :) I think your version of run_experiment.py is the one I am using, thankfully. What I have done is, I made some minor changes in ScenarioRunner class itself to "find" the sensor again once the scenario is loaded. Adding the sensor in run_experiment results in the sensor reference being lost due again when the world is reloaded by SRunner.

[laurayuzheng]

Hi Gustavo, I tried to paste those maps from DReyeVR Maps folder to the Unreal/CarlaUE4/Content/Carla/Maps location and rebuild, unfortunately it did not work. Then, I did a fresh install of DReyeVR to try again (in case I messed anything from previous attempts), and noticed that the DReyeVR make install also does that on its own. I copied one more time to be sure and ran make package. However, I still get the same error with map not found.

Would it be easier to use the additional maps in editor mode?

[GustavoSilvera]

Hm, yes the DReyeVR make install command does copy over all the maps from this repo over to Carla, that should work.

In the editor mode can you open these additional maps and use them (play-in-editor)?

What does your packaged build directory look like? Do you have the .uexp files in there too or only the .umap for Town01-4?

[laurayuzheng]

make launch works with the additional maps! Packaged version still does not. We can use editor version for now.

Here is what the packaged build directory looks like at carla\Build\UE4Carla\0.9.13-dirty\WindowsNoEditor\CarlaUE4\Content\Carla\Maps

[GustavoSilvera]

Gotcha, okay well hopefully the editor mode is good enough for now (for those maps).

This is unfortunately a known Carla issue that I thought we addressed but looks like we didn't. We'll look into it soon.

[laurayuzheng]

Will close this issue now, thanks for the help Gustavo!