search

HARPLab / DReyeVR

VR driving 🚙 + eye tracking 👀 simulator based on CARLA for driving interaction research
https://arxiv.org/abs/2201.01931
MIT License
139 stars 37 forks source link

Autopilot vehicles crashing into ego vehicle #104

Closed RStorm98 closed 1 year ago

RStorm98 commented 1 year ago

We're having issues with autopilot vehicles colliding with the ego vehicle, and we're unsure of the cause.

https://github.com/HARPLab/DReyeVR/blob/51ea84a65df001801f582051dc3899d996d3f7b9/DReyeVR/EgoVehicle.cpp#L622-L643 This was mentioned in issue #53, but we couldn't locate the equivalent in the version we're using, DREYEVR 0.1.2, ScenarioRunner 0.9.13.

We've been testing --sync, but this drastically speeds up the game, the ego vehicle drives incredibly fast with sync enabled.

We did attempt to paste the contents from the above link into EgoVehicle.cpp, but with sync enabled all traffic stood still and without sync it had no effect.

GustavoSilvera commented 1 year ago

When you are not running with --sync there is an asynchronous timestep communication between the PythonAPI (client) and UE4 (server) which causes weird behavior like what you're seeing; unless your client and server were running at perfectly the exact same framerate (highly unlikely).

If you are running with --sync then the frametimes are forced to be synchronized between the PythonAPI and UE4 server. If this speeds up your game that means your UE4 server is ticking at a much faster rate than the PythonAPI, which needs a while to "catch up". So if you force synchronization then you enable the server to skip some frame updates (giving a sense of "speeding up" in order to match the framerate of the PythonAPI).

Have you tried using --sync alongside fixing the server framerate using a fixed time-step such as this. For real-ish-time performance we like >30FPS so typically would fix the fixed_delta_seconds to 0.025 so each frame takes 1/40th of a second. This works well as long as your simulator server (UE4) can keep up with a consistent 40FPS, otherwise weird stuff will happen like what you're seeing.

In summary:

settings = world.get_settings()
settings.fixed_delta_seconds = 1.0 / desired_fps # limits server framerate
settings.synchronous_mode = True # Enables synchronous mode
world.apply_settings(settings)

See more about how Carla's timestep logic works here

Hope this helps!

GustavoSilvera commented 1 year ago

Also, if you are using DReyeVR 0.1.2 (the latest as of writing) you should no longer need that AEgoVehicle::Register snippet since this has been redone as part of the spawning mechanism.

RStorm98 commented 1 year ago

Thank you for the reply!

We're experimenting with framerate and --sync and the simulation is noticeably better. Framerate is currently set to 35, though there are still occassional speed-ups that happen when we make a right/left turn, drive through a tunnel or just simply when looking down and not seeing the surroundings. Could this have something to do with the server not being able to keep up?

Collisions with autopilot vehicles still happens in sync mode even after we have corrected the framerate, as if they do not react at all to our existence. Could that be caused by the PythonAPI and UE4 server still not being in sync?

RStorm98 commented 1 year ago

Pardon the closing and opening, miss click

GustavoSilvera commented 1 year ago

I'll look into the frame rate issue and post when I find something.

When you say collisions with other vehicles do you mean they collide with the DReyeVR Ego vehicle or with each other and the surrounding environment?

RStorm98 commented 1 year ago

Sorry, I should have elaborated more. I was referring to other vehicles colliding with the DreyeVR Ego vehicle, other than that their behavior is normal.

GustavoSilvera commented 1 year ago

Ah I see, okay then this is not the issue I thought it was. I'll look into it today and try to reproduce & address the issue.

GustavoSilvera commented 1 year ago

hey @RStorm98 after some debugging today I found a workaround that is still hacky so I don't want to push it to main yet.

This seems to be happening because the other vehicles can't "see" the DReyeVR EgoVehicle, although it is being registered correctly, the other vehicles only see other actors currently with the "vehicle.*" tag, which is not the case for the DReyeVR EgoVehicle as of v0.1.2 (since we use "harplab.dreyevr_vehicle.*" to avoid users accidentally spawning additional "vehicle.dreyevr_vehicle.*" ego-vehicles if they use a vanilla Carla script that calls world.get_actors().filter("vehicle.*") as many of them do.

However, if we rename ourselves to be under "vehicle.*" such as "vehicle.dreyevr_vehicle.*" then the other vehicles no longer ignore the EgoVehicle (as it should be).

So, long story short. You'll need to do 2 (but really 3) things:

  1. Go to DReyeVRFactory.cpp and change the CATEGORY to just vehicle: as follows #define CATEGORY TEXT("vehicle")
  2. Go to DReyeVR_utils.py and change the filter commands from "harplab.dreyevr_..." to "vehicle.dreyevr_vehicle.*" and "vehicle.dreyevr_sensor.*" for the find_ego_vehicle and find_ego_sensor functions respectively.
  3. (technically optional, but now you need to be extra careful) In order to no longer spawn additional EgoVehicles in the world (since this is undefined behavior for us) you'll need to verify that all the blueprints you use do NOT contain the DReyeVR blueprints. See the bottom of this issue for how that was resolved previously. You'll basically want to guarantee this by adding blueprints = [bp for bp in blueprints if "dreyevr" not in bp.id] every time you have a blueprints list that you sample from.

While this isn't a great fix, I wanted to let you know the workaround ASAP so you could continue with your work. We'll work on fixing the issue properly soon.

RStorm98 commented 1 year ago

Thank you for the fast replies and the workaround, we'll test it out soon and report back. Much appreciated.

GustavoSilvera commented 1 year ago

Hi again @RStorm98 I seem to have found the better solution as described in this pull request. If you haven't already, try to revert the changes I mentioned above (about renaming the CATEGORY back to "vehicle") and instead just update the ALSM.cpp file as shown in this commit. Then you should just rebuild the PythonAPI and reinstall the .whl file using pip install --no-deps --force-reinstall PythonAPI/carla/dist/YOUR_FILE.whl to make sure the changes take effect.

Let's put this bug behind us :)

RStorm98 commented 1 year ago

Hello again

We didn't try the workaround you mentioned yet, but updating ALSM.cpp and following your instructions has solved the issue.

Thank you for helping us out!