Enable Lidar to be calculated on each physics substep

[nsubiron]

Investigate if it is possible to update the Lidar measurements on each physics substep.

It seems to be possible to add a delegate that is called on every physics update https://docs.unrealengine.com/latest/INT/API/Runtime/Engine/PhysicsEngine/FBodyInstance/AddCustomPhysics/

See also

• https://docs.unrealengine.com/latest/INT/Engine/Physics/Substepping/
• http://www.aclockworkberry.com/unreal-engine-substepping/
• https://forums.unrealengine.com/unreal-engine/announcements-and-releases/4148-physics-sub-stepping

[simmates]

@nsubiron this is also what I came across. Apparently not every function can be called from within the Custom Physics thread since it is decoupled from the rendering thread. But we could check if raycasting can be done - and it would be necessary for the scene to update in between these substeps.

Motivation: For realistic LiDAR simulation it is crucial to incorporate the rolling shutter effect due to scanning, especially in situations where the car / scene is moving because consecutive LiDAR samples will be "distorted" due to that motion. E.g. straight static objects look slanted when the car is moving. And also pitch yaw and roll rotations when the car is in suspension will incorporate even more effects. That's why some people in the industry claim that simulations of sensors and automatic driving is not possible / won't be beneficial (if these effects cannot be modeled).

[simmates]

In http://www.aclockworkberry.com/unreal-engine-substepping/ it is mentioned that:

Here I’m showing you a little code example of how to setup a substep callback in your project. The first thing to note is that you can access substepping only from an object that can have a rigid body, i.e. any Component extending UPrimitiveComponent. Here I’m subclassing an UStaticMeshComponent.

I think the ALidar class isn't of such catergory, so the question would be if it could be altered to also have a rigid body...

[dirtshell]

I am also really interested in this work. I am currently using AirSim (another OSS UE4 based simulator for CV, but with a focus on drones), but have been looking to implement some kind of LIDAR simulator, so when I saw it was done in CARLA I was really excited. And then saddened when I saw it lacked the rolling shutter effect lol.

I may have some free time to try and hack on this in a month or two. IMO LIDAR sim is pointless if you can't accurately simulate the warping, and I really want an accurate LIDAR sim, so if this isn't taken care of by then I will look in to it.

[nsubiron]

@dirtshell For the record, we do have rolling shutter effect, but it is very slow. You have to run CARLA in sort of slow motion mode to be able to see it. See e.g. in our video https://youtu.be/S_XbKHC_hN0?t=18s, you do so by running CARLA at fixed time-step with a very high FPS

$ ./CarlaUE4.sh -benchmark -fps=200

The higher the frame rate, the slower the simulation will run, and the smaller will be the angle the shutter moves each frame. The problem with the current implementation is that for each Lidar update there is a render update too and that's not very optimal, that's why we want to update the Lidar on each physics sub-steps.

We do plan in the future to add another Lidar implementation as well, using the depth buffer at shader level. This should be faster.

We don't have a plan of action for these two issues yet, if think you can contribute it would be great :)

[dirtshell]

Oh, interesting. I didn't realize the video where the shutter effect was appearing was slowed down FPS wise. I thought the RPM of the scanner was just really slow. I will try this and see how the results look. I am testing SLAM algorithms for high velocity sensors, so the warping is important to test the efficacy of the algorithms.

Just an interesting aside, with the addition of real time ray tracing in UE 4.20, we may be able to generate point clouds even faster. That all depends on the API level we have access to though.

[simmates]

@nsubiron In order for the ray_casting in physics substeps to be useful it is necessary for the scene to get updated between the substeps. I haven't found any conclusive information about if it will get updated though yet...

[kimiya66]

how can I get information of detected vehicles from this lidar