Image requests cause camera shake

[M-J-Murray]

Hello

I'm trying to simulate a camera on the default drone included with the library. However, I'm coming across a weird issue when I am making image requests to the drone. I have a C++ application which uses the image api to request images as fast as possible. The code is being run on a seperate computer and connecting to the simulation over LAN. What I am seeing is that every time an image request is successfully completed the drone shakes slightly, which over time causes flight disruptions and isn't very realistic. As soon as I turn off the C++ application the shaking stops. Both the simulation and the C++ application are using a forked version of Airsim, that only have a minor change to the range finder sensor configuration (https://github.com/M-J-Murray/AirSim). This is quite a serious bug, as it directly affects your ability to simulate cameras on the drone realistically. Feel free to ask for more details, I hope this can be resolved promptly.

[msb336]

@M-J-Murray What resolution of images are you capturing? Continuous image capture in airsim is very demanding on the gpu, so it's possible that your entire sim is stuttering. Check your framerate by pressing ~ in the game window, and typing the command stat FPS

[M-J-Murray]

As I'm sending the raw data over LAN I had to reduce the resolution to get a higher framerate. With a resolution of 1280x960 I'm getting about 12fps on the C++ application side. I just checked with stat FPS and the simulation runs ay 120fps until I start image requests, then it seems to be dropping down to 60fps and back very rapidly. That does seem to have some relation to what the problem is. Reducing the resolution certainly increases the frame rate on both ends. However, I don't see why this should cause so much stuttering? Is the simulation not constantly calculating the simulated cameras perspective? It doesn't seem that is the issue or otherwise displaying the depth map perspective would have the same impact. It seems more like it is an issue in how the simulated camera frames are being handed over to the image api. I am no expert in the airsim code base, but it seems like this shouldn't affect performance as much as it does.

[msb336]

Rendering in unreal is single threaded, simGetImages() adds a new viewport request to the rendering queue. It is a known bottleneck, and we are looking into ways to get around it currently.

[M-J-Murray]

Ah that makes sense, let me know once you have resolved the issue. This will also affect your swarm functionality, especially if you are attempting to simulate cameras on each drone in the swarm. In the mean time I will just use "ViewMode": "NoDisplay". Is there an eta on when this will be improved?