Low FPS capturing while not using resources

[reza-shahriari]

• AirSim Version/#commit: Colosseum
• UE/Unity version: UE 5.27
• autopilot version: -
• OS Version: windows 10

Settings

setting Gist

How can the issue be reproduced?

Hi i am using Airsim to simulate drones(actually tow drones in an env) everything works well but the problem is when i capture images from the camera. i have a simple environment and a strong computer but arisim capturing about 5 images per second while i need at least 30! not many resources are in use while running the project and capturing data:

i dont know how i can improve the performance of the airsim! i used NoDisplay, changed SubWindows, and turned them off I also tested Python API but all are slow. i know i can add some sleep to my code then i can get the fps i wanted but for capturing a 10min video i should spend 60min then!

i have tested some other way to capture images from unreal. i mean just screenshotting or using a custom camera and rendering its view. both worked real time for me! and both were using more resources. i was trying to change this code to a faster way or at least make it just screenshotting to see if it works but did not work at all:(

do you have this problem too? Has anyone solved it? thank,

Include context on what you are trying to achieve

Context details

Include details of what you already did to find answers

[UnrealOrReal-Jack]

What a nice work you are doing. i have the same problem here! would you try this and tell me if it works? it seems you can use opencv to render images and its working realtime in the video. i am not experimented enough to do it but it would be nice if you do and tell me if its working :) hope that helps -Jack

[xxEoD2242]

I would recommend reducing your image size. 1080p is a lot for that system to render and what your seeing is the total lag of render + passing that through a network and reading it. This is a consequence of how AirSim (which is still the same code, haven't tried a new method yet) does image capturing through the view port. I would also recommend only doing 1 image stream at a time and repeating the process.

Another option is to build in a shared memory interface to do the image transfer, but that would be an enormous amount of work.

[reza-shahriari]

would you try this and tell me if it works?

I will try it and tell you ASAP

@xxEoD2242 Thanks for your fast answer. Yeah having 1 viewport and NoDisplay may help but unfortunately, I need even more FPS. As I have to do this project and I really need Big images (even bigger than 1080) would you tell me a little about the shared memory implementation? or if you have any other suggestions. do you know any other simulator that works with Unreal or Unity or any other engine that we can get realistic images and connect them to Arducopter? thanks again

[reza-shahriari]

I'm not sure if it's a fair comparison or not, but I've used the code below to capture images from a custom camera and save them in real time. The problem was that the lighting in the saved images wasn't as it should be.

void AMyActor::Tick(float DeltaTime)
{
    Super::Tick(DeltaTime);
    UGameViewportClient* GameViewport = GEngine->GameViewport;
    if (GameViewport)
    {
        FViewport* Viewport = GameViewport->Viewport;
        TArray<FColor> Bitmap;
        if (Viewport->ReadPixels(Bitmap))
        {
            int32 Width = Viewport->GetSizeXY().X;
            int32 Height = Viewport->GetSizeXY().Y;
            // Convert to OpenCV format
            cv::Mat Image(Height, Width, CV_8UC4, Bitmap.GetData());
            cv::Mat ImageBGR;
            cv::cvtColor(Image, ImageBGR, cv::COLOR_RGBA2BGR);
            // Save the image
            i++;
            std::string filePath = "C:/Users/AI-Control Lab/Pictures/Saved Pictures/OpenCVTestImage" + std::to_string(i) + ".png";
            cv::imwrite(filePath, ImageBGR);
            UE_LOG(LogTemp, Warning, TEXT("Screenshot saved as screenshot.png"));
        }
        else {
            UE_LOG(LogTemp, Warning, TEXT("NO screenSHOT at all :(("));
        }
    }
}

in the actor.h :

private:
    UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "Capture", meta = (AllowPrivateAccess = "true"))
    UCameraComponent* CameraComponent;

    UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "Capture", meta = (AllowPrivateAccess = "true"))
    USceneCaptureComponent2D* CaptureComponent;

    UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "Capture", meta = (AllowPrivateAccess = "true"))
    UTextureRenderTarget2D* RenderTarget;

captured res:

even setting CaptureComponent->PostProcessSettings.AutoExposureBias = 20000.0f; did not helped

[xxEoD2242]

Are you using Lumen for lighting?

[xxEoD2242]

would you try this and tell me if it works?

I will try it and tell you ASAP

@xxEoD2242 Thanks for your fast answer. Yeah having 1 viewport and NoDisplay may help but unfortunately, I need even more FPS. As I have to do this project and I really need Big images (even bigger than 1080) would you tell me a little about the shared memory implementation? or if you have any other suggestions. do you know any other simulator that works with Unreal or Unity or any other engine that we can get realistic images and connect them to Arducopter? thanks again

You'd need to build a shared memory interface that you could copy data from another process. It's very non-trivial but i'm sure ChatGPT could provide you with an example of how to do it. That's the fastest way to move images. In terms of rate though, you'd probably see something on the order of 20 - 30 FPS just thinking through how this is done with the ViewPort RHIBuffer unlock.

I'd be interested to see a profile of how long it takes to get those images from the ViewPort. Also, that ViewPort is just what the character sees and isn't from a specific camera, right?

[xxEoD2242]

In terms of other simulators, I don't know of any that support ArduCopter and are Unity/UE. Try our Discord to see if anyone else has found anything

[reza-shahriari]

Are you using Lumen for lighting?

Yes, the lighting method was Lumen but I have tested all combinations of lighting/reflection methods and none of them made sense

You'd need to build a shared memory interface that you could copy data from another process.

Ok, ill check it.

Also, that ViewPort is just what the character sees and isn't from a specific camera, right?

yeah you are right. i thought its about cameras

In terms of other simulators, I don't know of any that support ArduCopter and are Unity/UE. Try our Discord to see if anyone else has found anything

Thanks, ill ask it there.

[reza-shahriari]

ROS works nicely! I can capture at about 30 fps while having 25 drones on the screen. Thank you for your time and kindness :) for everyone having the same problem checking this link could be helpful. remember that Lidar may slow down the rendering process