search

carla-simulator / carla

Open-source simulator for autonomous driving research.
http://carla.org
MIT License
11.19k stars 3.61k forks source link

Fisheye Segmentation looks weird in carla #5678

Closed FrankZTH closed 1 year ago

FrankZTH commented 2 years ago

Hello, everyone, I'm trying to add segmentation to fisheye, but I stuck in this problem for a long time. If someone can solve this, that will be grateful.

CARLA version: 0.9.13 Platform/OS: Linux 18.04 Problem you have experienced: Sematic Segmentation looks weird in the fisheye sensor. image

What you expected to happen: The picture is like a standard Semantic Segmentation in the fisheye. Steps to reproduce:

` void AFisheyeSegmentationSensor::SetUpSceneCaptureComponentCube(USceneCaptureComponentCube &Fisheye) { Super::SetUpSceneCaptureComponentCube(Fisheye);

ApplyViewMode(VMI_Unlit, true, Fisheye.ShowFlags);

Fisheye.ShowFlags.SetNotDrawTaggedComponents(false); // TaggedComponent detects this and sets view relevance for proxy material

Fisheye.ShowFlags.SetAtmosphere(false);

Fisheye.PrimitiveRenderMode = ESceneCapturePrimitiveRenderMode::PRM_UseShowOnlyList;

TArray<UObject *> TaggedComponents; GetObjectsOfClass(UTaggedComponent::StaticClass(), TaggedComponents, false, EObjectFlags::RF_ClassDefaultObject, EInternalObjectFlags::AllFlags);

TArray<UPrimitiveComponent > ShowOnlyComponents; for (UObject Object : TaggedComponents) { UPrimitiveComponent *Component = Cast(Object); Fisheye.ShowOnlyComponents.Emplace(Component); } }

void AFisheyeSegmentationSensor::BeginPlay() { // Called when the game starts or when spawned // Determine the gamma of the player CaptureRenderTarget1-> Init(GetImageWidth(), PF_B8G8R8A8);

Fisheye->Deactivate(); Fisheye->TextureTarget = CaptureRenderTarget1; SetUpSceneCaptureComponentCube(*Fisheye); Fisheye->CaptureScene();

Fisheye->CaptureSource = ESceneCaptureSource::SCS_FinalColorLDR;

Fisheye->UpdateContent(); Fisheye->Activate();

UKismetSystemLibrary::ExecuteConsoleCommand( GetWorld(), FString("g.TimeoutForBlockOnRenderFence 300000"));

GetEpisode().GetWeather()->NotifyWeather();

Super::BeginPlay(); } void AFisheyeSegmentationSensor::PrePhysTick(float DeltaSeconds) { Super::PrePhysTick(DeltaSeconds);

// Add the view information every tick. It's only used for one tick and then // removed by the streamer. IStreamingManager::Get().AddViewInformation( Fisheye->GetComponentLocation(), XSize, YSize); } void AFisheyeSegmentationSensor::PostPhysTick(UWorld *World, ELevelTick TickType, float DeltaSeconds) //Add the view information every tick. It's only used for one tick and then removed by the streamer. { Super::PostPhysTick(World, TickType, DeltaSeconds); // ReadyToCapture = true;

TArray<UObject *> TaggedComponents; GetObjectsOfClass(UTaggedComponent::StaticClass(), TaggedComponents, false, EObjectFlags::RF_ClassDefaultObject, EInternalObjectFlags::AllFlags);

Fisheye->ClearShowOnlyComponents(); for (UObject Object : TaggedComponents) { UPrimitiveComponent Component = Cast(Object); Fisheye->ShowOnlyComponents.Emplace(Component); }

`

My fisheye is referred annaornatskaya's fisheye. https://github.com/carla-simulator/carla/pull/3755/commits/934e1b87a7a004f08acb671cd6903d2d53e15c4a

rawnam commented 2 years ago

@FrankZTH I am trying annaornatskaya's fisheye commits to create a fisheye sensor, but it turns out it uses a modified CubemapHelpers class called CubemapHelpersFisheye from UnrealEngine. Any idea how to resolve these errors?

carla/Unreal/CarlaUE4/Plugins/Carla/Source/Carla/Sensor/FisheyeSensor.cpp:219:3: error: use of undeclared identifier 'CubemapHelpersFisheye'; did you mean 'CubemapHelpers'? carla/Unreal/CarlaUE4/Plugins/Carla/Source/Carla/Sensor/FisheyeSensor.cpp:219:26: error: no type named 'FFisheyeParams' in namespace 'CubemapHelpers' carla/Unreal/CarlaUE4/Plugins/Carla/Source/Carla/Sensor/FisheyeSensor.cpp:229:25: error: use of undeclared identifier 'CubemapHelpersFisheye'; did you mean 'CubemapHelpers'? carla/Unreal/CarlaUE4/Plugins/Carla/Source/Carla/Sensor/FisheyeSensor.cpp:229:48: error: no member named 'GenerateLongLatUnwrapFisheye' in namespace 'CubemapHelpers'; did you mean 'GenerateLongLatUnwrap'?

Mnzs1701 commented 2 years ago

@arukan404 have you applied the patch for Unreal Engine for the fisheye lens as mentioned here, I have been trying to install the same patch but it seems to be compatible with Unreal Engine 4.24 and not Unreal Engine 4.26 @FrankZTH could clarify?

rawnam commented 2 years ago

Hi @Mnzs1701, Thanks for sharing the patch, I guess this is what I have been looking for. Let me try to integrate this and see if it works. Currently, I am doing equirectangular to fisheye projection on the python client side. It works but has a lot of latency... Hopefully, this patch should make it work

FrankZTH commented 2 years ago

@Mnzs1701 You are right, it seems not to be compatible with Unreal Engine 4.26 but I remembered that after del some not support 4.26 codes. It can run with 4.26 too.

rreddy78 commented 1 year ago

We would have to comment out two instances of the member function: virtual bool Serialize(FArchive& Ar) override in the file Engine/Source/Runtime/Engine/Public/CubemapUnwrapUtils.h and it should compile for 4.26.

Though the patch file seems corrupted and I could not apply it automatically.

rreddy78 commented 1 year ago

Hello @FrankZTH :

Wonder if you have got these compilation errors when building for 4.26 ?

/home/sxv1kor/Downloads/GitHub/UnrealEngine_4.26/Engine/Source/Runtime/Engine/Private/CubemapUnwrapUtils.cpp:298:32: error: use of undeclared identifier 'GetVertexShader'; did you mean 'FRHIVertexShader'?
    SetShaderValue(RHICmdList, GetVertexShader(), Transform, TransformValue);
                               ^
/home/sxv1kor/Downloads/GitHub/UnrealEngine_4.26/Engine/Source/Runtime/RHI/Public/RHIResources.h:490:7: note: 'FRHIVertexShader' declared here
class FRHIVertexShader : public FRHIGraphicsShader
      ^
/home/sxv1kor/Downloads/GitHub/UnrealEngine_4.26/Engine/Source/Runtime/Engine/Private/CubemapUnwrapUtils.cpp:313:37: error: use of undeclared identifier 'GetPixelShader'; did you mean 'FRHIPixelShader'?
    SetTextureParameter(RHICmdList, GetPixelShader(),CubeTexture,CubeTextureSampler,Texture);
                                    ^
...
/home/sxv1kor/Downloads/GitHub/UnrealEngine_4.26/Engine/Source/Runtime/Engine/Private/CubemapUnwrapUtils.cpp:349:80: error: indirection requires pointer operand ('TShaderMapRef<FCubemapTexturePropertiesVSFisheye>' invalid)
    GraphicsPSOInit.BoundShaderState.VertexShaderRHI = GETSAFERHISHADER_VERTEX(*VertexShader);
                                                                               ^~~~~~~~~~~~~
/home/sxv1kor/Downloads/GitHub/UnrealEngine_4.26/Engine/Source/Runtime/Engine/Private/CubemapUnwrapUtils.cpp:350:78: error: indirection requires pointer operand ('TShaderMapRef<FCubemapTexturePropertiesPSFisheye<true> >' invalid)
    GraphicsPSOInit.BoundShaderState.PixelShaderRHI = GETSAFERHISHADER_PIXEL(*PixelShader);
                                                                             ^~~~~~~~~~~~
/home/sxv1kor/Downloads/GitHub/UnrealEngine_4.26/Engine/Source/Runtime/Engine/Private/CubemapUnwrapUtils.cpp:332:9: note: in instantiation of function template specialization 'FMipLevelBatchedElementParametersFisheye::BindShaders<FCubemapTexturePropertiesPSFisheye<true> >' requested here
        BindShaders<FCubemapTexturePropertiesPSFisheye<true> >(RHICmdList, GraphicsPSOInit, InFeatureLevel, InTransform, InGamma, ColorWeights, Texture);
        ^
/home/sxv1kor/Downloads/GitHub/UnrealEngine_4.26/Engine/Source/Runtime/Engine/Private/CubemapUnwrapUtils.cpp:350:78: error: indirection requires pointer operand ('TShaderMapRef<FCubemapTexturePropertiesPSFisheye<false> >' invalid)
    GraphicsPSOInit.BoundShaderState.PixelShaderRHI = GETSAFERHISHADER_PIXEL(*PixelShader);
                                                                             ^~~~~~~~~~~~
/home/sxv1kor/Downloads/GitHub/UnrealEngine_4.26/Engine/Source/Runtime/Engine/Private/CubemapUnwrapUtils.cpp:336:9: note: in instantiation of function template specialization 'FMipLevelBatchedElementParametersFisheye::BindShaders<FCubemapTexturePropertiesPSFisheye<false> >' requested here
        BindShaders<FCubemapTexturePropertiesPSFisheye<false> >(RHICmdList, GraphicsPSOInit, InFeatureLevel, InTransform, InGamma, ColorWeights, Texture);
        ^
dongkeyan commented 1 year ago

Hello @FrankZTH :

Wonder if you have got these compilation errors when building for 4.26 ?

/home/sxv1kor/Downloads/GitHub/UnrealEngine_4.26/Engine/Source/Runtime/Engine/Private/CubemapUnwrapUtils.cpp:298:32: error: use of undeclared identifier 'GetVertexShader'; did you mean 'FRHIVertexShader'?
    SetShaderValue(RHICmdList, GetVertexShader(), Transform, TransformValue);
                               ^
/home/sxv1kor/Downloads/GitHub/UnrealEngine_4.26/Engine/Source/Runtime/RHI/Public/RHIResources.h:490:7: note: 'FRHIVertexShader' declared here
class FRHIVertexShader : public FRHIGraphicsShader
      ^
/home/sxv1kor/Downloads/GitHub/UnrealEngine_4.26/Engine/Source/Runtime/Engine/Private/CubemapUnwrapUtils.cpp:313:37: error: use of undeclared identifier 'GetPixelShader'; did you mean 'FRHIPixelShader'?
    SetTextureParameter(RHICmdList, GetPixelShader(),CubeTexture,CubeTextureSampler,Texture);
                                    ^
...
/home/sxv1kor/Downloads/GitHub/UnrealEngine_4.26/Engine/Source/Runtime/Engine/Private/CubemapUnwrapUtils.cpp:349:80: error: indirection requires pointer operand ('TShaderMapRef<FCubemapTexturePropertiesVSFisheye>' invalid)
    GraphicsPSOInit.BoundShaderState.VertexShaderRHI = GETSAFERHISHADER_VERTEX(*VertexShader);
                                                                               ^~~~~~~~~~~~~
/home/sxv1kor/Downloads/GitHub/UnrealEngine_4.26/Engine/Source/Runtime/Engine/Private/CubemapUnwrapUtils.cpp:350:78: error: indirection requires pointer operand ('TShaderMapRef<FCubemapTexturePropertiesPSFisheye<true> >' invalid)
    GraphicsPSOInit.BoundShaderState.PixelShaderRHI = GETSAFERHISHADER_PIXEL(*PixelShader);
                                                                             ^~~~~~~~~~~~
/home/sxv1kor/Downloads/GitHub/UnrealEngine_4.26/Engine/Source/Runtime/Engine/Private/CubemapUnwrapUtils.cpp:332:9: note: in instantiation of function template specialization 'FMipLevelBatchedElementParametersFisheye::BindShaders<FCubemapTexturePropertiesPSFisheye<true> >' requested here
        BindShaders<FCubemapTexturePropertiesPSFisheye<true> >(RHICmdList, GraphicsPSOInit, InFeatureLevel, InTransform, InGamma, ColorWeights, Texture);
        ^
/home/sxv1kor/Downloads/GitHub/UnrealEngine_4.26/Engine/Source/Runtime/Engine/Private/CubemapUnwrapUtils.cpp:350:78: error: indirection requires pointer operand ('TShaderMapRef<FCubemapTexturePropertiesPSFisheye<false> >' invalid)
    GraphicsPSOInit.BoundShaderState.PixelShaderRHI = GETSAFERHISHADER_PIXEL(*PixelShader);
                                                                             ^~~~~~~~~~~~
/home/sxv1kor/Downloads/GitHub/UnrealEngine_4.26/Engine/Source/Runtime/Engine/Private/CubemapUnwrapUtils.cpp:336:9: note: in instantiation of function template specialization 'FMipLevelBatchedElementParametersFisheye::BindShaders<FCubemapTexturePropertiesPSFisheye<false> >' requested here
        BindShaders<FCubemapTexturePropertiesPSFisheye<false> >(RHICmdList, GraphicsPSOInit, InFeatureLevel, InTransform, InGamma, ColorWeights, Texture);
        ^

@rreddy78 Hi，How did you solve the problem？

rreddy78 commented 1 year ago

@rreddy78 Hi，How did you solve the problem？

Use RHICmdList.GetBoundVertexShader() instead of GetVertexShader() RHICmdList.GetBoundPixelShader() instead of GetPixelShader()

Instead of the given macros, use code as below. GraphicsPSOInit.BoundShaderState.VertexShaderRHI = VertexShader.GetVertexShader(); GraphicsPSOInit.BoundShaderState.PixelShaderRHI = PixelShader.GetPixelShader();

JosenJiang commented 1 year ago

@FrankZTH Hello, Have you solved the problem? Would you mind sharing your method？

AHAHA124 commented 1 year ago

@JosenJiang Hello, Have you solved the problem? I tried @rreddy78 method,

Use RHICmdList.GetBoundVertexShader() instead of GetVertexShader() RHICmdList.GetBoundPixelShader() instead of GetPixelShader() Instead of the given macros, use code as below. GraphicsPSOInit.BoundShaderState.VertexShaderRHI = VertexShader.GetVertexShader(); GraphicsPSOInit.BoundShaderState.PixelShaderRHI = PixelShader.GetPixelShader();

But there is still 'error: corrupt patch at line 99'. Would you mind sharing your method？

Hassan-BABAOUSMAIL commented 1 year ago

@JosenJiang Hello, Have you solved the problem? I tried @rreddy78 method,

Use RHICmdList.GetBoundVertexShader() instead of GetVertexShader() RHICmdList.GetBoundPixelShader() instead of GetPixelShader() Instead of the given macros, use code as below. GraphicsPSOInit.BoundShaderState.VertexShaderRHI = VertexShader.GetVertexShader(); GraphicsPSOInit.BoundShaderState.PixelShaderRHI = PixelShader.GetPixelShader();

But there is still 'error: corrupt patch at line 99'. Would you mind sharing your method？

You can apply it manually. You can check this link: https://stackoverflow.com/questions/987372/what-is-the-format-of-a-patch-file
or this one: https://www.oreilly.com/library/view/git-pocket-guide/9781449327507/ch11.html

yunhao666 commented 1 year ago

We would have to comment out two instances of the member function: virtual bool Serialize(FArchive& Ar) override in the file Engine/Source/Runtime/Engine/Public/CubemapUnwrapUtils.h and it should compile for 4.26.

Though the patch file seems corrupted and I could not apply it automatically.

How to solve this error？

LittleBlackCQ commented 1 year ago

Hello, sorry to bother you! I installed UE 4.26 on my Windows 11. Then, I would like to add annaornatskaya's fisheye PR for my research purpose. Therefore, I modified the CubemapUnwrapUtils.cpp and CubemapUnwrapUtils.h exactly as you mentioned, then I rebuilt the UE through UE4.sln. Everything looked fine, but after I clicked the UE4Editor.exe in the folder 'Engine/Binaries/Win64' to launch the engine, it showed 应用程序无法正常启动. In order to know the reason, I deleted the extra code and rebuilt it again, everything worked fine. But I really need the fisheye sensor for my research, how can I solve it? Thank you very much!

EbyGunner commented 7 months ago

/data/UnrealEngine_4.26/Engine/Source/Runtime/Renderer/Private/SceneCaptureRendering.cpp:941:6: error: no matching function for call to 'UpdateSceneCaptureContent_RenderThread' UpdateSceneCaptureContent_RenderThread(RHICmdList, SceneRenderer, TextureRenderTarget, TextureRenderTarget, EventName, FResolveParams(FResolveRect(), TargetFace), false, FGenerateMipsParams()); ^~~~~~~~~~ /data/UnrealEngine_4.26/Engine/Source/Runtime/Renderer/Private/SceneCaptureRendering.cpp:379:13: note: candidate function not viable: requires 10 arguments, but 8 were provided static void UpdateSceneCaptureContent_RenderThread( ^ 1 error generated. make[1]: [Makefile:1080: CarlaUE4Editor] Error 6 make[1]: Leaving directory '/home/sim_user/carla/carla/Unreal/CarlaUE4' make: [Util/BuildTools/Linux.mk:7: launch] Error 2

How to solve this error?