CARLA crashes with some towns and low-quality mode

mctigger commented 2 years ago

https://github.com/carla-simulator/carla/blob/8854804f4d7748e14d937ec763a2912823a7e5f5/PythonAPI/examples/synchronous_mode.py#L135

When this line is changed to load_world("Town05") or any other town except the one already loaded and the simulator is run with ./CarlaUE4.sh -quality-level=Low the simulator will crash. If ran without low-quality mode everything is work fine.

$ ./CarlaUE4.sh -quality-level=Low
4.26.2-0+++UE4+Release-4.26 522 0
Disabling core dumps.
Signal 11 caught.
Malloc Size=65538 LargeMemoryPoolOffset=65554 
CommonUnixCrashHandler: Signal=11
Segmentation fault (core dumped)

Ubuntu 20.04, GTX1080TI. Similar happened on other machines I tried.

JochemTSR commented 2 years ago

Thank you for posting this, my simulator (version 0.9.13) also kept crashing and thought I was going crazy. Not enabling low-quality mode (or not changing the world) also works as a workaround for me. This behavior was not present in version 0.9.11 which I used before.

bernatx commented 2 years ago

Hi, thanks for reporting the problem. We are checking this issue.

svanimisetti commented 2 years ago

@bernatx - I see similar issues with low quality settings when using reload_world() and opened #4966 for the same.

xqrdot commented 2 years ago

Can confirm the same happening with CARLA 0.9.13, Ubuntu 18.04

alexx-km commented 2 years ago

Same happening with CARLA 0.9.13, Ubuntu 20.04

Console Output:

Signal 11 caught.
Malloc Size=65538 LargeMemoryPoolOffset=65554 
CommonUnixCrashHandler: Signal=11
[2022.01.02-14.52.40:536][ 40]LogCore: === Critical error: ===
Unhandled Exception: SIGSEGV: unaligned memory access (SIMD vectors?)

Unreal Engine Crash Reporter:

Caught signal 11 Segmentation fault

libUE4Editor-Engine.so!FDistanceFieldVolumeTexture::IsValidDistanceFieldVolume() const [/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/Engine/Private/DistanceFieldAtlas.cpp:868]
libUE4Editor-Engine.so!FPrimitiveSceneShaderData::FPrimitiveSceneShaderData(FPrimitiveSceneProxy const*) [/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/Engine/Private/PrimitiveUniformShaderParameters.cpp:86]
libUE4Editor-Renderer.so!void UpdateGPUSceneInternal<FRWBufferStructured>(FRHICommandListImmediate&, FScene&) [/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/Renderer/Private/GPUScene.cpp:291]
libUE4Editor-Renderer.so!FDeferredShadingSceneRenderer::Render(FRHICommandListImmediate&) [/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/Renderer/Private/DeferredShadingRenderer.cpp:1459]
libUE4Editor-Renderer.so!RenderViewFamily_RenderThread(FRHICommandListImmediate&, FSceneRenderer*) [/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/Renderer/Private/SceneRendering.cpp:3616]
libUE4Editor-Renderer.so!TEnqueueUniqueRenderCommandType<FRendererModule::BeginRenderingViewFamily(FCanvas*, FSceneViewFamily*)::FDrawSceneCommandName, FRendererModule::BeginRenderingViewFamily(FCanvas*, FSceneViewFamily*)::$_45>::DoTask(ENamedThreads::Type, TRefCountPtr<FGraphEvent> const&) [/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/RenderCore/Public/RenderingThread.h:183]
libUE4Editor-Renderer.so!TGraphTask<TEnqueueUniqueRenderCommandType<FRendererModule::BeginRenderingViewFamily(FCanvas*, FSceneViewFamily*)::FDrawSceneCommandName, FRendererModule::BeginRenderingViewFamily(FCanvas*, FSceneViewFamily*)::$_45> >::ExecuteTask(TArray<FBaseGraphTask*, TSizedDefaultAllocator<32> >&, ENamedThreads::Type) [/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/Core/Public/Async/TaskGraphInterfaces.h:886]
libUE4Editor-Core.so!FNamedTaskThread::ProcessTasksNamedThread(int, bool) [/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/Core/Private/Async/TaskGraph.cpp:709]
libUE4Editor-Core.so!FNamedTaskThread::ProcessTasksUntilQuit(int) [/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/Core/Private/Async/TaskGraph.cpp:600]
libUE4Editor-RenderCore.so!RenderingThreadMain(FEvent*) [/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/RenderCore/Private/RenderingThread.cpp:372]
libUE4Editor-RenderCore.so!FRenderingThread::Run() [/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/RenderCore/Private/RenderingThread.cpp:526]
libUE4Editor-Core.so!FRunnableThreadPThread::Run() [/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/Core/Private/HAL/PThreadRunnableThread.cpp:25]
libUE4Editor-Core.so!FRunnableThreadPThread::_ThreadProc(void*) [/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/Core/Private/HAL/PThreadRunnableThread.h:185]
libpthread.so.0!UnknownFunction(0x9608)
libc.so.6!clone(+0x42)

Additionally, when I reload the map I ~~often~~ always get the error:

[2022.01.02-15.03.12:737][513]LogTemp: Loaded OpenDrive file '/home/ax/Uni/CARLA/carla/Unreal/CarlaUE4/Content/Carla/Maps/OpenDrive/Town10HD_Opt.xodr'
[2022.01.02-15.03.12:775][513]LogCarla: There are 155 SpawnPoints in the map
[2022.01.02-15.03.12:775][513]LogWorld: Bringing up level for play took: 0.160081
[2022.01.02-15.03.12:776][513]LogOutputDevice: Warning: 

Script Stack (0 frames):

Assertion failed: !Primitive->SceneProxy [File:/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/Renderer/Private/RendererScene.cpp] [Line: 1233] 
Primitive has already been added to the scene!
Assertion failed: !Primitive->SceneProxy [File:/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/Renderer/Private/RendererScene.cpp] [Line: 1233] 
Primitive has already been added to the scene!
Assertion failed: !Primitive->SceneProxy [File:/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/Renderer/Private/RendererScene.cpp] [Line: 1233] 
Primitive has already been added to the scene!
Assertion failed: !Primitive->SceneProxy [File:/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/Renderer/Private/RendererScene.cpp] [Line: 1233] 
Primitive has already been added to the scene!
Signal 11 caught.
Malloc Size=65538 LargeMemoryPoolOffset=65554 
Signal 11 caught.
Signal 11 caught.
CommonUnixCrashHandler: Signal=11
[2022.01.02-15.03.12:807][513]LogCore: === Critical error: ===
Unhandled Exception: SIGSEGV: invalid attempt to write memory at address 0x0000000000000003

[2022.01.02-15.03.12:807][513]LogCore: Assertion failed: !Primitive->SceneProxy [File:/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/Renderer/Private/RendererScene.cpp] [Line: 1233] 
Primitive has already been added to the scene!
0x00007fa38a43c302 libUE4Editor-Renderer.so!FScene::AddPrimitive(UPrimitiveComponent*) [/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/Renderer/Private/RendererScene.cpp:1233]
0x00007fa39ba76ddc libUE4Editor-Engine.so!UE4Function_Private::TFunctionRefCaller<FRegisterComponentContext::Process()::$_27, void (int)>::Call(void*, int&) [/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/Core/Public/Templates/Function.h:548]
0x00007fa39ec38149 libUE4Editor-Core.so!ParallelForImpl::TParallelForData<TFunctionRef<void (int)> >::Process(int, TSharedRef<ParallelForImpl::TParallelForData<TFunctionRef<void (int)> >, (ESPMode)1>&, ENamedThreads::Type, bool) [/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/Core/Public/Async/ParallelFor.h:179]
0x00007fa39ec38d7f libUE4Editor-Core.so!ParallelForImpl::TParallelForTask<TFunctionRef<void (int)> >::DoTask(ENamedThreads::Type, TRefCountPtr<FGraphEvent> const&) [/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/Core/Public/Async/ParallelFor.h:134]
0x00007fa39ec386b0 libUE4Editor-Core.so!TGraphTask<ParallelForImpl::TParallelForTask<TFunctionRef<void (int)> > >::ExecuteTask(TArray<FBaseGraphTask*, TSizedDefaultAllocator<32> >&, ENamedThreads::Type) [/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/Core/Public/Async/TaskGraphInterfaces.h:886]
0x00007fa39ebc4a3e libUE4Editor-Core.so!FTaskThreadAnyThread::ProcessTasks() [/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/Core/Private/Async/TaskGraph.cpp:1065]
0x00007fa39ebc3bb8 libUE4Editor-Core.so!FTaskThreadAnyThread::ProcessTasksUntilQuit(int) [/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/Core/Private/Async/TaskGraph.cpp:887]
0x00007fa39ebc39fb libUE4Editor-Core.so!FTaskThreadAnyThread::Run() [/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/Core/Private/Async/TaskGraph.cpp:965]
0x00007fa39ec779b7 libUE4Editor-Core.so!FRunnableThreadPThread::Run() [/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/Core/Private/HAL/PThreadRunnableThread.cpp:25]
0x00007fa39ec3bdc3 libUE4Editor-Core.so!FRunnableThreadPThread::_ThreadProc(void*) [/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/Core/Private/HAL/PThreadRunnableThread.h:185]
0x00007fa39f5c8609 libpthread.so.0!UnknownFunction(0x9608)
0x00007fa395a01293 libc.so.6!clone(+0x42)

rszeto commented 2 years ago

I am seeing the same issue when running CARLA in the Docker image carlasim/carla:0.9.13.

mossr commented 2 years ago

Life savers! Same thing is happening with CARLA 0.9.13 on Windows 10 (removing -quality-level=Low fixes this in the short-term).

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

andrewbest-tri commented 2 years ago

Bump. Has anyone tested recently to see if this is fixed?

dburakovtechsolutions commented 1 year ago

The issue persists in CARLA 9.14 for Linux

Quick reproduction steps:

Get the latest CARLA release and unpack it to some folder. I will refer to it as CARLA_0.9.14 later on
Add a simple empty map I have provided in the attachment to CARLA_0.9.14/CarlaUE4/Content/Carla/Maps/MyMap.uexp and CARLA_0.9.14/CarlaUE4/Content/Carla/Maps/MyMap.umap. The map was created from the Town02 map by removing everything except for the Floor mesh:

Configure CARLA to launch the MyMap on startup by changing the following block in CARLA_0.9.14/CarlaUE4/Config/DefaultEngine.ini:

[/Script/EngineSettings.GameMapsSettings]
EditorStartupMap=/Game/Carla/Maps/MyMap.MyMap
GameDefaultMap=/Game/Carla/Maps/MyMap.MyMap
ServerDefaultMap=/Game/Carla/Maps/MyMap.MyMap
GlobalDefaultGameMode=/Game/Carla/Blueprints/Game/CarlaGameMode.CarlaGameMode_C
GameInstanceClass=/Script/Carla.CarlaGameInstance
TransitionMap=/Game/Carla/Maps/MyMap.MyMap
GlobalDefaultServerGameMode=/Game/Carla/Blueprints/Game/CarlaGameMode.CarlaGameMode_C

Launch CARLA with low quality level parameter:

cd CARLA_0.9.14
./CarlaUE4.sh -prefernvidia -quality-level=Low

Run the bash script from the attachment:

bash test.bash

It will spawn a vehicle and subscribe to it's cameras ten times in a row. Wait for CARLA to crash with:

Pure virtual function called!
Signal 6 caught.
Malloc Size=65538 LargeMemoryPoolOffset=65554 
CommonUnixCrashHandler: Signal=6
Malloc Size=131160 LargeMemoryPoolOffset=196744 
Malloc Size=131160 LargeMemoryPoolOffset=327928 
Engine crash handling finished; re-raising signal 6 for the default handler. Good bye.
Aborted (core dumped)

In addition, GDB shows this:

Pure virtual function called!

Thread 70 "RHIThread" received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffd000c640 (LWP 187807)]
__pthread_kill_implementation (no_tid=0, signo=6, threadid=140736683099712) at ./nptl/pthread_kill.c:44
44  ./nptl/pthread_kill.c: No such file or directory.
(gdb) bt
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=140736683099712) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=140736683099712) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=140736683099712, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007ffff5e42476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007ffff5e287f3 in __GI_abort () at ./stdlib/abo
rt.c:79
#5  0x0000000006a3413b in abort_message ()
#6  0x0000000006a33edf in __cxa_pure_virtual ()
#7  0x000000000455aeda in FVulkanTextureBase::Move (this=0x7fff91dca7e0, Device=..., Context=..., NewAllocation=...) at Runtime/VulkanRHI/Private/VulkanTexture.cpp:2443
#8  0x000000000451cda2 in VulkanRHI::FVulkanSubresourceAllocator::DefragTick (this=<optimized out>, Device=..., Context=..., Heap=0x7fffdb650b00, Count_=<optimized out>) at Runtime/VulkanRHI/Private/VulkanMemory.cpp:4227
#9  0x000000000451cb52 in VulkanRHI::FVulkanResourceHeap::DefragTick (this=0x7fffdb650b00, Device=..., Context=..., Count=3489704176) at Runtime/VulkanRHI/Private/VulkanMemory.cpp:1890
#10 0x000000000451ed57 in VulkanRHI::FMemoryManager::ReleaseFreedPages (this=<optimized out>, Context=...) at Runtime/VulkanRHI/Private/VulkanMemory.cpp:2453
#11 0x000000000454b7c8 in FVulkanCommandListContext::RHIEndFrame (this=0x7ffff3a899a0) at Runtime/VulkanRHI/Private/VulkanRHI.cpp:911
#12 0x0000000004620d39 in FRHICommandListExecutor::ExecuteInner_DoExecute (CmdList=...) at Runtime/RHI/Private/RHICommandList.cpp:373
#13 0x000000000464b4e6 in FExecuteRHIThreadTask::DoTask (this=0x7ffff40c9960, CurrentThread=<optimized out>, MyCompletionGraphEvent=...) at Runtime/RHI/Private/RHICommandList.cpp:428
#14 0x000000000464ae1b in TGraphTask<FExecuteRHIThreadTask>::ExecuteTask (this=0x7ffff40c9950, NewTasks=empty, CurrentThread=ENamedThreads::RHIThread) at Runtime/Core/Public/Async/TaskGraphInterfaces.h:886
#15 0x0000000003087cf2 in FBaseGraphTask::Execute (this=0x7ffff40c9950, NewTasks=empty, CurrentThread=6) at Runtime/Core/Public/Async/TaskGraphInterfaces.h:524
#16 FNamedTaskThread::ProcessTasksNamedThread (this=<optimized out>, QueueIndex=<optimized out>, bAllowStall=true) at Runtime/Core/Private/Async/TaskGraph.cpp:709
#17 0x000000000308693e in FNamedTaskThread::ProcessTasksUntilQuit (this=0x7ffff3a6a4c0, QueueIndex=0) at Runtime/Core/Private/Async/TaskGraph.cpp:600
#18 0x000000000467dc3c in FRHIThread::Run (this=<optimized out>) at Runtime/RenderCore/Private/RenderingThread.cpp:319
#19 0x00000000030e20d3 in FRunnableThreadPThread::Run (this=0x7fffda640000) at Runtime/Core/Private/HAL/PThreadRunnableThread.cpp:25
#20 0x00000000030c4230 in FRunnableThreadPThread::_ThreadProc (pThis=0x7fffda640000) at Runtime/Core/Private/HAL/PThreadRunnableThread.h:185
#21 0x00007ffff5e94b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#22 0x00007ffff5f26a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

This can be reproduced with other maps, but, usually, takes longer time. If -quality-level=Low is omitted, CARLA does not crash

stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.