Open mctigger opened 2 years ago
Thank you for posting this, my simulator (version 0.9.13) also kept crashing and thought I was going crazy. Not enabling low-quality mode (or not changing the world) also works as a workaround for me. This behavior was not present in version 0.9.11 which I used before.
Hi, thanks for reporting the problem. We are checking this issue.
@bernatx - I see similar issues with low quality settings when using reload_world()
and opened #4966 for the same.
Can confirm the same happening with CARLA 0.9.13, Ubuntu 18.04
Same happening with CARLA 0.9.13, Ubuntu 20.04
Console Output:
Signal 11 caught.
Malloc Size=65538 LargeMemoryPoolOffset=65554
CommonUnixCrashHandler: Signal=11
[2022.01.02-14.52.40:536][ 40]LogCore: === Critical error: ===
Unhandled Exception: SIGSEGV: unaligned memory access (SIMD vectors?)
Unreal Engine Crash Reporter:
Caught signal 11 Segmentation fault
libUE4Editor-Engine.so!FDistanceFieldVolumeTexture::IsValidDistanceFieldVolume() const [/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/Engine/Private/DistanceFieldAtlas.cpp:868]
libUE4Editor-Engine.so!FPrimitiveSceneShaderData::FPrimitiveSceneShaderData(FPrimitiveSceneProxy const*) [/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/Engine/Private/PrimitiveUniformShaderParameters.cpp:86]
libUE4Editor-Renderer.so!void UpdateGPUSceneInternal<FRWBufferStructured>(FRHICommandListImmediate&, FScene&) [/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/Renderer/Private/GPUScene.cpp:291]
libUE4Editor-Renderer.so!FDeferredShadingSceneRenderer::Render(FRHICommandListImmediate&) [/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/Renderer/Private/DeferredShadingRenderer.cpp:1459]
libUE4Editor-Renderer.so!RenderViewFamily_RenderThread(FRHICommandListImmediate&, FSceneRenderer*) [/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/Renderer/Private/SceneRendering.cpp:3616]
libUE4Editor-Renderer.so!TEnqueueUniqueRenderCommandType<FRendererModule::BeginRenderingViewFamily(FCanvas*, FSceneViewFamily*)::FDrawSceneCommandName, FRendererModule::BeginRenderingViewFamily(FCanvas*, FSceneViewFamily*)::$_45>::DoTask(ENamedThreads::Type, TRefCountPtr<FGraphEvent> const&) [/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/RenderCore/Public/RenderingThread.h:183]
libUE4Editor-Renderer.so!TGraphTask<TEnqueueUniqueRenderCommandType<FRendererModule::BeginRenderingViewFamily(FCanvas*, FSceneViewFamily*)::FDrawSceneCommandName, FRendererModule::BeginRenderingViewFamily(FCanvas*, FSceneViewFamily*)::$_45> >::ExecuteTask(TArray<FBaseGraphTask*, TSizedDefaultAllocator<32> >&, ENamedThreads::Type) [/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/Core/Public/Async/TaskGraphInterfaces.h:886]
libUE4Editor-Core.so!FNamedTaskThread::ProcessTasksNamedThread(int, bool) [/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/Core/Private/Async/TaskGraph.cpp:709]
libUE4Editor-Core.so!FNamedTaskThread::ProcessTasksUntilQuit(int) [/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/Core/Private/Async/TaskGraph.cpp:600]
libUE4Editor-RenderCore.so!RenderingThreadMain(FEvent*) [/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/RenderCore/Private/RenderingThread.cpp:372]
libUE4Editor-RenderCore.so!FRenderingThread::Run() [/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/RenderCore/Private/RenderingThread.cpp:526]
libUE4Editor-Core.so!FRunnableThreadPThread::Run() [/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/Core/Private/HAL/PThreadRunnableThread.cpp:25]
libUE4Editor-Core.so!FRunnableThreadPThread::_ThreadProc(void*) [/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/Core/Private/HAL/PThreadRunnableThread.h:185]
libpthread.so.0!UnknownFunction(0x9608)
libc.so.6!clone(+0x42)
Additionally, when I reload the map I often always get the error:
[2022.01.02-15.03.12:737][513]LogTemp: Loaded OpenDrive file '/home/ax/Uni/CARLA/carla/Unreal/CarlaUE4/Content/Carla/Maps/OpenDrive/Town10HD_Opt.xodr'
[2022.01.02-15.03.12:775][513]LogCarla: There are 155 SpawnPoints in the map
[2022.01.02-15.03.12:775][513]LogWorld: Bringing up level for play took: 0.160081
[2022.01.02-15.03.12:776][513]LogOutputDevice: Warning:
Script Stack (0 frames):
Assertion failed: !Primitive->SceneProxy [File:/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/Renderer/Private/RendererScene.cpp] [Line: 1233]
Primitive has already been added to the scene!
Assertion failed: !Primitive->SceneProxy [File:/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/Renderer/Private/RendererScene.cpp] [Line: 1233]
Primitive has already been added to the scene!
Assertion failed: !Primitive->SceneProxy [File:/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/Renderer/Private/RendererScene.cpp] [Line: 1233]
Primitive has already been added to the scene!
Assertion failed: !Primitive->SceneProxy [File:/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/Renderer/Private/RendererScene.cpp] [Line: 1233]
Primitive has already been added to the scene!
Signal 11 caught.
Malloc Size=65538 LargeMemoryPoolOffset=65554
Signal 11 caught.
Signal 11 caught.
CommonUnixCrashHandler: Signal=11
[2022.01.02-15.03.12:807][513]LogCore: === Critical error: ===
Unhandled Exception: SIGSEGV: invalid attempt to write memory at address 0x0000000000000003
[2022.01.02-15.03.12:807][513]LogCore: Assertion failed: !Primitive->SceneProxy [File:/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/Renderer/Private/RendererScene.cpp] [Line: 1233]
Primitive has already been added to the scene!
0x00007fa38a43c302 libUE4Editor-Renderer.so!FScene::AddPrimitive(UPrimitiveComponent*) [/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/Renderer/Private/RendererScene.cpp:1233]
0x00007fa39ba76ddc libUE4Editor-Engine.so!UE4Function_Private::TFunctionRefCaller<FRegisterComponentContext::Process()::$_27, void (int)>::Call(void*, int&) [/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/Core/Public/Templates/Function.h:548]
0x00007fa39ec38149 libUE4Editor-Core.so!ParallelForImpl::TParallelForData<TFunctionRef<void (int)> >::Process(int, TSharedRef<ParallelForImpl::TParallelForData<TFunctionRef<void (int)> >, (ESPMode)1>&, ENamedThreads::Type, bool) [/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/Core/Public/Async/ParallelFor.h:179]
0x00007fa39ec38d7f libUE4Editor-Core.so!ParallelForImpl::TParallelForTask<TFunctionRef<void (int)> >::DoTask(ENamedThreads::Type, TRefCountPtr<FGraphEvent> const&) [/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/Core/Public/Async/ParallelFor.h:134]
0x00007fa39ec386b0 libUE4Editor-Core.so!TGraphTask<ParallelForImpl::TParallelForTask<TFunctionRef<void (int)> > >::ExecuteTask(TArray<FBaseGraphTask*, TSizedDefaultAllocator<32> >&, ENamedThreads::Type) [/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/Core/Public/Async/TaskGraphInterfaces.h:886]
0x00007fa39ebc4a3e libUE4Editor-Core.so!FTaskThreadAnyThread::ProcessTasks() [/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/Core/Private/Async/TaskGraph.cpp:1065]
0x00007fa39ebc3bb8 libUE4Editor-Core.so!FTaskThreadAnyThread::ProcessTasksUntilQuit(int) [/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/Core/Private/Async/TaskGraph.cpp:887]
0x00007fa39ebc39fb libUE4Editor-Core.so!FTaskThreadAnyThread::Run() [/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/Core/Private/Async/TaskGraph.cpp:965]
0x00007fa39ec779b7 libUE4Editor-Core.so!FRunnableThreadPThread::Run() [/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/Core/Private/HAL/PThreadRunnableThread.cpp:25]
0x00007fa39ec3bdc3 libUE4Editor-Core.so!FRunnableThreadPThread::_ThreadProc(void*) [/home/ax/UnrealEngine_4.26/Engine/Source/Runtime/Core/Private/HAL/PThreadRunnableThread.h:185]
0x00007fa39f5c8609 libpthread.so.0!UnknownFunction(0x9608)
0x00007fa395a01293 libc.so.6!clone(+0x42)
I am seeing the same issue when running CARLA in the Docker image carlasim/carla:0.9.13
.
Life savers! Same thing is happening with CARLA 0.9.13 on Windows 10 (removing -quality-level=Low
fixes this in the short-term).
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Bump. Has anyone tested recently to see if this is fixed?
The issue persists in CARLA 9.14 for Linux
Quick reproduction steps:
Get the latest CARLA release and unpack it to some folder. I will refer to it as CARLA_0.9.14
later on
Add a simple empty map I have provided in the attachment to CARLA_0.9.14/CarlaUE4/Content/Carla/Maps/MyMap.uexp
and CARLA_0.9.14/CarlaUE4/Content/Carla/Maps/MyMap.umap
. The map was created from the Town02 map by removing everything except for the Floor mesh:
Configure CARLA to launch the MyMap
on startup by changing the following block in CARLA_0.9.14/CarlaUE4/Config/DefaultEngine.ini
:
[/Script/EngineSettings.GameMapsSettings]
EditorStartupMap=/Game/Carla/Maps/MyMap.MyMap
GameDefaultMap=/Game/Carla/Maps/MyMap.MyMap
ServerDefaultMap=/Game/Carla/Maps/MyMap.MyMap
GlobalDefaultGameMode=/Game/Carla/Blueprints/Game/CarlaGameMode.CarlaGameMode_C
GameInstanceClass=/Script/Carla.CarlaGameInstance
TransitionMap=/Game/Carla/Maps/MyMap.MyMap
GlobalDefaultServerGameMode=/Game/Carla/Blueprints/Game/CarlaGameMode.CarlaGameMode_C
Launch CARLA with low quality level parameter:
cd CARLA_0.9.14
./CarlaUE4.sh -prefernvidia -quality-level=Low
bash test.bash
It will spawn a vehicle and subscribe to it's cameras ten times in a row. Wait for CARLA to crash with:
Pure virtual function called!
Signal 6 caught.
Malloc Size=65538 LargeMemoryPoolOffset=65554
CommonUnixCrashHandler: Signal=6
Malloc Size=131160 LargeMemoryPoolOffset=196744
Malloc Size=131160 LargeMemoryPoolOffset=327928
Engine crash handling finished; re-raising signal 6 for the default handler. Good bye.
Aborted (core dumped)
In addition, GDB shows this:
Pure virtual function called!
Thread 70 "RHIThread" received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffd000c640 (LWP 187807)]
__pthread_kill_implementation (no_tid=0, signo=6, threadid=140736683099712) at ./nptl/pthread_kill.c:44
44 ./nptl/pthread_kill.c: No such file or directory.
(gdb) bt
#0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=140736683099712) at ./nptl/pthread_kill.c:44
#1 __pthread_kill_internal (signo=6, threadid=140736683099712) at ./nptl/pthread_kill.c:78
#2 __GI___pthread_kill (threadid=140736683099712, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3 0x00007ffff5e42476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4 0x00007ffff5e287f3 in __GI_abort () at ./stdlib/abo
rt.c:79
#5 0x0000000006a3413b in abort_message ()
#6 0x0000000006a33edf in __cxa_pure_virtual ()
#7 0x000000000455aeda in FVulkanTextureBase::Move (this=0x7fff91dca7e0, Device=..., Context=..., NewAllocation=...) at Runtime/VulkanRHI/Private/VulkanTexture.cpp:2443
#8 0x000000000451cda2 in VulkanRHI::FVulkanSubresourceAllocator::DefragTick (this=<optimized out>, Device=..., Context=..., Heap=0x7fffdb650b00, Count_=<optimized out>) at Runtime/VulkanRHI/Private/VulkanMemory.cpp:4227
#9 0x000000000451cb52 in VulkanRHI::FVulkanResourceHeap::DefragTick (this=0x7fffdb650b00, Device=..., Context=..., Count=3489704176) at Runtime/VulkanRHI/Private/VulkanMemory.cpp:1890
#10 0x000000000451ed57 in VulkanRHI::FMemoryManager::ReleaseFreedPages (this=<optimized out>, Context=...) at Runtime/VulkanRHI/Private/VulkanMemory.cpp:2453
#11 0x000000000454b7c8 in FVulkanCommandListContext::RHIEndFrame (this=0x7ffff3a899a0) at Runtime/VulkanRHI/Private/VulkanRHI.cpp:911
#12 0x0000000004620d39 in FRHICommandListExecutor::ExecuteInner_DoExecute (CmdList=...) at Runtime/RHI/Private/RHICommandList.cpp:373
#13 0x000000000464b4e6 in FExecuteRHIThreadTask::DoTask (this=0x7ffff40c9960, CurrentThread=<optimized out>, MyCompletionGraphEvent=...) at Runtime/RHI/Private/RHICommandList.cpp:428
#14 0x000000000464ae1b in TGraphTask<FExecuteRHIThreadTask>::ExecuteTask (this=0x7ffff40c9950, NewTasks=empty, CurrentThread=ENamedThreads::RHIThread) at Runtime/Core/Public/Async/TaskGraphInterfaces.h:886
#15 0x0000000003087cf2 in FBaseGraphTask::Execute (this=0x7ffff40c9950, NewTasks=empty, CurrentThread=6) at Runtime/Core/Public/Async/TaskGraphInterfaces.h:524
#16 FNamedTaskThread::ProcessTasksNamedThread (this=<optimized out>, QueueIndex=<optimized out>, bAllowStall=true) at Runtime/Core/Private/Async/TaskGraph.cpp:709
#17 0x000000000308693e in FNamedTaskThread::ProcessTasksUntilQuit (this=0x7ffff3a6a4c0, QueueIndex=0) at Runtime/Core/Private/Async/TaskGraph.cpp:600
#18 0x000000000467dc3c in FRHIThread::Run (this=<optimized out>) at Runtime/RenderCore/Private/RenderingThread.cpp:319
#19 0x00000000030e20d3 in FRunnableThreadPThread::Run (this=0x7fffda640000) at Runtime/Core/Private/HAL/PThreadRunnableThread.cpp:25
#20 0x00000000030c4230 in FRunnableThreadPThread::_ThreadProc (pThis=0x7fffda640000) at Runtime/Core/Private/HAL/PThreadRunnableThread.h:185
#21 0x00007ffff5e94b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#22 0x00007ffff5f26a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
This can be reproduced with other maps, but, usually, takes longer time. If -quality-level=Low
is omitted, CARLA does not crash
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
It is still an issue for CARLA_0.9.14 at UIbuntu 20.04
Issue persists, tested it with 0.9.13 and 0.9.14 on Ubuntu 20.04.
same issue Carla 0.9.14 on Windows
same issue with Carla 0.9.15 on Ubuntu 22.04, RTX 4090
same issue with Carla 0.9.15 on Windows 10, RTX 3080
https://github.com/carla-simulator/carla/blob/8854804f4d7748e14d937ec763a2912823a7e5f5/PythonAPI/examples/synchronous_mode.py#L135
When this line is changed to
load_world("Town05")
or any other town except the one already loaded and the simulator is run with./CarlaUE4.sh -quality-level=Low
the simulator will crash. If ran without low-quality mode everything is work fine.Ubuntu 20.04, GTX1080TI. Similar happened on other machines I tried.