Fatal error when Packaging

[mdblabs]

Hi, I've been using this client on the UE4.27.2 Editor during development and it works just fine. I'm packaging a first version of my solution (a Simulator) and the executable crash after packaging. I've test it removing the plugin from the equation and it worked, so the issue is on the plug-in. Seems like it tries to access some code on a pre-defined path / hardcoded path which is not there. It is not taking care of UE4 environment variables (??).

Here is the error: Fatal error: [File:/Users/build/Build/++UE4/Sync/Engine/Source/Runtime/Core/Private/Mac/MacPlatformMisc.cpp]

My setup: -MacOS running UE4.27.2 -Plugin: last release for this UE version.

Any idea? Thanks!

[mdblabs]

Update: I've tested installing it directly on Unreal Engine Project, inside Plugin folder and using repository code, and buying it from Epic's Bazar and installing it directly on the Engine, with same results. The issue appears when combining it with Cesium Plugin.

[getnamo]

Need more details/log stacks to point at issue. What happens when Cesium plugin is disabled?

[mdblabs]

hi! here is a more-complete trace: LogPlayLevel: [2022-10-13 19:50:59] [connect] WebSocket Connection [::1]:8080 v-2 "WebSocket++/0.8.1" /socket.io/?EIO=4&transport=websocket&t=1665683459 101 LogPlayLevel: [2022-10-13 19:50:59] [connect] WebSocket Connection [::1]:8080 v-2 "WebSocket++/0.8.1" /socket.io/?EIO=4&transport=websocket&t=1665683459 101 LogPlayLevel: Warning: [2022.10.13-17.51.00:089][ 69]LogOutputDevice: Warning: LogPlayLevel: Script Stack (0 frames): LogPlayLevel: Error: [2022.10.13-17.51.00:089][ 69]LogMac: Error: Error reentered: Fatal error: [File:/Users/build/Build/++UE4/Sync/Engine/Source/Runtime/Core/Private/Mac/MacPlatformMisc.cpp] [Line: 2380] LogPlayLevel: FMacPlatformMisc::GenerateCrashInfoAndLaunchReporter: posix_spawn() failed (13, Permission denied) LogPlayLevel: Error: [2022.10.13-17.51.00:089][ 69]LogMac: Error: HandleError re-entered. LogPlayLevel: Error: 2022-10-13 19:51:00.089 VertiportSimulator-Mac-DebugGame[42572:39220722] [UE4] Fatal error: [File:/Users/build/Build/++UE4/Sync/Engine/Source/Runtime/Core/Private/Mac/MacPlatformMisc.cpp] [Line: 2380] LogPlayLevel: FMacPlatformMisc::GenerateCrashInfoAndLaunchReporter: posix_spawn() failed (13, Permission denied) LogPlayLevel: Error: ERROR: Client exited with error code: 3 LogPlayLevel: (see /Users/mariano/Library/Logs/Unreal Engine/LocalBuildLogs/Log.txt for full exception trace) LogPlayLevel: AutomationTool exiting with ExitCode=1 (Error_Unknown) LogPlayLevel: Completed Launch On Stage: Run Task, Time: 20.220606 LogPlayLevel: Error: RunUAT ERROR: AutomationTool was unable to run successfully. PackagingResults: Error: Launch failed! Unknown Error

If Cesium is disabled it works just fine. Same if I disable Socket-IO. Seems like is some kind of incompatibility. I've post same info on Cesium forums.

[getnamo]

Only line mentioned is https://github.com/EpicGames/UnrealEngine/blob/4.27/Engine/Source/Runtime/Core/Private/Mac/MacPlatformMisc.cpp#L2380 which is an error reporter (thread spawn?) so that's not a useful hint. What about the file mentioned at Error: ERROR: Client exited with error code: 3 LogPlayLevel: (see /Users/mariano/Library/Logs/Unreal Engine/LocalBuildLogs/Log.txt for full exception trace ?

[mdblabs]

Here it is (tailed from the error):

ProcessResult.StdOut: [2022-10-13 19:50:59] [connect] WebSocket Connection [::1]:8080 v-2 "WebSocket++/0.8.1" /socket.io/?EIO=4&transport=websocket&t=1665683459 101 ProcessResult.StdOut: [2022-10-13 19:50:59] [connect] WebSocket Connection [::1]:8080 v-2 "WebSocket++/0.8.1" /socket.io/?EIO=4&transport=websocket&t=1665683459 101 ProcessResult.StdOut: [2022-10-13 19:50:59] [connect] WebSocket Connection [::1]:8080 v-2 "WebSocket++/0.8.1" /socket.io/?EIO=4&transport=websocket&t=1665683459 101 ProcessResult.StdOut: [2022-10-13 19:50:59] [connect] WebSocket Connection [::1]:8080 v-2 "WebSocket++/0.8.1" /socket.io/?EIO=4&transport=websocket&t=1665683459 101 ProcessResult.StdOut: [2022-10-13 19:50:59] [connect] WebSocket Connection [::1]:8080 v-2 "WebSocket++/0.8.1" /socket.io/?EIO=4&transport=websocket&t=1665683459 101 ProcessResult.StdOut: [2022-10-13 19:50:59] [connect] WebSocket Connection [::1]:8080 v-2 "WebSocket++/0.8.1" /socket.io/?EIO=4&transport=websocket&t=1665683459 101 ProcessResult.StdOut: [2022.10.13-17.51.00:089][ 69]LogOutputDevice: Warning: ProcessResult.StdOut: ProcessResult.StdOut: Script Stack (0 frames): ProcessResult.StdOut: ProcessResult.StdOut: [2022.10.13-17.51.00:089][ 69]LogMac: Error: Error reentered: Fatal error: [File:/Users/build/Build/++UE4/Sync/Engine/Source/Runtime/Core/Private/Mac/MacPlatformMisc.cpp] [Line: 2380] ProcessResult.StdOut: FMacPlatformMisc::GenerateCrashInfoAndLaunchReporter: posix_spawn() failed (13, Permission denied) ProcessResult.StdOut: ProcessResult.StdOut: ProcessResult.StdOut: ProcessResult.StdOut: [2022.10.13-17.51.00:089][ 69]LogMac: Error: HandleError re-entered. ProcessResult.StdErr: 2022-10-13 19:51:00.089 VertiportSimulator-Mac-DebugGame[42572:39220722] [UE4] Fatal error: [File:/Users/build/Build/++UE4/Sync/Engine/Source/Runtime/Core/Private/Mac/MacPlatformMisc.cpp] [Line: 2380] ProcessResult.StdErr: FMacPlatformMisc::GenerateCrashInfoAndLaunchReporter: posix_spawn() failed (13, Permission denied) Log.WriteException: ============================================================================== Log.WriteException: ERROR: Client exited with error code: 3 Log.WriteException: (see /Users/mariano/Library/Logs/Unreal Engine/LocalBuildLogs/Log.txt for full exception trace) Log.WriteException: Log.WriteException: AutomationException: Client exited with error code: 3 Log.WriteException: at Project.RunStandaloneClient (System.Collections.Generic.List1[T] DeployContextList, System.String ClientLogFile, AutomationTool.CommandUtils+ERunOptions ClientRunFlags, System.String ClientApp, System.String ClientCmdLine, AutomationTool.ProjectParams Params) [0x0057c] in :0 Log.WriteException: at Project.RunInternal (AutomationTool.ProjectParams Params, System.String ServerLogFile, System.String ClientLogFile) [0x00155] in :0 Log.WriteException: at Project.Run (AutomationTool.ProjectParams Params) [0x000a2] in :0 Log.WriteException: at BuildCookRun.DoBuildCookRun (AutomationTool.ProjectParams Params) [0x000b0] in :0 Log.WriteException: at BuildCookRun.ExecuteBuild () [0x00040] in :0 Log.WriteException: at AutomationTool.BuildCommand.Execute () [0x00001] in <538229d5ae744bf38e5a5850079734b9>:0 Log.WriteException: at AutomationTool.Automation.Execute (System.Collections.Generic.List1[T] CommandsToExecute, System.Collections.Generic.Dictionary2[TKey,TValue] Commands) [0x0007c] in <538229d5ae744bf38e5a5850079734b9>:0 Log.WriteException: at AutomationTool.Automation.Process (System.String[] Arguments, UnrealBuildTool.StartupTraceListener StartupListener) [0x00286] in <538229d5ae744bf38e5a5850079734b9>:0 Log.WriteException: at AutomationTool.Program.MainProc (System.String[] Arguments, UnrealBuildTool.StartupTraceListener StartupListener) [0x00001] in <4f556dff359e450ca5015839534cd675>:0 Log.WriteException: at AutomationTool.Program+<>c__DisplayClass1_0.

b__2 () [0x00000] in <4f556dff359e450ca5015839534cd675>:0 Log.WriteException: at AutomationTool.InternalUtils.RunSingleInstance (System.Func1[TResult] Main) [0x000a0] in <538229d5ae744bf38e5a5850079734b9>:0 Log.WriteException: at AutomationTool.Program.Main (System.String[] Arguments) [0x0020a] in <4f556dff359e450ca5015839534cd675>:0 Log.WriteException: ============================================================================== Program.Main: AutomationTool exiting with ExitCode=1 (Error_Unknown)

[getnamo]

Only thing I'm noticing is 4 rapid connects in a row, what happens if you change your connection address/port? (before connecting)

[mdblabs]

Well, I removed all SocketIO connections in my code, but left the plug-in activated, with same result.

LogPlayLevel: [2022.10.14-18.28.25:695][ 0]LogRHI: Display: Encountered a new graphics PSO: 1437120915 LogPlayLevel: [2022.10.14-18.28.25:695][ 0]LogRHI: Display: New Graphics PSO (1437120915) Description: 498FC725F5DF4A3E45E706C4104D6E4D123EB302,D9362DB216436F606DDED024A350193848BCA239,0000000000000000000000000000000000000000,0000000000000000000000000000000000000000,0000000000000000000000000000000000000000,<0 1 0 0 1 0 15 0 1 0 0 1 0 15 0 1 0 0 1 0 15 0 1 0 0 1 0 15 0 1 0 0 1 0 15 0 1 0 0 1 0 15 0 1 0 0 1 0 15 0 1 0 0 1 0 15 1 0>,<0.000000 0.000000 2 1 1 0>,<0 3 1 7 0 0 2 0 7 0 0 0 255 240>,1,11,1049100,1,1,1,1,0,6,10,1114121,0,0,18,9,0,0,2,1048585,0,0,2,25,0,0,2,9,0,0,2,9,0,0,0,0,0,0,0,0,0,0,0,0,10,<0 0 3 0 12 0>,<1 0 20 13 4 1>,<2 0 5 1 8 0>,<2 4 5 2 8 0>,<3 0 8 3 0 0>,<4 0 13 4 8 0>,<4 0 13 5 8 0>,<4 0 13 6 8 0>,<4 0 13 7 8 0>,<4 4 12 15 8 0>,<0 0 0 0 0 0>,<0 0 0 0 0 0>,<0 0 0 0 0 0>,<0 0 0 0 0 0>,<0 0 0 0 0 0>,<0 0 0 0 0 0> LogPlayLevel: [2022.10.14-18.28.25:760][ 2]LogRHI: Display: Encountered a new graphics PSO: 4115306594 LogPlayLevel: [2022.10.14-18.28.25:760][ 2]LogRHI: Display: New Graphics PSO (4115306594) Description: 31277B03C13239DB429683562D5CAB6CA0A637FC,EF5CDE18E0E3E8750870F303880CAAA1524FF398,0000000000000000000000000000000000000000,0000000000000000000000000000000000000000,0000000000000000000000000000000000000000,<0 4 5 0 0 5 15 0 1 0 0 1 0 15 0 1 0 0 1 0 15 0 1 0 0 1 0 15 0 1 0 0 1 0 15 0 1 0 0 1 0 15 0 1 0 0 1 0 15 0 1 0 0 1 0 15 1 0>,<0.000000 0.000000 2 1 1 0>,<0 3 1 7 0 0 2 0 7 0 0 0 255 240>,1,11,1049100,1,1,1,1,0,1,10,9,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,10,<0 0 3 0 12 0>,<1 0 20 13 4 1>,<2 0 5 1 8 0>,<2 4 5 2 8 0>,<3 0 8 3 0 0>,<4 0 13 4 8 0>,<4 0 13 5 8 0>,<4 0 13 6 8 0>,<4 0 13 7 8 0>,<4 4 12 15 8 0>,<0 0 0 0 0 0>,<0 0 0 0 0 0>,<0 0 0 0 0 0>,<0 0 0 0 0 0>,<0 0 0 0 0 0>,<0 0 0 0 0 0> LogPlayLevel: Error: 2022-10-14 20:28:26.641 VertiportSimulator-Mac-DebugGame[79237:39580382] [UE4] Fatal error: [File:/Users/build/Build/++UE4/Sync/Engine/Source/Runtime/Core/Private/Mac/MacPlatformMisc.cpp] [Line: 2380] LogPlayLevel: FMacPlatformMisc::GenerateCrashInfoAndLaunchReporter: posix_spawn() failed (13, Permission denied) LogPlayLevel: Error: [2022.10.14-18.28.26:641][ 94]LogMac: Error: Error reentered: Fatal error: [File:/Users/build/Build/++UE4/Sync/Engine/Source/Runtime/Core/Private/Mac/MacPlatformMisc.cpp] [Line: 2380] LogPlayLevel: FMacPlatformMisc::GenerateCrashInfoAndLaunchReporter: posix_spawn() failed (13, Permission denied) LogPlayLevel: Error: [2022.10.14-18.28.26:641][ 94]LogMac: Error: HandleError re-entered. LogPlayLevel: Error: ERROR: Client exited with error code: 3

Is there any way I can trace where the plug-in is crashing? If I use the repository code, can I add some traces, and it will compiled, right?

[getnamo]

Yeah you can launch it from development build (not editor) after packaged and in debug mode from VS studio, err Xcode I guess, then you'll see a stack trace of the error when it crashes. At least that's how it works on PC.

[slyring]

Any solutions for this yet? Am facing same issue after packaging the game along with the cesium.

UE 5.0.3

Only information I have from dump.

`rapidjson::MemoryPoolAllocator::Malloc(unsigned long long) allocators.h:194 rapidjson::GenericValue<rapidjson::UTF8,rapidjson::MemoryPoolAllocator >::GenericValue<rapidjson::UTF8,rapidjson::MemoryPoolAllocator >(char const ptr64,unsigned int,rapidjson::MemoryPoolAllocator & ptr64) ptr64 0x00007ff78f231812 rapidjson::GenericReader<rapidjson::UTF8,rapidjson::UTF8,rapidjson::CrtAllocator>::ParseString<0,rapidjson::EncodedInputStream<rapidjson::UTF8,rapidjson::MemoryStream>,rapidjson::GenericDocument<rapidjson::UTF8,rapidjson::MemoryPoolAllocator,rapidjson::CrtAllocator> >(rapidjson::EncodedInputStream<rapidjson::UTF8,rapidjson::MemoryStream> & ptr64,rapidjson::GenericDocument<rapidjson::UTF8,rapidjson::MemoryPoolAllocator,rapidjson::CrtAllocator> & ptr64,bool) ptr64 0x00007ff78f2300a0 rapidjson::GenericReader<rapidjson::UTF8,rapidjson::UTF8,rapidjson::CrtAllocator>::ParseObject<0,rapidjson::EncodedInputStream<rapidjson::UTF8,rapidjson::MemoryStream>,rapidjson::GenericDocument<rapidjson::UTF8,rapidjson::MemoryPoolAllocator,rapidjson::CrtAllocator> >(rapidjson::EncodedInputStream<rapidjson::UTF8,rapidjson::MemoryStream> & ptr64,rapidjson::GenericDocument<rapidjson::UTF8,rapidjson::MemoryPoolAllocator,rapidjson::CrtAllocator> & ptr64) ptr64 0x00007ff78f22fa48 rapidjson::GenericDocument<rapidjson::UTF8,rapidjson::MemoryPoolAllocator,rapidjson::CrtAllocator>::ParseStream<0,rapidjson::UTF8,rapidjson::EncodedInputStream<rapidjson::UTF8,rapidjson::MemoryStream> >(rapidjson::EncodedInputStream<rapidjson::UTF8,rapidjson::MemoryStream> & ptr64) __ptr64 0x00007ff78f22fcd6 rapidjson::GenericDocument<rapidjson::UTF8,rapidjson::MemoryPoolAllocator,rapidjson::CrtAllocator>::Parse(char const ptr64,unsigned int64) __ptr64 0x00007ff78f28f029

0x00007ff78f29141c 0x00007ff78f289c7c async::fifo_scheduler::run_all_tasks(void) __ptr64 0x00007ff78f20f1ed CesiumAsync::CesiumImpl::QueuedScheduler::dispatchQueuedContinuations(void) __ptr64 0x00007ff78f2fcffe Cesium3DTilesSelection::Tileset::updateView(std::vector > const & __ptr64,float) __ptr64 0x00007ff78f22e194 ACesium3DTileset::Tick(float) 0x00007ff787cef142 AActor::TickActor(float,ELevelTick,FActorTickFunction & __ptr64) __ptr64 0x00007ff78d331f82 FActorTickFunction::ExecuteTick(float,ELevelTick,ENamedThreads::Type,TRefCountPtr const & __ptr64) __ptr64 0x00007ff78d3100fb FTickFunctionTask::DoTask(ENamedThreads::Type,TRefCountPtr const & __ptr64) __ptr64 0x00007ff78e2c06bb TGraphTask::ExecuteTask(TArray > & __ptr64,ENamedThreads::Type,bool) __ptr64 0x00007ff78e2c56b5 FNamedTaskThread::ProcessTasksNamedThread(int,bool) __ptr64 0x00007ff7888c0564 FNamedTaskThread::ProcessTasksUntilQuit(int) __ptr64 0x00007ff7888c09fe FTaskGraphCompatibilityImplementation::WaitUntilTasksComplete(TArray,TSizedInlineAllocator<4,32,TSizedDefaultAllocator<32> > > const & __ptr64,ENamedThreads::Type) __ptr64 0x00007ff7888c8815 FTickTaskSequencer::ReleaseTickGroup(ETickingGroup,bool) __ptr64 0x00007ff78e2d3588 FTickTaskManager::RunTickGroup(ETickingGroup,bool) __ptr64 0x00007ff78e2d8207 UWorld::RunTickGroup(ETickingGroup,bool) __ptr64 0x00007ff78dafb73f UWorld::Tick(ELevelTick,float) __ptr64 0x00007ff78db044e1 UGameEngine::Tick(float,bool) __ptr64 0x00007ff78d930b5c FEngineLoop::Tick(void) __ptr64 0x00007ff786c0e1a7 GuardedMain(wchar_t const * __ptr64) 0x00007ff786c25afc GuardedMainWrapper(wchar_t const * __ptr64) 0x00007ff786c25bca LaunchWindowsStartup(HINSTANCE__ * __ptr64,HINSTANCE__ * __ptr64,char * __ptr64,int,wchar_t const * __ptr64) 0x00007ff786c28c4c WinMain 0x00007ff786c3a404 [Inlined] invoke_main() 0x00007ff78fc60d8a __scrt_common_main_seh() 0x00007ff78fc60d69 0x00007ffec42b244d 0x00007ffec530dfb8`

[slyring]

@getnamo Do have a look at this thread. Hope it helps in resolving the issue

https://community.cesium.com/t/built-project-crash-at-rapidjson-call-v1-8-1/16468/2

[slyring]

So the main reason for this issue is when multiple plugins are using rapidjson. Here, in this case, am using SocketIOClient and Cesium for Unreal. Removing either of the plugins works else any 1 plugin crashes. That's what I could figure out as of now. I desperately need a fix.

[mdblabs]

Yeah @slyring, it's a real pain. I talked with Cesium developers as well, with same conclusion. It is related to how Unreal manages symbols, and it has no trivial solution. After a month trying it on different platforms with same results, I'm migrating my code to gRPC.

[slyring]

@mdblabs Oh okay. So how is it going with gRPC?

[mdblabs]

gRPC has no symbol collision. I'm using protobuffer for the protocol definition and I was able to build in Desktop mode (not only in Editor) and package the solution in a binary without any issue so far.

[slyring]

Ah, I see nice. My use case is kinda different here. I've created a remote control for my application. It's like UE(remote app) to Socket Server to UE(main application). I did go through a bit about gRPC and I don't think it's gonna solve the problem for me. I Am stuck, unfortunately. :|

[getnamo]

It's unlikely this plugin will pivot away from rapidjson in the near term, so you may need an alternative.

One option might be to use a pure websocket plugin, e.g. https://github.com/dajiangningjing/UEWebsocket

[slyring]

Thanks for the alternative. Gotta try different things now but it will be a huge change for me because of the extensive use of SIOJsonValue. Alternatively am also trying to switch from cesium to ArcGIS as well. In the combination with ArcGIS it works as it doesn't utilize RapidJson.

[getnamo]

There's a thought that came to mind: if they're also using rapidjson and you have their plugin source, you can delete the rapidjson source in socket.io in the plugin and point the header include to the one in cesium. If the API is stable and same, it should work.

[getnamo]

Would probably be enough to change this line in build.cs

https://github.com/getnamo/SocketIOClient-Unreal/blob/fd7cb7aec136bcd6d67690159b816739eb7c3c24/Source/SocketIOLib/SocketIOLib.Build.cs#L35

[slyring]

There's a thought that came to mind: if they're also using rapidjson and you have their plugin source, you can delete the rapidjson source in socket.io in the plugin and point the header include to the one in cesium. If the API is stable and same, it should work.

This they did mention in one of their posts. Since cesium is engine plugin. I have to try how I can use it.

[slyring]

@getnamo @mdblabs Guys! I found the fix and it's just simple.

Bear with me for a while. Let me explain

in rapidjson.h it's already mentioned in the comment regarding symbol clashes.

// RAPIDJSON_NAMESPACE_(BEGIN|END)
/*! \def RAPIDJSON_NAMESPACE
    \ingroup RAPIDJSON_CONFIG
    \brief   provide custom rapidjson namespace

    In order to avoid symbol clashes and/or "One Definition Rule" errors
    between multiple inclusions of (different versions of) RapidJSON in
    a single binary, users can customize the name of the main RapidJSON
    namespace.

    In case of a single nesting level, defining \c RAPIDJSON_NAMESPACE
    to a custom name (e.g. \c MyRapidJSON) is sufficient.  If multiple
    levels are needed, both \ref RAPIDJSON_NAMESPACE_BEGIN and \ref
    RAPIDJSON_NAMESPACE_END need to be defined as well:

So based on this, in rapidjson.h I changed

from

#define RAPIDJSON_NAMESPACE rapidjson
#endif

to this

#define RAPIDJSON_NAMESPACE rapidjsonsockio
#endif

then in sio_packet.cpp

from

using namespace rapidjson;

to

using namespace rapidjsonsockio;

That's it.

@getnamo Do you think it's valid for you to change it and release the plugin with this fix? Am sure it shouldn't affect anything else.

[getnamo]

I'd have to fork the submodule and push the change there then merge it here. Definitely plausible though, just need to find the time to make the change.

[getnamo]

Fix pushed in https://github.com/getnamo/SocketIOClient-Unreal/commit/e238b31fc8a0a20bd9de54177e8cc323d01feacc (2.4.3), pull and confirm its working for you (you will need to pull the rapidjson submodule to get it's update)

[slyring]

@getnamo Tried git clone and download zip, but it's not receiving asio, rapidjson and websocketpp contents for some reason.

[getnamo]

Project uses submodules so you need to clone with --recurse-submodules (as mentioned in https://github.com/getnamo/SocketIOClient-Unreal#via-git-clone)

[slyring]

@getnamo But shouldn't download zip just work as well? Last time that's how I did.

[getnamo]

Only if you get the manually prepared .7z from releases, getting a zipped file from github source does not add submodule source or compiled binaries.

[slyring]

@getnamo Plugin is working now. Tested on packaged build :)

[slyring]

Only if you get the manually prepared .7z from releases, getting a zipped file from github source does not add submodule source or compiled binaries.

Oops yup. My bad

[mdblabs]

Thank you very much @getnamo and @slyring. I can confirm it is also working on Linux. I'm closing the issue. Best