RocketWerkz / IcarusDedicatedServer

Information for setting up and running a dedicated server in Icarus
36 stars 5 forks source link

Icarus server crashes after a while with out of memory error #6

Open fclante opened 1 year ago

fclante commented 1 year ago

At first I thought it was my rig that was resource depleted, but after upgrading with an additional 16Gb RAM (32Gb DDR4 3200 G.SKILL AEGIS) I still see these errors in the logs with lots of remaining memory available.

LogNet: Warning: UNetDriver::ProcessRemoteFunction: Remote function MULTI_StopMontage called from actor BP_NPC_Chamois_M_Character_C_2147448041 while actor i "This Error is plastered all over the logs for virtually every kind of actor objects."

LogMemory: Warning: Freeing 33554432 bytes from backup pool to handle out of memory. LogMemory: Warning: MemoryStats: AvailablePhysical 24280547328 AvailableVirtual 25303957504 UsedPhysical 8545566720 PeakUsedPhysical 8549249024 UsedVirtual 8504619008 PeakUsedVirtual 8504619008 00e8:err:ntdll:RtlpWaitForCriticalSection section 0000000001162578 "?" wait timed out in thread 00e8, blocked by 0024, retrying (60 sec) LogWindows: Error: === Critical error: === 0024:err:eventlog:ReportEventW L"=== Critical error: ===" 0024:err:eventlog:ReportEventW L"" LogWindows: Error: 0024:err:eventlog:ReportEventW L"Fatal error: [File:Unknown] [Line: 197] " LogWindows: Error: Fatal error: [File:Unknown] [Line: 197] 0024:err:eventlog:ReportEventW L"Ran out of memory allocating 65536 bytes with alignment 0" LogWindows: Error: Ran out of memory allocating 65536 bytes with alignment 0 0024:err:eventlog:ReportEventW L"" LogWindows: Error: 0024:err:eventlog:ReportEventW L"" LogWindows: Error: 0024:err:eventlog:ReportEventW L"" LogWindows: Error: 0024:err:eventlog:ReportEventW L"PHYSX: (D:\Build\++Fortnite\Sync\Engine\Source\ThirdParty\PhysX3\PxShared\src\foundation\src\PsFoundation.cpp 18 0) eINVALID_OPERATION : Foundation destruction failed due to pending module references. Close/release all depending modules first." LogPhysicsCore: Error: PHYSX: (D:\Build++Fortnite\Sync\Engine\Source\ThirdParty\PhysX3\PxShared\src\foundation\src\PsFoundation.cpp 180) eINVALID_OPERATION : Foundation destruction failed due to pending module references. Close/release all depending modules first.

SageDaemon commented 1 year ago

Having the same or at very least a similar issue, this issue however only started occurring recently, log snippet below:

[2023.05.08-16.02.31:370][806]LogIcarusSession: Icarus Update Session - Update Session [2023.05.08-16.02.31:370][806]LogIcarusSession: Update Advanced Session Success - Entered [2023.05.08-16.02.31:370][806]LogIcarusSession: Update Advanced Session Success [2023.05.08-16.02.31:370][806]LogIcarusSession: Icarus Session Node Exit - Success [2023.05.08-16.02.31:370][806]LogIcarusGameModeSurvival: Display: OnUpdateUnrealSessionSuccess [2023.05.08-16.02.31:370][806]LogOnlineProspect: RPC: GetProspect response - Succeeded [2023.05.08-16.02.31:999][825]LogAnimMontage: Warning: JumpToSectionName Action_1 bEndOfSection: 0 failed for Montage 3RD_CHA_1HAxeSwingSlow_Montage 17:02:32 [2023.05.08-16.02.32:833][850]LogAnimMontage: Warning: JumpToSectionName Action_2 bEndOfSection: 0 failed for Montage 3RD_CHA_1HAxeSwingSlow_Montage 17:02:33 [2023.05.08-16.02.33:255][863]LogVoxelResource: Warning: Voxel generation taking a long time - BP_Voxel_GEN_057961_GENERATED_429 - Copper_Normal - 5.0ms [2023.05.08-16.02.33:656][875]LogAnimMontage: Warning: JumpToSectionName Action_1 bEndOfSection: 0 failed for Montage 3RD_CHA_1HAxeSwingSlow_Montage [2023.05.08-16.02.33:812][880]LogIcarusGameStateRecording: Display: RecordToProspectSaveState - NumRecorders: 1670 [2023.05.08-16.02.33:886][880]LogIcarusGameStateRecording: Display: RecordToProspectSaveState complete [2023.05.08-16.02.33:886][880]LogIcarusGameModeSurvival: Verbose: RequestUpdateProspectState [2023.05.08-16.02.33:997][880]LogIcarusGameModeSurvival: OnUpdateProspectSuccess 17:02:34 [2023.05.08-16.02.34:470][895]LogAnimMontage: Warning: JumpToSectionName Action_0 bEndOfSection: 0 failed for Montage 3RD_CHA_1HAxeSwingSlow_Montage 17:02:35 [2023.05.08-16.02.35:303][920]LogAnimMontage: Warning: JumpToSectionName Action_1 bEndOfSection: 0 failed for Montage 3RD_CHA_1HAxeSwingSlow_Montage [2023.05.08-16.02.35:564][928]LogVoxelResource: Warning: Voxel generation taking a long time - BP_Voxel_GEN_057961_GENERATED_429 - Copper_Normal - 9.8ms 17:02:36 [2023.05.08-16.02.36:119][945]LogAnimMontage: Warning: JumpToSectionName Action_2 bEndOfSection: 0 failed for Montage 3RD_CHA_1HAxeSwingSlow_Montage [2023.05.08-16.02.36:451][955]LogVoxelResource: Warning: Voxel generation taking a long time - BP_Voxel_GEN_057961_GENERATED_429 - Copper_Normal - 6.0ms [2023.05.08-16.02.36:948][970]LogAnimMontage: Warning: JumpToSectionName Action_0 bEndOfSection: 0 failed for Montage 3RD_CHA_1HAxeSwingSlow_Montage 17:02:37 [2023.05.08-16.02.37:777][995]LogAnimMontage: Warning: JumpToSectionName Action_0 bEndOfSection: 0 failed for Montage 3RD_CHA_1HAxeSwingSlow_Montage 17:02:38 [2023.05.08-16.02.38:660][ 20]LogUObjectHash: Compacting FUObjectHashTables data took 6.23ms [2023.05.08-16.02.38:664][ 21]LogMemory: Warning: Freeing 33554432 bytes from backup pool to handle out of memory. [2023.05.08-16.02.38:664][ 21]LogMemory: Warning: MemoryStats: AvailablePhysical 40396566528 AvailableVirtual 44691529728 UsedPhysical 7860858880 PeakUsedPhysical 7867686912 UsedVirtual 7827464192 PeakUsedVirtual 7827464192 [2023.05.08-16.02.38:664][ 21]Allocator Stats for binned2 are not in this build set BINNED2_ALLOCATOR_STATS 1 in MallocBinned2.cpp [2023.05.08-16.02.38:664][ 21]LogWindows: Windows GetLastError: Success. (0) 17:03:38 [2023.05.08-16.03.38:666][ 21]LogWindows: Error: === Critical error: === [2023.05.08-16.03.38:666][ 21]LogWindows: Error: [2023.05.08-16.03.38:666][ 21]LogWindows: Error: Fatal error: [File:Unknown] [Line: 197] [2023.05.08-16.03.38:666][ 21]LogWindows: Error: Ran out of memory allocating 65536 bytes with alignment 0 [2023.05.08-16.03.38:666][ 21]LogWindows: Error: [2023.05.08-16.03.38:667][ 21]LogWindows: Error: [2023.05.08-16.03.38:667][ 21]LogWindows: Error: [2023.05.08-16.03.38:681][ 21]LogExit: Executing StaticShutdownAfterError [2023.05.08-16.03.38:783][ 21]LogPhysicsCore: Error: PHYSX: (D:\Build++Fortnite\Sync\Engine\Source\ThirdParty\PhysX3\PxShared\src\foundation\src\PsFoundation.cpp 180) eINVALID_OPERATION : Foundation destruction failed due to pending module references. Close/release all depending modules first. [2023.05.08-16.03.38:783][ 21]LogWindows: FPlatformMisc::RequestExit(1) [2023.05.08-16.03.38:783][ 21]LogWindows: FPlatformMisc::RequestExitWithStatus(1, 3) [2023.05.08-16.03.38:783][ 21]LogCore: Engine exit requested (reason: Win RequestExit) 17:03:42 [2023.05.08-16.03.42:539][ 0]LogInit: High frequency timer resolution =10.000000 MHz [2023.05.08-16.03.42:539][ 0]LogMemory: Memory total: Physical=62.6GB (63GB approx) [2023.05.08-16.03.42:539][ 0]LogMemory: Platform Memory Stats for WindowsServer [2023.05.08-16.03.42:539][ 0]LogMemory: Process Physical Memory: 267.38 MB used, 275.66 MB peak [2023.05.08-16.03.42:539][ 0]LogMemory: Process Virtual Memory: 237.18 MB used, 237.18 MB peak [2023.05.08-16.03.42:539][ 0]LogMemory: Physical Memory: 8715.66 MB used, 55393.50 MB free, 64109.15 MB total [2023.05.08-16.03.42:539][ 0]LogMemory: Virtual Memory: 134158240.00 MB used, 59489.49 MB free, 134217728.00 MB total [2023.05.08-16.03.42:540][ 0]LogWindows: WindowsPlatformFeatures disabled or dedicated server build [2023.05.08-16.03.42:548][ 0]LogInit: Physics initialised using underlying interface: PhysX [2023.05.08-16.03.42:548][ 0]LogInit: Using OS detected language (en-US). [2023.05.08-16.03.42:549][ 0]LogInit: Using OS detected locale (en-US). [2023.05.08-16.03.42:550][ 0]LogTextLocalizationManager: No specific localization for 'en-US' exists, so the 'en' localization will be used. [2023.05.08-16.03.42:562][ 0]LogRendererCore: Ray tracing is disabled. Reason: not supported by current RHI. [2023.05.08-16.03.42:566][ 0]LogInit: Using OS detected language (en-US). [2023.05.08-16.03.42:566][ 0]LogInit: Using OS detected locale (en-US). [2023.05.08-16.03.42:567][ 0]LogTextLocalizationManager: No specific localization for 'en-US' exists, so the 'en' localization will be used. [2023.05.08-16.03.42:572][ 0]LogAssetRegistry: FAssetRegistry took 0.0042 seconds to start up [2023.05.08-16.03.42:632][ 0]LogTextLocalizationManager: Compacting localization data took 1.21ms [2023.05.08-16.03.42:632][ 0]LogStreaming: Display: FlushAsyncLoading: 1 QueuedPackages, 0 AsyncPackages [2023.05.08-16.03.42:632][ 0]LogPackageLocalizationCache: Processed 49 localized package path(s) for 1 prioritized culture(s) in 0.000068 seconds [2023.05.08-16.03.42:633][ 0]LogInit: Selected Device Profile: [WindowsServer] [2023.05.08-16.03.42:633][ 0]LogInit: Active device profile: [000000001D0C6AC0][00000000103EEAB0 49] WindowsServer [2023.05.08-16.03.42:633][ 0]LogInit: Profiles: [000000001D0C6C40][00000000107A0040 49] Windows, [000000001D0C6AC0][00000000103EEAB0 49] WindowsServer, [2023.05.08-16.03.42:640][ 0]LogSerialization: Display: AllowBulkDataInIoStore: 'true' [2023.05.08-16.03.42:646][ 0]LogNetVersion: Icarus 1.0.0, NetCL: 0, EngineNetVer: 17, GameNetVer: 0 (Checksum: 1447163286) [2023.05.08-16.03.42:657][ 0]LogAudioCaptureCore: Display: No Audio Capture implementations found. Audio input will be silent. [2023.05.08-16.03.42:657][ 0]LogAudioCaptureCore: Display: No Audio Capture implementations found. Audio input will be silent. [2023.05.08-16.03.42:671][ 0]LogUObjectArray: 31764 objects as part of root set at end of initial load. [2023.05.08-16.03.42:671][ 0]LogUObjectAllocator: 6732872 out of 0 bytes used by permanent object pool. [2023.05.08-16.03.42:671][ 0]LogUObjectArray: CloseDisregardForGC: 31764/0 objects in disregard for GC pool [2023.05.08-16.03.42:674][ 0]LogGauntlet: Display: Gauntlet Initialized

SageDaemon commented 1 year ago

[2023.05.08-16.35.00:287][668]LogIcarusSession: Icarus Update Session - Update Prospect Info [2023.05.08-16.35.00:287][668]LogLocalSessionSubsystem: Display: UpdateActiveProspectInfo - ProspectID: Daemon Olympus Open World | ProspectDTKey: Outpost006_Olympus [2023.05.08-16.35.00:287][668]LogIcarusSession: Icarus Update Session - No Session Info Changes [2023.05.08-16.35.00:287][668]LogIcarusSession: Icarus Session Node Exit - Success [2023.05.08-16.35.00:287][668]LogIcarusGameModeSurvival: Display: OnUpdateUnrealSessionSuccess [2023.05.08-16.35.00:287][668]LogOnlineProspect: RPC: GetProspect response - Succeeded [2023.05.08-16.35.20:995][297]LogIcarusGameStateRecording: Display: RecordToProspectSaveState - NumRecorders: 1670 [2023.05.08-16.35.21:061][297]LogIcarusGameStateRecording: Display: RecordToProspectSaveState complete [2023.05.08-16.35.21:061][297]LogIcarusGameModeSurvival: Verbose: RequestUpdateProspectState [2023.05.08-16.35.21:165][297]LogIcarusGameModeSurvival: OnUpdateProspectSuccess [2023.05.08-16.35.26:719][465]LogMemory: Warning: Freeing 33554432 bytes from backup pool to handle out of memory. [2023.05.08-16.35.26:719][465]LogMemory: Warning: MemoryStats: AvailablePhysical 50043314176 AvailableVirtual 54338277376 UsedPhysical 8042000384 PeakUsedPhysical 8044122112 UsedVirtual 8007491584 PeakUsedVirtual 8007491584 [2023.05.08-16.35.26:719][465]Allocator Stats for binned2 are not in this build set BINNED2_ALLOCATOR_STATS 1 in MallocBinned2.cpp [2023.05.08-16.35.26:719][465]LogWindows: Windows GetLastError: Success. (0) [2023.05.08-16.36.26:721][465]LogWindows: Error: === Critical error: === [2023.05.08-16.36.26:721][465]LogWindows: Error: [2023.05.08-16.36.26:721][465]LogWindows: Error: Fatal error: [File:Unknown] [Line: 197] [2023.05.08-16.36.26:721][465]LogWindows: Error: Ran out of memory allocating 45056 bytes with alignment 0 [2023.05.08-16.36.26:721][465]LogWindows: Error: [2023.05.08-16.36.26:721][465]LogWindows: Error: [2023.05.08-16.36.26:721][465]LogWindows: Error:

SageDaemon commented 1 year ago

This is consistent each time the server crashes: Allocator Stats for binned2 are not in this build set BINNED2_ALLOCATOR_STATS 1 in MallocBinned2.cpp

Joey-Costa3 commented 1 year ago

On linux virtual machines increase the max_map_count

echo "vm.max_map_count=262144" >> /etc/sysctl.conf

sudo sysctl -p