OutOfMemory

[RoosterDragon]

Ticket combining OutOfMemory issues into one.

Other tickets (closed some as they will not contain enough information for us to solve them individually):

5116

8073

8494

9739

11300

11489

12342

12412

I currently suspect the graphics drivers are eating up most of our virtual memory for textures - and the System.OutOfMemoryException indicates our teeny 32-bit address space is being exhausted. This seems to be the case indicated in #12125, but that is for the texture heavy RA2 so we'd need to confirm it for the default mods. If anybody gets this crash again that contains the extra logging added in #12014 - that will help to narrow down whether that is the case or not.

[RoosterDragon]

Things that might help:

• 12125 - Would buy us extra virtual memory on Windows.

• 12131 - Would help minimize the size of the managed heap (if the LOH is indeed significantly fragmented).

[GraionDilach]

I frequently run into this issue with AS on Windows during a map loading, but LARGEADDRESSAWARE seems to be fixing it for now. I agree with the suspicions though since I acknowledge I already grown over RA2 in terms of texture space I think.

Attacque Superior Mod at Version {DEV_VERSION}
on map efca0162327e23b4d6ade493f69b6b91f3dbae91 (Blank Testmap Mk IV by Graion Dilach).
Date: 2016-12-04 23:31:08Z
Operating System: Windows (Microsoft Windows NT 6.2.9200.0)
Runtime Version: .NET CLR 4.0.30319.42000
Exception of type `System.OutOfMemoryException`: OpenGL Error: See graphics.log for details.
GC Memory (post-collect)=283 853 792
GC Memory (pre-collect)=448 836 564
Working Set=1 062 514 688
Private Memory=1 181 339 648
Virtual Memory=1 618 849 792
   at OpenRA.Platforms.Default.OpenGL.CheckGLError() in d:\OpenRA-AS-dev\OpenRA.Platforms.Default\OpenGL.cs:line 498
   at OpenRA.Platforms.Default.Texture.SetData(Byte[] colors, Int32 width, Int32 height) in d:\OpenRA-AS-dev\OpenRA.Platforms.Default\Texture.cs:line 98
   at OpenRA.Graphics.Sheet.GetTexture() in d:\OpenRA-AS-dev\OpenRA.Game\Graphics\Sheet.cs:line 75
   at OpenRA.Graphics.SpriteRenderer.Flush() in d:\OpenRA-AS-dev\OpenRA.Game\Graphics\SpriteRenderer.cs:line 40
   at OpenRA.Renderer.set_CurrentBatchRenderer(IBatchRenderer value) in d:\OpenRA-AS-dev\OpenRA.Game\Renderer.cs:line 195
   at OpenRA.Renderer.Flush() in d:\OpenRA-AS-dev\OpenRA.Game\Renderer.cs:line 176
   at OpenRA.Graphics.WorldRenderer.Draw() in d:\OpenRA-AS-dev\OpenRA.Game\Graphics\WorldRenderer.cs:line 209
   at OpenRA.Game.RenderTick() in d:\OpenRA-AS-dev\OpenRA.Game\Game.cs:line 604
   at OpenRA.Game.Loop() in d:\OpenRA-AS-dev\OpenRA.Game\Game.cs:line 735
   at OpenRA.Game.Run() in d:\OpenRA-AS-dev\OpenRA.Game\Game.cs:line 753
   at OpenRA.Program.Run(String[] args) in d:\OpenRA-AS-dev\OpenRA.Game\Support\Program.cs:line 134
   at OpenRA.Program.Main(String[] args) in d:\OpenRA-AS-dev\OpenRA.Game\Support\Program.cs:line 41

[RoosterDragon]

From #13415

Red Alert Mod at Version release-20170421
on map aae0ea0b52e1b2050f1ebe608c70813f5c608063 (Desert Shellmap by Scott_NZ).
Date: 2017-05-31 06:58:18Z
Operating System: OSX (Unix 16.5.0.0)
Runtime Version: Mono 4.8.0 (mono-4.8.0-branch/9d74414) CLR 4.0.30319.42000
Exception of type `System.OutOfMemoryException`: Out of memory
GC Memory (post-collect)=1,265,223,056
GC Memory (pre-collect)=1,973,597,976
Working Set=2,115,325,952
Private Memory=2,115,330,048
Virtual Memory=3,380,146,176

and

Red Alert Mod at Version release-20170527
on map ab1e2a5cc55feaf6d2734b39d58d4751ca728872 (God's Great Divide Fixed - Non-Faction by Vigor, God, Stex, AMHOLE).
Date: 2017-06-01 06:54:11Z
Operating System: OSX (Unix 16.5.0.0)
Runtime Version: Mono 4.8.0 (mono-4.8.0-branch/9d74414) CLR 4.0.30319.42000
Exception of type `System.OutOfMemoryException`: Out of memory
GC Memory (post-collect)=499,380,584
GC Memory (pre-collect)=962,014,120
Working Set=921,563,136
Private Memory=921,563,136
Virtual Memory=2,156,314,624

From #13435

Red Alert Mod at Version release-20170527
on map 1c6bd120f5b724c5e14103a702caf648e427d33b (Countercross by Comrade Tiki).
Date: 2017-06-03 07:07:25Z
Operating System: OSX (Unix 16.6.0.0)
Runtime Version: Mono 5.0.1.1 (2017-02/5077205) CLR 4.0.30319.42000
Exception of type `System.OutOfMemoryException`: Out of memory
GC Memory (post-collect)=351,365,872
GC Memory (pre-collect)=757,225,520
Working Set=615,952,384
Private Memory=615,956,480
Virtual Memory=1,860,829,184

[RoosterDragon]

The high virtual memory in the latter two reports would seem to suggest texture memory as again the likely cause.

The first report with the 2G in managed memory is weirder - I don't believe the shellmap itself leaks memory - perhaps a specific part of the menu has to be open to allocate/leak that much memory?

Another thing that might help:

• GC.TryStartNoGCRegion - (Available since .NET 4.6) This would help avoid collections occurring during a tick which would help prevent temporary objects being promoted to higher generations. It would also induce the GC to run a full collection when memory is low which might then free enough memory to keep running for a while rather than dying mid-tick somewhere.

[pchote]

The high virtual memory in the latter two reports would seem to suggest texture memory as again the likely cause.

Could it be also be music? We've seen a lot of reports over the last year about music tracks behaving weirdly, and they are typically tens of MB each when uncompressed. If we're leaking sound data in general it could reasonably add up to a few hundred MB over a few games.

It would be really useful if these exception logs could also record the amount of time that OpenRA has been running.

[rob-v]

You're right, music is surely reason for memory leaks, on first look it could be sounds = new Cache<string, ISoundSource>(s => LoadSound(loaders, fileSystem, s)); in Sound. (not time to check it now in more detail)

Start game, in Music menu skip to next track, more memory used. When you skip/load all tracks, lot of memory used. Now restart your match, memory not released. Do it again - load tracks. Soon (after few restarts) it will crash due to Out of memory.

[rob-v]

In addition to solving this memory leak, I propose to exclude music tracks from cache. Memory Commit KB: at beginning of match: 306k after 37 RA tracks (avg 4,5 min): 767k (no gameplay) When tracks played in order, a cached track will played again in 166 minutes while in the meantime it consumes hundreds of MB.

[reaperrr]

Might be worth a try. I assume that most games are short enough that most tracks won't play twice, so the main advantage of caching - avoiding the noticable jank of decoding the file - will only come into effect late in the game, if at all, and we should implement either streaming or decoding on a separate thread at some point in the future to get rid of that jank entirely anyway.

For 3rd-party mods this would be a net-win, because they could now use even 44khz/stereo without worrying about running into OOM due to that.

[GraionDilach]

Currently AS OST is 450 MB so I can very well imagine this being a problem. There's no point in getting 44kHz stereo songs in WAV though atm due to the sheer HDD requirements. I really need to dwelve into Opus inclusion these days, it might be possible that gets decoded faster simply because the lower filesize.

[pchote]

I may be missing something, but it looks like we aren't disposing sequence providers created when maps define their own sequences. This may explain why/if we may be still leaking textures. @RoosterDragon can you confirm?

[pchote]

I solved this for the voxel/model code in #13485 by having World own and dispose the artwork. We could do something similar for sequences.

[RoosterDragon]

You appear to be correct. In fact, I'm having trouble locating where we dispose of any SequenceProviders. Even the DefaultSequences in ModData appear to be untouched? That could imply leaked memory until the finalizer got around to freeing the textures.

[theKalash]

Just got the crash right after the game finished and the Shell Map loaded:

exception-2017-10-15T180746Z.log

[RoosterDragon]

From comment above

OpenRA engine version release-20171014
Red Alert mod version release-20171014
on map e5abf7ceab7a5194072437c8fa617872f07f4474 (Desert Shellmap by Scott_NZ).
Date: 2017-10-15 18:07:46Z
Operating System: OSX (Unix 15.6.0.0)
Runtime Version: Mono 5.2.0.215 (d15-3/da80840) CLR 4.0.30319.42000
Exception of type `System.OutOfMemoryException`: Out of memory
GC Memory (post-collect)=196,782,976
GC Memory (pre-collect)=239,892,864
Working Set=556,773,376
Private Memory=556,773,376
Virtual Memory=1,564,315,648

From #14178

OpenRA engine version release-20171014
Red Alert mod version release-20171014
on map 33aba80bb13138b1f3b26a4ba825daa340e535af (Spinning Earth by American Blunt / Trump).
Date: 2017-10-14 17:34:29Z
Operating System: OSX (Unix 16.7.0.0)
Runtime Version: Mono 5.2.0.224 (d15-3/14f2c81) CLR 4.0.30319.42000
Exception of type `System.OutOfMemoryException`: Out of memory
GC Memory (post-collect)=677,707,984
GC Memory (pre-collect)=948,941,824
Working Set=1,396,244,480
Private Memory=1,396,244,480
Virtual Memory=2,481,725,440

OpenRA engine version release-20171014
Red Alert mod version release-20171014
on map 99ef9de45324e94c9b4cdf7eb237777de728e689 (Advanced Naval Wars II [Rev. 7.0] by Aneyrin).
Date: 2017-10-14 17:02:28Z
Operating System: OSX (Unix 16.7.0.0)
Runtime Version: Mono 5.2.0.224 (d15-3/14f2c81) CLR 4.0.30319.42000
Exception of type `System.OutOfMemoryException`: Out of memory
GC Memory (post-collect)=521,322,000
GC Memory (pre-collect)=824,600,472
Working Set=1,187,274,752
Private Memory=1,187,278,848
Virtual Memory=2,146,385,920

OpenRA engine version release-20171014
Red Alert mod version release-20171014
Date: 2017-10-14 19:13:05Z
Operating System: OSX (Unix 16.7.0.0)
Runtime Version: Mono 5.2.0.224 (d15-3/14f2c81) CLR 4.0.30319.42000
Exception of type `System.OutOfMemoryException`: Out of memory
GC Memory (post-collect)=338,249,128
GC Memory (pre-collect)=432,633,696
Working Set=891,158,528
Private Memory=891,158,528
Virtual Memory=1,850,171,392

[theKalash]

I can reproduce the crash quite reliably. If I play the large Europe map against a couple of modified AI (build MCs and tons of structures) and just defend against the waves while letting the AI build, it crashes about 30-40 minutes into the game.

Doing the same thing on smaller maps allows for games to up to an hour before it crashes.

[RoosterDragon]

For anybody following this issue, just wanted to give you a quick update, the changes in the pull requests you can see above should go a decent way to reducing the amount this issue occurs (and thanks to the OpenRA maintainers for dealing with all of them!)

On a replay involving a very large map, these PRs save at least 400Mb of memory, which should buy quite a bit of breathing room. OSX users should also benefit from #14378 greatly.

If you see crashes in the new version when it arrives I would be super interested in any replays you can share that caused it.

---

Running the game and loading the noted replay in release-20171014. It dies from out of memory loading on my system when it reaches the 1800Mb virtual memory threshold.

Loading the same replay in release-20171014 with the memory fix PRs applied. It actually loads and the replay runs! It peaks just below 1400Mb of virtual memory.

[theKalash]

Since I switched to the 64bit OSX version provided by @pchote I couldn't reproduce the crash at all.

[LipkeGu]

It would be really useful if these exception logs could also record the amount of time that OpenRA has > been running.

i suggested this in: #11191

[abcdefg30]

I haven't seen any oom reports lately, so closing as resolved for now.