Huge memory+CPU spike on batch 1 only when saving large single parsers

[fpiesche]

First off: I suspect this overlaps with #563 and #564. I reported this on discord a little while ago but only now got around to writing this up as a GitHub issue.

Platform: Steam Deck SRM version: 2.4.17 AppImage

I've observed that when using "Save to Steam" on single parsers with more than a couple hundred games or so (the point seems to be somewhere around 250-300 games as far as I can tell?), there's an enormous spike in RAM and CPU usage before batch 1 gets saved, regardless of how many batches there actually are.

Running multiple parsers at once totalling up to >500 games with each individual parser at under 200 games does not exhibit this behaviour; it appears to only happen when a single parser exceeds a certain number of games.

The behaviour I've observed, with multiple different parsers, is this:

• "Doing batch 1" is output to the terminal log
• Memory consumption spikes by multiple gigabytes over a few seconds
• Processing stalls for many minutes if not hours as memory consumption very slowly drops
• Once memory usage has returned to normal levels, batch 1 finishes within moments and all remaining batches process similarly quickly.
• CPU usage similarly seems to max out all 8 cores on my Steam Deck while this spike is happening; once RAM consumption has normalised again CPU usage also immediately drops back to idle levels while the batches are actually processed.

Is there anything particular SRM is doing before actually handling batch 1 that it doesn't do with the other batches that could cause this behaviour?

Screenshots of this occurring on my Steam Deck, both test cases with a batch size of 20:

268 games, 58 batches. Memory usage shown spiking from ~3GB to 7.6GB once "Doing batch 1" is output to terminal.

413 games, 103 batches (?!). Once "Doing batch 1" is output to terminal log, memory usage spikes from ~3GB to 12GB (!), and then very slowly decreased by about 1GB/5 minutes with the usage pattern shown here.

In both cases, once memory usage dropped back down to the idle ~3GB, adding games proceeded quickly and finished within a few minutes.

Crucially, if this spike actually exhausts the memory available on the system, it will just stall the machine indefinitely. If I've got Discord and Firefox running in the background (for an idle memory use of about 6GB instead of 3GB), the 413-game parser will max out the system memory and stall the system for 10+ hours while CPU usage is maxed and the fan spins at maximum speed.

I'd be happy to test out a build with more debug output or some such if it'd help pinpoint the problem - hit me up on Discord (ektoutie).

[fpiesche]

Another couple of observations of things I've just noticed running into this again:

• This happens with this parser even in offline mode when there's no images to download
• Increasing the batch size to 500 in online mode makes the spike not completely lock up the system, but has it still taking over 6 hours before ever finishing batch 1
• Recently I've noticed this doesn't always deadlock the system anymore but instead ends with Electron's render process getting killed (presumably by Linux's OOM killer), even when running via the add CLI command without a GUI. I'm not sure if this is due to changes in SteamOS or changes in SRM...

[fpiesche]

Managed to resolve this by wiping my shortcuts.vdf and addeditems.json.

[krazykriskomar]

This is not fixed. happens to me every time.