Out of memory crash while checking for updates. (variable alloc amount)

FagSoft commented 2 years ago

Your system information

Steam client version (build number or date):
Distribution (e.g. Ubuntu): $ lsb_release -a LSB Version: core-11.1.0ubuntu2-noarch:printing-11.1.0ubuntu2-noarch:security-11.1.0ubuntu2-noarch Distributor ID: Linuxmint Description: Linux Mint 20.1 Release: 20.1 Codename: ulyssa
Opted into Steam client beta?: [Yes/No] No, but trying it now.
Have you checked for system updates?: [Yes/No] Yes, system is current.

Please describe your issue in as much detail as possible:

Steam crashes every day with the following error message: * OUT OF MEMORY! attempted allocation size: ####

The amount is always changing, but mostly around 4000-6000. This has been occationally happening for years (4+) now, but in the last year or so this has become a daily occurance. Often multiple times a day, and I finally got tired of my games crashing.

This crash usually happens after 5+ hours, but some times it happens only minutes after starting Steam. I haven't been able to, as of yet, to find any user-initiated actions that can explain those outliers. Checking for game updates are limited to between 24:00 and 07:00. Unsure if this has any affect on the issue, but after I changed it Steam seems to be crashing less during the day. Though, it might also be because I've started Steam later in the day after I made that change.

I've done a bit of debugging on this one, to try to pin down exactly what's happening. Including making a script that monitors the memory usage. There is no dump file created for these crashes, but I've attached the gdb stacktrace as well as the logs from my memory-script.

See this gist for the logs: https://gist.github.com/FagSoft/db3d4c83574cc5073100ffd89fa801db

Note that the script checks every 30 seconds, and only prints out a message if the difference of memory usage is more than ±5%. Which is why it failed to capture the memory usage of Steam right before it crashed on all but the last run (when gdb kept steam alive).

Steps for reproducing this issue:

Start Steam
Let it sit in the background.
Wait for it to crash.

FagSoft commented 2 years ago

I did check out issue #7630. While it is quite similar I never get the same memory amount, which leads me to think these are two different issues.

FagSoft commented 2 years ago

Just happened again, with the latest Steam beta. For this one I started Steam (which opens the library), went to the Store page, then back to the library and started to download OneShot. Crash happened about 20-30 minutes after the download was finished.

$ ./steamcheck.php 1941101 2021-11-08 13:19 -- Mem: 230 908 (diff 100%) 2021-11-08 13:22 -- Mem: 242 596 (diff 5%) 2021-11-08 13:23 -- Mem: 255 052 (diff 5%) 2021-11-08 13:26 -- Mem: 270 036 (diff 5%) 2021-11-08 13:27 -- Mem: 317 840 (diff 17%) 2021-11-08 13:51 -- Mem: 334 416 (diff 5%) 2021-11-08 13:51 -- Mem: 3 662 332 (diff 995%)

Stack trace: https://gist.github.com/FagSoft/87c37172bce2a78a97773c69a625cb00

FagSoft commented 2 years ago

Unfortunately Steam beta seems to be even more prone to crashing with this error. Second time it crashed now. Though seemingly in a slightly different place, further into the methods of libtier0_s.so: https://gist.github.com/FagSoft/b03a47e9fa696df4a5a583ce49854712

Due to some other graphical issues with the beta client, I'm going back to the regular steam client. To avoid other beta-issues polluting this issue.

FagSoft commented 2 years ago

Have had a couple of crashes that I haven't logged, but this is still happening very frequently: https://gist.github.com/FagSoft/4ed8a9fdd2bb68adb40c3bdb9da77aaa

Updated to the latest version of Mint in the meantime, so now I'm running on 20.2 Cinnamon. Please let me know if you need any further details on this issue.

FagSoft commented 2 years ago

A little update to this issue, as I was able to catch the memory leak before Steam tried to allocate too much RAM. Which enabled me to exit steam normally, before it crashed.

I continued the dbg session from last crashlog, just ran the program again. Thus no header info in this dump: https://gist.github.com/FagSoft/f6aa7f84ff11cd1798846e22ece95b85

$ bin/steamcheck.php 1266443 2021-11-10 15:28 -- Mem: 209 600 (diff 100%) 2021-11-10 15:43 -- Mem: 220 612 (diff 5%) 2021-11-10 16:28 -- Mem: 232 072 (diff 5%) 2021-11-10 18:23 -- Mem: 218 324 (diff -5%) 2021-11-10 18:25 -- Mem: 230 452 (diff 5%) 2021-11-10 18:26 -- Mem: 218 664 (diff -5%) 2021-11-10 19:11 -- Mem: 230 084 (diff 5%) 2021-11-10 22:10 -- Mem: 241 624 (diff 5%) 2021-11-10 22:10 -- Mem: 229 308 (diff -5%) 2021-11-10 22:25 -- Mem: 241 832 (diff 5%) 2021-11-10 22:26 -- Mem: 229 516 (diff -5%) 2021-11-10 22:30 -- Mem: 216 032 (diff -5%) 2021-11-10 22:30 -- Mem: 226 964 (diff 5%) 2021-11-10 22:39 -- Mem: 239 632 (diff 5%) 2021-11-10 22:45 -- Mem: 227 188 (diff -5%) 2021-11-11 00:00 -- Mem: 241 088 (diff 6%) 2021-11-11 00:00 -- Mem: 362 420 (diff 50%) 2021-11-11 00:18 -- Mem: 333 320 (diff -8%) 2021-11-11 00:20 -- Mem: 350 152 (diff 5%) 2021-11-11 00:21 -- Mem: 500 828 (diff 43%) 2021-11-11 11:29 -- Mem: 457 124 (diff -8%) 2021-11-11 14:12 -- Mem: 480 660 (diff 5%) 2021-11-11 14:13 -- Mem: 505 016 (diff 5%) 2021-11-11 14:13 -- Mem: 563 868 (diff 11%) 2021-11-11 14:40 -- Mem: 597 472 (diff 5%) 2021-11-11 14:40 -- Mem: 3 082 876 (diff 415%) 2021-11-11 14:51 -- Mem: 2 902 236 (diff -5%) 2021-11-11 14:52 ** Steam dead; Quitting!

FagSoft commented 2 years ago

Still happens on the latest version of Steam, and it seems to be happening a lot more often now as well (3-4 times a day). Same as I saw when I tried the beta at Nov. 8th: https://gist.github.com/FagSoft/bc8c8e9b08e67927d90feeef96cc4946

Memory usage at the stage where it crashed (just after backtrace):

~ $ ps ax | grep /usr/games/steam
   4641 pts/0    S+     0:00 /bin/sh -e /usr/games/steam
  47550 pts/1    S+     0:00 grep --color=auto /usr/games/steam
~ $ ps --pid 4806 --no-headers -o%mem,rss
11.1 3659164

FagSoft commented 2 years ago

Am I the only one who experiences this issue..?

Still happens, and actually got more common after the last steam client update. Tried to completely uninstall Steam (except the steamapps folders), without any change. (Did trigger quite a few game updates, which crashed the client a couple of times for some other reason.) Opted into the Beta, which allowed my games to update without crashing. However, the memory crash is still occouring frequently and with a lot more variety in the time between events.

$ bin/steamcheck.php 1365919 2021-12-13 21:16 -- Mem: 204 704 (diff 100%) 2021-12-13 21:16 -- Mem: 221 276 (diff 8%) 2021-12-13 21:18 -- Mem: 233 688 (diff 5%) 2021-12-13 21:19 -- Mem: 246 348 (diff 5%) 2021-12-13 21:19 -- Mem: 267 824 (diff 8%) 2021-12-13 21:20 -- Mem: 252 300 (diff -5%) 2021-12-13 21:24 -- Mem: 265 064 (diff 5%) 2021-12-13 21:30 -- Mem: 278 408 (diff 5%) 2021-12-13 21:33 -- Mem: 256 848 (diff -7%) 2021-12-13 21:33 ** Steam dead; Quitting!

$ bin/steamcheck.php 1544397 2021-12-13 21:34 -- Mem: 207 264 (diff 100%) 2021-12-13 23:48 -- Mem: 217 764 (diff 5%) 2021-12-14 17:50 -- Mem: 228 672 (diff 5%) 2021-12-14 19:05 -- Mem: 240 452 (diff 5%) 2021-12-14 19:20 -- Mem: 253 236 (diff 5%) 2021-12-15 02:39 -- Mem: 266 520 (diff 5%) 2021-12-15 02:39 -- Mem: 337 724 (diff 26%) 2021-12-15 03:03 -- Mem: 355 052 (diff 5%) 2021-12-15 03:03 -- Mem: 3 649 596 (diff 927%) 2021-12-15 08:46 ** Steam dead; Quitting!

$ bin/steamcheck.php 719650 2021-12-15 08:47 -- Mem: 223 284 (diff 100%) 2021-12-15 08:48 -- Mem: 235 680 (diff 5%) 2021-12-15 08:48 -- Mem: 250 788 (diff 6%) 2021-12-15 08:49 -- Mem: 285 708 (diff 13%) 2021-12-15 08:51 -- Mem: 300 136 (diff 5%) 2021-12-15 08:51 -- Mem: 366 752 (diff 22%) 2021-12-15 09:16 -- Mem: 386 440 (diff 5%) 2021-12-15 09:16 -- Mem: 3 672 348 (diff 850%) 2021-12-15 11:51 ** Steam dead; Quitting!

ValveSoftware / steam-for-linux