gameoverlayrenderer.so crashes a 32-bit glibc built with gcc-12

[ghost]

Your system information

• Steam client version (build number or date): Dec 2 2021 07:35:48
• Distribution (e.g. Ubuntu): Gentoo
• Opted into Steam client beta?: [Yes/No] Yes
• Have you checked for system updates?: [Yes/No] Yes

Please describe your issue in as much detail as possible:

Glibc built with gcc-12 makes gameoverlayrenderer.so crash during a call to dlvsym():

• gcc-12 enables "-ftree-vectorize" by default even at -O2, where dlvsym() is built with vectorization enabled, and a glibc-internal struct is being populated with a vector store (no problem so far).
• gameoverlayrenderer.so and/or its user (the "reaper" binary) however are apparently built with a different stack layout (pre-SSE2 32bit alignment instead of the now common 16-byte alignment), where the above mentioned vector op tries to store to a misaligned address and causes a #GP (movaps / vmovdqa).
• Now this might not be a problem for Ubuntu/Debian since they probably build their 32bit x86 glibc for i586 targets, but it will become problematic for other more modern systems, and it'd be really nice of the steam client to accomodate these as well.

Steps for reproducing this issue:

1. build 32-bit x86 glibc with either gcc-12 OR add the gcc flag "-ftree-vectorize" to the build ("-O2 -ftree-vectorize -msse" is enough)
2. try to start any game from within the steam client
3. watch "reaper" crash in 32bit libc.so.6::dlvsym()

[thesamesam]

Note that in Gentoo, we've worked around this by adding a USE=stack-realign USE flag to the sys-libs/glibc and sys-libs/ncurses ebuilds.

But it's indeed not Gentoo specific. There's a lot of history there.

https://gitweb.gentoo.org/repo/gentoo.git/commit/sys-libs/glibc?id=02aa6328a720c86d0157c4582f7e5bac72ae9296.

[ghost]

What I do not understand is why the steam client and its associated ecosystem still rely on the over-2-decade old 32bit stack layout. The new layout existed and was used long before the steam client came to linux, but now it seems it forces the ecosystem around it use technologies which have long been obsoleted by better alternatives (this and now the reliance on DT_HASH of EAC).

[tgurr]

We just ran into this issue on Exherbo Linux as well now that we unmasked GCC 12.x, .local/share/Steam/ubuntu12_32/reaper segfaulting:

[   81.971623] traps: reaper[3223] general protection fault ip:f7c86819 sp:ff95c840 error:0 in libc.so.6[f7c22000+182000]
[  527.975995] traps: reaper[75543] general protection fault ip:f7c86819 sp:ffae4490 error:0 in libc.so.6[f7c22000+182000]
[  823.272890] traps: reaper[134429] general protection fault ip:f7c86819 sp:ff82ed80 error:0 in libc.so.6[f7c22000+182000]

and no game making use of Proton was working anymore. We now also apply the workaround to build our 32bit glibc with -mstackrealign as already done on Gentoo stated above, thanks @thesamesam for mentioning this here. Considering this was reported such long time ago already it's sad it hasn't already been fixed, usually you guys are pretty fast to fix things.