steamcmd uses executable stacks

[idl0r]

Hi,

steamcmd on Linux, actually steamclient.so uses executable stacks which it shouldn't. Below is a Kernel log snipped as well as some links that may help to understand and fix the cause of it.

How to reproduce: Boot a Linux Kernel with at least PaX patches Run ./steamcmd.sh

https://wiki.gentoo.org/wiki/Project:Hardened/GNU_stack_quickstart http://pax.grsecurity.net/docs/index.html

[6466122.906072] grsec: From 109.90.39.105: denied marking stack executable as requested by PT_GNU_STACK marking in /home/arma/steamcmd/linux32/steamclient.so by /home/arma/steamcmd/linux32/steamcmd[steamcmd:14155] uid/euid:1036/1036 gid/egid:1051/1051, parent /bin/bash[bash:14152] uid/euid:1036/1036 gid/egid:1051/1051
[6466122.906099] grsec: From 109.90.39.105: denied RWX mprotect of /lib32/ld-2.16.so by /home/arma/steamcmd/linux32/steamcmd[steamcmd:14155] uid/euid:1036/1036 gid/egid:1051/1051, parent /bin/bash[bash:14152] uid/euid:1036/1036 gid/egid:1051/1051
[6466122.908442] grsec: From 109.90.39.105: denied resource overstep by requesting 4197364 for RLIMIT_CORE against limit 4194304 for /home/arma/steamcmd/linux32/steamcmd[steamcmd:14155] uid/euid:1036/1036 gid/egid:1051/1051, parent /bin/bash[bash:14152] uid/euid:1036/1036 gid/egid:1051/1051

I can't even disable mprotect on the library as steamcmd always wants to upgrade itself which will then remove the PaX markings as they're written into the ELF header / PT_GNU_STACK/PT_PAX_FLAGS section.

DEBUGGER=gdb ./steamcmd.sh
GNU gdb (Gentoo 7.5.1 p2) 7.5.1
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
For bug reporting instructions, please see:
<http://bugs.gentoo.org/>...
Reading symbols from /home/arma/steamcmd/linux32/steamcmd...(no debugging symbols found)...done.
(gdb) run
Starting program: /home/arma/steamcmd/linux32/steamcmd
warning: Cannot call inferior functions, Linux kernel PaX protection forbids return to non-executable pages!
warning: the debug information found in "/usr/lib64/debug/lib64/ld-2.16.so.debug" does not match "/lib/ld-linux.so.2" (CRC mismatch).

Redirecting stderr to '/home/arma/Steam/logs/stderr.txt'
Looks like steam didn't shutdown cleanly, scheduling immediate update check
[ 0%] Checking for available updates...
[----] Verifying installation...
Steam Console Client (c) Valve Corporation
-- type 'quit' to exit --

Program received signal SIGSEGV, Segmentation fault.
0xedbeb049 in _dl_map_object_from_fd () from /lib/ld-linux.so.2
(gdb) bt
#0 0xedbeb049 in _dl_map_object_from_fd () from /lib/ld-linux.so.2
#1 0xedbec239 in _dl_map_object () from /lib/ld-linux.so.2
#2 0xedbf709d in dl_open_worker () from /lib/ld-linux.so.2
#3 0xedbf304e in _dl_catch_error () from /lib/ld-linux.so.2
#4 0xedbf6b44 in _dl_open () from /lib/ld-linux.so.2
#5 0xedb82e7e in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?) 

[gdrewb-valve]

We don't use executable stacks and we aren't doing anything special in gcc options so it should be marked, but it isn't. We'll have to figure out what's going on.

[gdrewb-valve]

An upcoming Steam client beta will fix this.

[idl0r]

Thanks!

Is there any ETA by chance?

[gdrewb-valve]

The current Steam client beta has the change.

[idl0r]

Hi,

I am not quite sure then how to enable/subscribe to the beta though.

mkdir steamcmd
wget http://media.steampowered.com/installer/steamcmd_linux.tar.gz
tar xvf steamcmd_linux.tar.gz -C steamcmd/
echo 'universe = beta' > steamcmd/steam.cfg

./steamcmd/steamcmd.sh
Redirecting stderr to '/home/arma/SteamBeta/logs/stderr.txt'
ILocalize::AddFile() failed to load file "public/steambootstrapper_english.txt".
[  0%] Checking for available update...
SteamUpdater: Error: Download failed: http error 0
[  0%] Download Complete.
[----] Verifying installation...
[  0%] Downloading Update...
[  0%] Checking for available update...
SteamUpdater: Error: Download failed: http error 0
[  0%] Download Complete.
SteamUpdater: Error: Steam needs to be online to update.  Please confirm your network connection and try again.
[  0%] !!! Fatal Error: Steam needs to be online to update.  Please confirm your network connection and try again.

The mentioned error log doesn't exist and my network is fine. Fetching the stable version works fine as well.

[gdrewb-valve]

Select the Steam -> Settings menu option. In the dialog look in the Account tab. In the middle there'll be a section called "Beta participation." Opt into the beta and a new client will be downloaded, then restart and you should be running the beta.

[idl0r]

Thanks for your quick reply but I am talking about "steamcmd", the command line interface. I don't have any GUI on my server.

[gdrewb-valve]

Switching to the beta will update steamcmd too. You can also inspect the binaries to see if the GNU_STACK segment is properly marked.

[idl0r]

Ok, but how can I switch to the beta by using steamcmd so on the command line with no GUI?

[gdrewb-valve]

There's a file you can create and put 'publicbeta' in but then you need to run the client to download the beta. You may need to wait for a new public release.

[idl0r]

Hi,

first of all, thanks! It's almost fixed:

[8361642.195112] grsec: From xx.xx.xx.xx: denied RWX mprotect of /home/arma/steamcmd/linux32/steamclient.so by /home/arma/steamcmd/linux32/steamcmd[steamcmd:3430] uid/euid:1036/1036 gid/egid:1051/1051, parent /bin/bash[bash:3426] uid/euid:1036/1036 gid/egid:1051/1051

./steamcmd.sh
...
Loading Steam3.../home/buildbot/buildslave_steam/steam_rel_client_linux/build/src/steamconsole/../common/steam/client_api.cpp (303) : Assertion Failed: ClientAPI_InitGlobalInstance: InternalAPI_Init_Internal failed, most likely because you are missing a 32-bit dependency of steamclient.so (the Steam client is a 32-bit app).

The dependencies are all there so far so it looks like it was caused by preventing the RWX mprotect call.

[gdrewb-valve]

That's going to be more difficult since Valve code doesn't do RWX mprotects. It could be in some third-party code we include (for example there are known issues with Miles and libcef), but finding and seeing what should be done about them is difficult. There won't be any change here in the near term.

[idl0r]

Could you try tunning scanelf or similar tools on it? https://wiki.gentoo.org/wiki/Hardened/PaX_Quickstart

[gdrewb-valve]

I don't believe scanelf does anything for mprotect, what options do you mean?

Also, the problem in third-party code is usually that since it's not a Valve issue we have to go looking elsewhere for a fix and what happens then is out of our hands. For example, the Miles issue requires the Miles owners to fix it and put out an updated libmiles, it isn't something Valve can fix.

[idl0r]

"scanelf -qeR ." might help.

[gdrewb-valve]

The stack markings have already been fixed, so scanelf -qeR doesn't show anything.

[idl0r]

I meant running it over the third-party libraries. Otherwise I'd need a pre/post hook to be able to set the PAX flags without being overridden by steam.

[gdrewb-valve]

I think the report said that steamcmd (a Valve binary) is doing an RWX protect of steamclient.so (a Valve binary) so it would be some third-party source compiled into the Steam binaries.

[idl0r]

Well, yeah. I thought you may link statically against those third-party stuff. Are those parts open source by chance? Or are there any plans to make parts of it open source? So the community could help to fix bugs / send patches.

Or otherwise it should be not that difficult to you to figure out where the mmap* calls are and you could probably forward that to the upstream devs since it also affects Steam.

[idl0r]

Hi,

it looks like you guys changed/fixed some more things. Now I just get:

./steamcmd.sh
/home/arma/steamcmd/linux32/steamcmd: error while loading shared libraries: cannot make segment writable for relocation: Permission denied

Which thus also results in a:

[476702.885388] grsec: From xx.xx.xx.xx: denied RWX mprotect of /home/arma/steamcmd/linux32/steamcmd by /home/arma/steamcmd/linux32/steamcmd[steamcmd:22096] uid/euid:1036/1036 gid/egid:1051/1051, parent /bin/bash[bash:28822] uid/euid:1036/1036 gid/egid:1051/1051

[idl0r]

Could you at least provide a workaround by adding an option that will not override/update specific files or so? steam/steamcmd overrides the files each time I run it and thus it is not usable to me at all since disabling mprotect for Steam will be overridden by itself :/

Or let it call a pre/post hook/script ./steamcmd.sh --pre-cmd /path/to/script --post-cmd /path/to/script.

[gdrewb-valve]

I don't know much about PaX but it seems like using the XATTR_PAX approach would not result in Steam thinking binaries had changed.

A special package exclusion is unlikely as then people could corrupt their installations arbitrarily in ways that would be difficult to diagnose remotely. For example, you exclude steamclient.so from updates because you have marked it for PaX but then it gets out of sync with the rest of Steam and bizarre failures occur due to the mismatch.

[idl0r]

XATTR_PAX is indeed a workaround. Thanks.

[muellermartin]

I'm also using PaX with a Gentoo Hardened Linux kernel and ran into this issue, with the same symptoms idl0r described in his comment from 22 Feb. But thanks to the XATTR_PAX tip, I was able to solve this so far by running paxctl-ng -l -PEmRS /home/steam/linux32/steamcmd.

[idl0r]

paxctl-ng -lm should be enough actually. You may need to do this each time steamcmd gets an update.

[AlexanderCurl]

i have the same issue on debian with grsec. i'm running 4.0.4 kernel with grsec, when i was convert the bin with paxctl -c and paxctl -m the steamcmd gots update and again stuck with this error: error while loading shared libraries: cannot make segment writable for relocation: Permission denied The OS is Debian 8..

[mcgaw]

I've run into this problem trying to set up a dedicated server in a Docker container. The Docker image is based on Archlinux and the host is Alpine Linux. I can get steamcmd to run the first time by altering the binary withpaxctl and the 'm' option, however it then gets updated. If I use attr to set file attribute user.pax.flags to 'm' then this seems to be ignored. I'm not sure where the issue is. According to the PAX config, the Alpine Linux kernel is compiled to respect these pax file attributes. I don't know if I need to do anything with the Docker mounts in this case.