last update: dota2 new runtime misses libraries then segfaults

[sylware]

last update dota2 is not starting anymore, seems to reall happen in libtier0:

[29163.326708] dota2[3465]: segfault at 0 ip 00007faf5d6d3606 sp 00007ffe46399d00 error 6 in libtier0.so[7faf5d61f000+2e5000] likely on CPU 1 (core 1, socket 0) [29163.326719] Code: 2d 3c 24 24 00 31 db 0f 1f 44 00 00 49 8b 74 dc 08 48 85 f6 74 0a 4c 89 ef 31 c0 e8 14 17 f5 ff 48 83 c3 01 41 39 1c 24 7f e2 04 25 00 00 00 00 00 00 00 00 0f 0b c7 04 25 00 00 00 00 00 00

[sylware]

The segfault happens always at 2e5000 in libtiers0.so.

[sylware]

I missed them in the log, but he runtime update misses libs: Loaded libpangoft2-1.0.so, got (nil) failed to dlopen "libpangoft2-1.0.so" error=libpangoft2-1.0.so: cannot open shared object file: No such file or directory Loaded /mnt/128/home/user/.local/share/Steam/steamapps/common/dota 2 beta/game/bin/linuxsteamrt64/libpanorama_text_pango.so, got (nil) failed to dlopen "/mnt/128/home/user/.local/share/Steam/steamapps/common/dota 2 beta/game/bin/linuxsteamrt64/libpanorama_text_pango.so" error=libharfbuzz.so.0: cannot open shared object file: No such file or directory

[sylware]

It misses many more libs: I tried to fix the new dota2 runtime while pulling libs from the heavy steamruntime from the beta steam client: "libpangoft2-1.0.so" is wrong is in the new runtime it is "libpangoft2-1.0.so.0", it misses libharbuzz, libfribid, libthai, etc.

[manticore-projects]

Same here on Arch Linux:

Jul 12 17:13:32 ryzen crash_20230712171332_2.dmp[69933]: Uploading dump (out-of-process)
                                                         /tmp/dumps/crash_20230712171332_2.dmp
Jul 12 17:13:32 ryzen kernel: dota2[69890]: segfault at 0 ip 00007f1ad456f606 sp 00007fff93edb110 error 6 in libtier0.so[7f1ad44bb000+2e5000] likely on CPU 5 (core 2, socket 0)
Jul 12 17:13:32 ryzen kernel: Code: 2d 3c 24 24 00 31 db 0f 1f 44 00 00 49 8b 74 dc 08 48 85 f6 74 0a 4c 89 ef 31 c0 e8 14 17 f5 ff 48 83 c3 01 41 39 1c 24 7f e2 <c7> 04 25 00 00 00 00 00 00 00 00 0f 0b c7 04 25 00 00 00 00 00 00
Jul 12 17:13:32 ryzen systemd[1]: Started Process Core Dump (PID 69934/UID 0).
Jul 12 17:13:33 ryzen systemd-coredump[69936]: [🡕] Process 69890 (dota2) of user 1000 dumped core.

                                               Stack trace of thread 69890:
                                               #0  0x00007f1ad456f606 n/a (/home/are/.local/share/Steam/steamapps/common/dota 2 beta/game/bin/linuxsteamrt64/libtier0.so + 0x16f606)
                                               #1  0x00007f1ad4570673 n/a (/home/are/.local/share/Steam/steamapps/common/dota 2 beta/game/bin/linuxsteamrt64/libtier0.so + 0x170673)
                                               #2  0x00007f1ad4570ac0 n/a (/home/are/.local/share/Steam/steamapps/common/dota 2 beta/game/bin/linuxsteamrt64/libtier0.so + 0x170ac0)
                                               #3  0x00007f1ad3005b0d n/a (/home/are/.local/share/Steam/steamapps/common/dota 2 beta/game/bin/linuxsteamrt64/libengine2.so + 0x405b0d)
                                               #4  0x00007f1ad3006a33 n/a (/home/are/.local/share/Steam/steamapps/common/dota 2 beta/game/bin/linuxsteamrt64/libengine2.so + 0x406a33)
                                               #5  0x00007f1a5113381b n/a (/home/are/.local/share/Steam/steamapps/common/dota 2 beta/game/bin/linuxsteamrt64/libmaterialsystem2.so + 0x6f81b)
                                               #6  0x00007f1ad300586a n/a (/home/are/.local/share/Steam/steamapps/common/dota 2 beta/game/bin/linuxsteamrt64/libengine2.so + 0x40586a)
                                               #7  0x00007f1ad3006e71 n/a (/home/are/.local/share/Steam/steamapps/common/dota 2 beta/game/bin/linuxsteamrt64/libengine2.so + 0x406e71)
                                               #8  0x00007f1ad2e3eeb6 n/a (/home/are/.local/share/Steam/steamapps/common/dota 2 beta/game/bin/linuxsteamrt64/libengine2.so + 0x23eeb6)
                                               #9  0x00007f1ad2d75577 n/a (/home/are/.local/share/Steam/steamapps/common/dota 2 beta/game/bin/linuxsteamrt64/libengine2.so + 0x175577)
                                               #10 0x00007f1ad2d75a0f n/a (/home/are/.local/share/Steam/steamapps/common/dota 2 beta/game/bin/linuxsteamrt64/libengine2.so + 0x175a0f)
                                               #11 0x000055bdb1ea6a20 n/a (/home/are/.local/share/Steam/steamapps/common/dota 2 beta/game/bin/linuxsteamrt64/dota2 + 0x3a20)
                                               #12 0x000055bdb1ea681f n/a (/home/are/.local/share/Steam/steamapps/common/dota 2 beta/game/bin/linuxsteamrt64/dota2 + 0x381f)
                                               #13 0x00007f1ad6360850 n/a (libc.so.6 + 0x23850)
                                               #14 0x00007f1ad636090a __libc_start_main (libc.so.6 + 0x2390a)
                                               #15 0x000055bdb1ea693a n/a (/home/are/.local/share/Steam/steamapps/common/dota 2 beta/game/bin/linuxsteamrt64/dota2 + 0x393a)
                                               ELF object binary architecture: AMD x86-64

crash_20230712171332_2.dmp.gz

[sylware]

@manticore-projects if you look carefully at the steam client output, you should see there are missing libraries in the new linuxruntime for doat2 they did deploy yesterday.

[sylware]

Ok, I did workaround the broken new dota2 linux runtime (missing libs, symbolic links, symbols, and lucky enough not symbol version issue), how to do it: download in the steam client the "Steam Linux Runtime - Sniper". Then go into the your dota2 folder (adapt the path to your system): "/home/user/.local/share/Steam/steamapps/common/dota 2 beta/game/bin/linuxsteamrt64"

Then create the following symbolic links, adapting the paths to your system:

libdatrie.so.1 -> /home/user/.local/share/Steam/steamapps/common/SteamLinuxRuntime_sniper/sniper_platform_0.20230509.49493/files/lib/x86_64-linux-gnu/libdatrie.so.1.4.0

libfribidi.so.0 -> /home/user/.local/share/Steam/steamapps/common/SteamLinuxRuntime_sniper/sniper_platform_0.20230509.49493/files/lib/x86_64-linux-gnu/libfribidi.so.0.4.0

libffi.so.7 -> /home/user/.local/share/Steam/steamapps/common/SteamLinuxRuntime_sniper/sniper_platform_0.20230509.49493/files/lib/x86_64-linux-gnu/libffi.so.7.1.0

libglib-2.0.so.0 -> /home/user/.local/share/Steam/steamapps/common/SteamLinuxRuntime_sniper/sniper_platform_0.20230509.49493/files/lib/x86_64-linux-gnu/libglib-2.0.so.0.6600.8

libgobject-2.0.so.0 -> /home/user/.local/share/Steam/steamapps/common/SteamLinuxRuntime_sniper/sniper_platform_0.20230509.49493/files/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6600.8

libgraphite2.so.3 -> /home/user/.local/share/Steam/steamapps/common/SteamLinuxRuntime_sniper/sniper_platform_0.20230509.49493/files/lib/x86_64-linux-gnu/libgraphite2.so.3.2.1

libharfbuzz.so.0 -> /home/user/.local/share/Steam/steamapps/common/SteamLinuxRuntime_sniper/sniper_platform_0.20230509.49493/files/lib/x86_64-linux-gnu/libharfbuzz.so.0.20704.0

libpangoft2-1.0.so -> libpangoft2-1.0.so.0

libthai.so.0 -> /home/user/.local/share/Steam/steamapps/common/SteamLinuxRuntime_sniper/sniper_platform_0.20230509.49493/files/lib/x86_64-linux-gnu/libthai.so.0.3.1

Depending on how bloated/"old" is your distro, you may need more, or less.

[smcv]

The first time I tried to launch Dota 2 after updating to the new version, it tried to launch in the older soldier + scout runtime, which is not expected to work. The new version of Dota 2 is designed to launch in the new sniper runtime, which works.

Workaround: after letting Steam update Dota 2, completely exit from Steam, then run Steam again. When I did that, Steam's idea of which game should use which runtime caught up with the new situation, and Dota 2 correctly launched in the sniper runtime.

If Steam output is going to a terminal, log file or the systemd journal, when you launch Dota 2, the good result is that it should log a command-line something like this. It'll all be on one long line, I broke it up into shorter lines here for better readability. The important part is that it mentions sniper:

.../reaper SteamLaunch AppId=570 --
.../steam-launch-wrapper --
.../SteamLinuxRuntime_sniper/_v2-entry-point --verb=waitforexitandrun --
.../dota 2 beta/game/dota.sh +engine_experimental_drop_frame_ticks 1 +@panorama_min_comp_layer_dimension 0 -prewarm_parorama\0

The bad result is that it still tries to use the old runtime, which will not work any more for the new version of Dota 2 (notice that it mentions soldier and scout):

.../reaper SteamLaunch AppId=570 --
.../steam-launch-wrapper --
.../SteamLinuxRuntime_soldier/_v2-entry-point --verb=waitforexitandrun --
.../SteamLinuxRuntime/scout-on-soldier-entry-point-v2 --
.../dota 2 beta/game/dota.sh +engine_experimental_drop_frame_ticks 1 +@panorama_min_comp_layer_dimension 0 -prewarm_parorama\0

I tried to fix the new dota2 runtime while pulling libs from the heavy steamruntime from the beta steam client

Copying libraries out of the heavy runtime is not a correct solution and it is not surprising for it not to work.

download in the steam client the "Steam Linux Runtime - Sniper"

The Steam client is meant to do this automatically, because the new version of Dota 2 depends on it. If it does not download it automatically, that's a Steam client bug which should be investigated by client developers.

create the following symbolic links

That should not be necessary. In general, modifying Dota 2 is not meant to be necessary.

Again, the workaround for this is to let Steam update Dota 2, then completely exit from Steam, and run Steam again.

[smcv]

It might also be helpful to use the beta Steam client instead of the normal branch. There have been some recent fixes for games that change between different runtimes which have not all made it into the normal branch yet.

However, I was able to reproduce the problem described in my previous comment with the beta client, so not everything is fixed in the beta yet either.

[smcv]

The first time I tried to launch Dota 2 after updating to the new version, it tried to launch in the older soldier + scout runtime, which is not expected to work

I think this is a Steam client bug, reported as https://github.com/ValveSoftware/steam-for-linux/issues/9835

[sylware]

I hear you: the steam client switching of the steam runtimes is not working.

That would explain many of the issues I have been having lately with recent demos and games.

[sylware]

libthai from the "sniper" libthai will segfault if you go in some options in dota2.

Are you sure dota2 is now using the "sniper" steam runtime?

[sylware]

It is worse, "sniper" steam runtime libthai will 100% and randomly segfault during draft phase.

[sylware]

Okay, found out the issue, since the "sniper" steam-runtime is not setup at all, some libs may miss some critical information... which is the case for libthai looking for its dictionary. Did setup the right env variable, everything seems to work now.

[sylware]

I don't know what's going on, but dota2 is never run into those sniper/scout runtimes, I cannot get any log of anything from that, then since there is very little missing to make dota2 run in the client steamruntime, we can expect dota2 devs and steam client devs to fix all this weirdness.

I did post a temporary "fix" here:

https://steamcommunity.com/app/570/discussions/0/3811784078994041979/?tscn=1689188704

[Espionage724]

Can Dota 2 not be ran with system libraries anymore? I was just running Dota 2 for over a week with steamcmd and Goldberg after that context menu mess made Steam unusable, and all I needed was a symlink for libbz2.so.1.0 to run Dota 2 directly.

Now I get this:

[espionage724@Spinesnap ~]$ cd ~/'Steam/steamapps/common/dota 2 beta/game/bin/linuxsteamrt64' && SDL_AUDIODRIVER='pipewire' SDL_VIDEODRIVER='x11' ~/'Steam/steamapps/common/dota 2 beta/game/bin/linuxsteamrt64/dota2'
Loaded /home/espionage724/Steam/steamapps/common/dota 2 beta/game/bin/linuxsteamrt64/libengine2.so, got 0x55d6d8fcbfe0
Using breakpad crash handler
[S_API] SteamAPI_Init(): SteamAPI_IsSteamRunning() did not locate a running instance of Steam.
[S_API] SteamAPI_Init(): Loaded '/home/espionage724/.steam/sdk64/steamclient.so' OK.
Setting breakpad minidump AppID = 570
Forcing breakpad minidump interfaces to load
Looking up breakpad interfaces from steamclient
Calling BreakpadMiniDumpSystemInit
Looking up breakpad interfaces from steamclient
Calling BreakpadMiniDumpSystemInit
SteamInternal_SetMinidumpSteamID:  Caching Steam ID:  76561199874894657 [API loaded yes]
SteamInternal_SetMinidumpSteamID:  Setting Steam ID:  76561199874894657
Setting breakpad minidump AppID = 373300
Loaded libSDL2-2.0.so.0, got 0x55d6d904c590
Loaded /home/espionage724/Steam/steamapps/common/dota 2 beta/game/bin/linuxsteamrt64/libtier0.so, got 0x55d6d8f3a0a0
Loaded /home/espionage724/Steam/steamapps/common/dota 2 beta/game/bin/linuxsteamrt64/libfilesystem_stdio.so, got 0x55d6d909c630
Loaded /home/espionage724/Steam/steamapps/common/dota 2 beta/game/bin/linuxsteamrt64/libinputsystem.so, got 0x55d6d95ce4d0
Loaded /home/espionage724/Steam/steamapps/common/dota 2 beta/game/bin/linuxsteamrt64/liblocalize.so, got 0x55d6d9910a60
Loaded /home/espionage724/Steam/steamapps/common/dota 2 beta/game/bin/linuxsteamrt64/librendersystemvulkan.so, got 0x55d6d9906630
Loaded /home/espionage724/Steam/steamapps/common/dota 2 beta/game/bin/linuxsteamrt64/libresourcesystem.so, got 0x55d6d98ed180
Loaded /home/espionage724/Steam/steamapps/common/dota 2 beta/game/bin/linuxsteamrt64/libschemasystem.so, got 0x55d6d9909120
ATTENTION: default value of option mesa_glthread overridden by environment.
Loaded /home/espionage724/Steam/steamapps/common/dota 2 beta/game/bin/linuxsteamrt64/libmaterialsystem2.so, got 0x55d6dbd7b900
Loaded libpangoft2-1.0.so, got (nil)
 failed to dlopen "libpangoft2-1.0.so" error=libpangoft2-1.0.so: cannot open shared object file: No such file or directory
Loaded /home/espionage724/Steam/steamapps/common/dota 2 beta/game/bin/linuxsteamrt64/libpanorama_text_pango.so, got (nil)
 failed to dlopen "/home/espionage724/Steam/steamapps/common/dota 2 beta/game/bin/linuxsteamrt64/libpanorama_text_pango.so" error=libbz2.so.1.0: cannot open shared object file: No such file or directory
Segmentation fault (core dumped)

Which doesn't make sense to me since libbz2.so.1.0 is still in the folder, still symlinked, and it was just working fine this morning before I decided to do an app "update" on 570.

[smcv]

Can Dota 2 not be ran with system libraries anymore?

Running Dota 2 without using the Steam Runtime was never a supported thing to do. It might have accidentally worked on some machines, depending on your host operating system and the exact list of libraries you have installed.

The recent change is that recent versions of Dota 2 have are designed to run in a newer, container-based version of the Steam Runtime, and have correspondingly higher library requirements.

cd ~/'Steam/steamapps/common/dota 2 beta/game/bin/linuxsteamrt64' && ... .../game/bin/linuxsteamrt64/dota2

This was not the correct way to start Dota 2, even as an unsupported workaround. When Dota 2 is started by Steam, it is started with current working directory dota 2 beta and started by running the script game/dota.sh. If you launch the executable directly, then library paths are not set up correctly and it is normal that it will fail to start.

[smcv]

libthai from the "sniper" libthai will segfault if you go in some options in dota2.

Are you sure dota2 is now using the "sniper" steam runtime?

The only supported way to use the sniper runtime is for Steam to run the game with sniper as a container. It is meant to do that automatically, but there is a Steam client bug (possibly more than one) that makes this not completely reliable at the moment.

Creating arbitrary symlinks and setting arbitrary environment variables is not a supported way to use any library from sniper, and is not expected to work reliably. The libraries in sniper expect to be used as a container, which will put them in the container's /usr, making it unnecessary to set LD_LIBRARY_PATH or any other special environment variable.

[sylware]

Wait, I don't understand, don't you tell me you did manage to make the "sniper" steamruntime have more complex and expensive requirements than STEAM_RUNTIME=0 ???

Why would we not need LD_LIBRARY_PATH?? This is the only user way to override the library search paths (with LD_PRELOAD). You start to be really scary now.

I repeat the steam client [beta] does not even try to run dota2 in the "sniper" steam-runtime.

[Espionage724]

Running Dota 2 without using the Steam Runtime was never a supported thing to do.

dota.sh has notes about USE_STEAM_RUNTIME=0 implying it was supported in some manner.

[sylware]

@espionage724, USE_STEAM_RUNTIME is for the SDK, STEAM_RUNTIME is to enable or disable the use of the steamruntime, never heard or seen what you said and I went in dota.sh several times, I even made a SH port (to remove the toxic bash dep).

[smcv]

I repeat the steam client [beta] does not even try to run dota2 in the "sniper" steam-runtime

That appears to be a Steam client bug, now described on https://github.com/ValveSoftware/steam-for-linux/issues/9835. I hope that a Steam client developer will be able to solve it. (I am not a Steam client developer and cannot fix this myself.)

dota.sh has notes about USE_STEAM_RUNTIME=0 implying it was supported in some manner.

Something being technically possible doesn't mean that it's supported, even if there is code that specifically checks for it: there is often special-case code to do something unsupportable during development or debugging. dota.sh also has special code for running Dota 2 under a debugger like gdb, but that isn't a supported way to play the game either (I'm reasonably sure!)

Why would we not need LD_LIBRARY_PATH?? This is the only user way to override the library search paths (with LD_PRELOAD)

The sniper runtime uses a container. There is a lot of public information available about it: https://gitlab.steamos.cloud/steamrt/steam-runtime-tools/-/blob/main/docs/container-runtime.md and https://github.com/ValveSoftware/steam-runtime are good places to start.

I have several years' worth of war stories about why all the simpler options (and especially LD_LIBRARY_PATH) don't work reliably, but they would be off topic here. Solving difficult problems requires complexity, and the number of constraints that apply to the Steam Runtime make it a difficult problem. The short version is that if an easier way was viable, we'd have done that instead.

[manticore-projects]

This is so strange: Activating Proton worked for me and Dota is starting now.

[2A4U]

The short version is that if an easier way was viable, we'd have done that instead

I have 2 'easier ways' for you:

1. if-it-ain't-broke-don't-fix-it.
2. REVERT

[sylware]

Don't forget: your infrastructure broke dota2 on distros not using it and where dota2 was running more than fine for more than a decade on different types of distros.

The real culprits are c++ gcc and glibc devs with their manic abuse of symbol versioning and the sourceware binutils devs which force always the last available versions of a symbol/module to be used, namely binaries linked with recent versions won't, very probably, run on older systems (often for no real pertinent reasons).

If fixing has to be done, it is up there which this has to happen. And if you don't want to tackle this issue, the only way is to mandate to link binaries with the oldest set of glibc libs (and you can workaround c++ ABI and libgcc ABI issues with -static-libgcc and -static-libstdc++ compile/link options): which could take the form of ready to use linking environment with a properly configured toolchain to use it, that to ease the work of the "normal" game devs, like a "container".

And now you tell us we must compile in our linux kernel this complex and expansive namespace support which any sane desktop distro could not care less about.

Pretending that "namespace" swapping /usr will magically workaround those issues is very suspicious at best, not to mention it will force distros to follow very probably obscene and rigid specs, and the slight deviation won't be forgiving. In other words, you are yelling at us: use a popular distro or die (exactly what the msft windows toxic troll users say: "use a popular OS").

And I don't know what do you want to achieve here.

Let's take a real life example (I don't recall the name of the game), a game binaries linked with glibc 2.34, which won't run on distros with an "older" glibc (I have 2.33 and the game does not run), now what? You want to completely override the host system libs with namespaces? (including the elf interpreter).

So, you will provide a "container" with a full glibc 2.34 based runtime which will hide perfectly the "older" 2.33 runtime from the host system... and you want that to work for all variants of all distros. Do you realize how obscene and rigid this is? And that down to the formats of the data files.

You will have to load the driver libs from the host with their data files, but how will you let them thru? Some kind of "file system overlay"? How you will know how to do the cherry picking? And you have to pray the "container" has compatible versions of all their system dependencies. The glibc 2.34 would load them, but how you will cherry pick them and sort out them, with their dependencies. For instance, you may have an "older" runtime, but get bleeding-edge drivers with different dependencies, data file formats, layout, etc. How do you want to deal with that with a "containerized" /usr (presuming everything is there, at exactly the right spot, the right format, etc)?

Now, you know how I feel about this "container" workaround of the version issues? And don't forget, dota2 has been working fine for years until now on many different types of distros.

But I stay humble, and listen to you at how you plan to tackle all that.

[kisak-valve]

Hello @sylware, you've already excessively expressed your opinions on how the C++ ABI is handled in GLIBC. Take that issue up with those developers. This is not the place to rehash your opinions on that matter.

[jn64]

My Dota crashes while starting with sniper. Fedora 38, X11, Steam package from RPM Fusion. Tried with both stable and beta Steam client.

Only way I found to launch Dota so far is by setting the compat to Steam Linux Runtime, which runs it with soldier.

I unfollowed the issue, not interested in some delusional user rants. If devs need more info or help testing please @ me.

---

Log of the crash with sniper. Seems to be crashing at the same as other user's logs in this thread

``` GameAction [AppID 570, ActionID 1] : LaunchApp changed task to ProcessingInstallScript with "" GameAction [AppID 570, ActionID 1] : LaunchApp changed task to SynchronizingCloud with "" GameAction [AppID 570, ActionID 1] : LaunchApp changed task to SynchronizingControllerConfig with "" GameAction [AppID 570, ActionID 1] : LaunchApp changed task to SiteLicenseSeatCheckout with "" GameAction [AppID 570, ActionID 1] : LaunchApp changed task to CreatingProcess with "" GameAction [AppID 570, ActionID 1] : LaunchApp waiting for user response to CreatingProcess "" GameAction [AppID 570, ActionID 1] : LaunchApp continues with user response "CreatingProcess" /bin/sh\0-c\0/home/jn/.local/share/Steam/ubuntu12_32/reaper SteamLaunch AppId=570 -- /home/jn/.local/share/Steam/ubuntu12_32/steam-launch-wrapper -- '/home/jn/.local/share/Steam/steamapps/common/SteamLinuxRuntime_sniper'/_v2-entry-point --verb=waitforexitandrun -- '/home/jn/.local/share/Steam/steamapps/common/dota 2 beta/game/dota.sh' +engine_experimental_drop_frame_ticks 1 +@panorama_min_comp_layer_dimension 0 -prewarm_panorama\0 Game process added : AppID 570 "/home/jn/.local/share/Steam/ubuntu12_32/reaper SteamLaunch AppId=570 -- /home/jn/.local/share/Steam/ubuntu12_32/steam-launch-wrapper -- '/home/jn/.local/share/Steam/steamapps/common/SteamLinuxRuntime_sniper'/_v2-entry-point --verb=waitforexitandrun -- '/home/jn/.local/share/Steam/steamapps/common/dota 2 beta/game/dota.sh' +engine_experimental_drop_frame_ticks 1 +@panorama_min_comp_layer_dimension 0 -prewarm_panorama", ProcID 180948, IP 0.0.0.0:0 chdir "/home/jn/.local/share/Steam/steamapps/common/dota 2 beta" ERROR: ld.so: object '/home/jn/.local/share/Steam/ubuntu12_32/gameoverlayrenderer.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS32): ignored. ERROR: ld.so: object '/home/jn/.local/share/Steam/ubuntu12_64/gameoverlayrenderer.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS64): ignored. ERROR: ld.so: object '/home/jn/.local/share/Steam/ubuntu12_32/gameoverlayrenderer.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS32): ignored. ERROR: ld.so: object '/home/jn/.local/share/Steam/ubuntu12_32/gameoverlayrenderer.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS32): ignored. ERROR: ld.so: object '/home/jn/.local/share/Steam/ubuntu12_32/gameoverlayrenderer.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS32): ignored. GameAction [AppID 570, ActionID 1] : LaunchApp changed task to WaitingGameWindow with "" GameAction [AppID 570, ActionID 1] : LaunchApp changed task to Completed with "" pid 181075 != 181074, skipping destruction (fork without exec?) pid 181076 != 181074, skipping destruction (fork without exec?) Loaded /home/jn/.local/share/Steam/steamapps/common/dota 2 beta/game/bin/linuxsteamrt64/libengine2.so, got 0x5576e0d5a1c0 Using breakpad crash handler [S_API] SteamAPI_Init(): Loaded '/home/jn/.local/share/Steam/linux64/steamclient.so' OK. Game process updated : AppID 570 "/home/jn/.local/share/Steam/ubuntu12_32/reaper SteamLaunch AppId=570 -- /home/jn/.local/share/Steam/ubuntu12_32/steam-launch-wrapper -- '/home/jn/.local/share/Steam/steamapps/common/SteamLinuxRuntime_sniper'/_v2-entry-point --verb=waitforexitandrun -- '/home/jn/.local/share/Steam/steamapps/common/dota 2 beta/game/dota.sh' +engine_experimental_drop_frame_ticks 1 +@panorama_min_comp_layer_dimension 0 -prewarm_panorama", ProcID 181078, IP 0.0.0.0:0 Setting breakpad minidump AppID = 570 Forcing breakpad minidump interfaces to load Looking up breakpad interfaces from steamclient Calling BreakpadMiniDumpSystemInit 07/14 18:33:46 Init: Starting. 07/14 18:33:46 InitHandler 07/14 18:33:46 Init Initializing. 07/14 18:33:46 Init: Installing breakpad exception handler for appid(570)/version(8191169)/tid(181078) 07/14 18:33:46 Init Done. 07/14 18:33:46 SetAppID Setting app ID to 570. Looking up breakpad interfaces from steamclient Calling BreakpadMiniDumpSystemInit SteamInternal_SetMinidumpSteamID: Caching Steam ID: 76561197995603452 [API loaded yes] SteamInternal_SetMinidumpSteamID: Setting Steam ID: 76561197995603452 07/14 18:33:46 Setting Steam ID: 76561197995603452 07/14 18:33:46 Setting SteamUniverse: 1 = Public Setting breakpad minidump AppID = 373300 07/14 18:33:46 SetAppID Setting app ID to 373300. Loaded libSDL2-2.0.so.0, got 0x5576e1041a40 Loaded /home/jn/.local/share/Steam/steamapps/common/dota 2 beta/game/bin/linuxsteamrt64/libtier0.so, got 0x5576e0cc8a10 Loaded /home/jn/.local/share/Steam/steamapps/common/dota 2 beta/game/bin/linuxsteamrt64/libfilesystem_stdio.so, got 0x5576e0da9c30 Loaded /home/jn/.local/share/Steam/steamapps/common/dota 2 beta/game/bin/linuxsteamrt64/libinputsystem.so, got 0x5576e15b2000 Loaded /home/jn/.local/share/Steam/steamapps/common/dota 2 beta/game/bin/linuxsteamrt64/liblocalize.so, got 0x5576e18d8800 Loaded /home/jn/.local/share/Steam/steamapps/common/dota 2 beta/game/bin/linuxsteamrt64/librendersystemvulkan.so, got 0x5576e18d27d0 Loaded /home/jn/.local/share/Steam/steamapps/common/dota 2 beta/game/bin/linuxsteamrt64/libresourcesystem.so, got 0x5576e18e2320 Loaded /home/jn/.local/share/Steam/steamapps/common/dota 2 beta/game/bin/linuxsteamrt64/libschemasystem.so, got 0x5576e15b5200 Loaded /home/jn/.local/share/Steam/steamapps/common/dota 2 beta/game/bin/linuxsteamrt64/libmaterialsystem2.so, got 0x5576e1899370 Loaded libpangoft2-1.0.so, got (nil) failed to dlopen "libpangoft2-1.0.so" error=libpangoft2-1.0.so: cannot open shared object file: No such file or directory Loaded /home/jn/.local/share/Steam/steamapps/common/dota 2 beta/game/bin/linuxsteamrt64/libpanorama_text_pango.so, got (nil) failed to dlopen "/home/jn/.local/share/Steam/steamapps/common/dota 2 beta/game/bin/linuxsteamrt64/libpanorama_text_pango.so" error=libbz2.so.1.0: cannot open shared object file: No such file or directory 07/14 18:34:36 Breakpad_FilterCallback 07/14 18:34:36 FilterCallback Calling pre minidump callback /home/jn/.local/share/Steam/steamapps/common/dota 2 beta/game/dota.sh: line 117: 181078 Segmentation fault (core dumped) ${STEAM_RUNTIME_PREFIX} ${GAME_DEBUGGER} "${GAMEROOT}"/${GAMEEXE} "$@" Game process removed: AppID 570 "/home/jn/.local/share/Steam/ubuntu12_32/reaper SteamLaunch AppId=570 -- /home/jn/.local/share/Steam/ubuntu12_32/steam-launch-wrapper -- '/home/jn/.local/share/Steam/steamapps/common/SteamLinuxRuntime_sniper'/_v2-entry-point --verb=waitforexitandrun -- '/home/jn/.local/share/Steam/steamapps/common/dota 2 beta/game/dota.sh' +engine_experimental_drop_frame_ticks 1 +@panorama_min_comp_layer_dimension 0 -prewarm_panorama", ProcID 181078 ```

[smcv]

@manticore-projects:

This is so strange: Activating Proton worked for me and Dota is starting now.

Let's take this to ValveSoftware/steam-for-linux#9835 since it seems more like a Steam client bug than a Dota 2 bug, and I want Steam client developers to be able to see all the necessary info in one place.

@jn64: same.

[smcv]

@jn64:

Only way I found to launch Dota so far is by setting the compat to Steam Linux Runtime, which runs it with soldier (??)

That's not guaranteed to work, especially in the long term, but if it happens to work on your system then it's OK as a short term workaround. Please remember that you have changed this setting: after the underlying Steam client bug gets fixed, you will probably need to turn off "Force the use of ..." so that the game will run in sniper as it is meant to.

[sylware]

@kisak-valve you are wrong, this is not off-topic. The c++ and glibc ABI stuff is the context and start of my post. If you read the rest, it is mostly about how the container is supposed to fix a basic usecase which is based on a real life broken game (and not actually breaking a working game like here). That to evaluate how much more the "container" will need, and how costly for it is to please its requirements, that to be allowed to run dota2 again, the "valve right way", then directly related to the topic of this issue.

[kisak-valve]

@sylware, I'll make this abundantly clear once I have no intention of debating this comment or going further off topic. There will not be a two sided argument in response to this comment.

https://github.com/ValveSoftware/Dota-2/issues/2390#issuecomment-1635074211 very much reads as a call for a developer to waste their time long form defending the existence of Pressure Vessel and the Steam Linux Runtime container environment. That's not going to happen and is off-topic. The high level goal is to get the game running for as many people as possible while being maintainable by the cross-platform Dota 2 team.

You, as an individual user, demand more personal attention than some major distributions and regularly withhold details so that you don't need to defend your personal choice to have a distro-of-one-user system. Often, that means you will hit edge cases that the majority of users will never see. If you want to operate outside of a common distro setup, so be it, but that doesn't warrant additional attention for your specific system setup.

Your philosophy on how software should be maintained is tangential to the specific issue you've reported here, which is that Dota 2 is not running in an environment that it's been configured to use by the game dev(s) and as a result, it's not resolving several dependencies. As of this writing, that is the Steam Linux Runtime - Sniper container environment. Last week, that was the Steam Linux Runtime - Soldier environment.

Having written that, as a moderator and not a Valve developer myself, I'm generally indifferent to the feedback you post on Github, because it's more or less civilized.

---

I suspect that the functional detail as to why Steam is not calling Dota 2 with the Steam Linux Runtime - Sniper entry point is the same as the details to reproduce the issue in #2019. This could be interpreted as the root cause of this issue and since Flatpak 1.14 or newer is generally available in all major distros, from a support perspective it wouldn't be wrong if a Steam dev chose to stronger enforce the use of the Steam Linux Runtime - Sniper environment if the game dev has configured the game to use it.

[smcv]

https://github.com/ValveSoftware/steam-for-linux/issues/9852 is a possible root cause for this. If it's that, then the solution is to go into Steam's Settings window and make sure that "Compatibility" -> "Enable Steam Play for supported titles" is enabled.

I don't think #2019 is necessarily a useful reference here: the original issue reporter with Flatpak 1.10 was disabling a setting that is normally desirable as a workaround for older versions of Flatpak being incompatible with the Steam Linux Runtime framework, and the more recent commenter on that issue with Flatpak 1.14 has not provided enough information to diagnose what is happening, and was asked to open a separate issue.

[sylware]

@smcv, I did enable the compat tools and force it on dota2 (restarted the beta client): and it is now trying to load the dota2 in the soldier runtime, not the sniper runtime. And then the obvious:"bwrap: Creating new namespace failed, likely because the kernel does not support user namespaces" (yeah, I run a lean kernel), need to recompile my kernel with that inside and retest.

@kisak-valve, I have noticed your passive-aggressive attitude towards myself because I am from the minority which is not using a "common"/"popular" (aka "obscenely massive" and "hardly less worth than msft windows"). Weird, this is what I get from windows troll users: "use a popular OS".

[sylware]

@smcv, have those namespaces in (BTW user namespace was disabled by default linux ). Still trying to load in the soldier runtime, but then it complains about a missing /var/cache/ldconfig/ld.so.cache (I don't even have /var).

My distro doesn't even have a ld.so.cache, everything is in LD_LIBRARY_PATH, and the default search dirs of the elf interpreter (I don't recall them, probably /lib /usr/lib).

Then I create an empty /var/cache/ldconfig/ld.so.cache, but then: "x86_64-linux-gnu-capsule-capture-libs: code 61: "/var/cache/ldconfig/ld.so.cache" too small (0 < 16 bytes)"

I guess I have to trick it, I don't even have ldconfig since everything is in LD_LIBRARY_PATH, I guess I need to trick it, or maybe the sniper runtime has some proper fallback code to ignore that.

[smcv]

I don't even have ldconfig

Not having ldconfig is not compatible with the container runtime framework, and honestly, I'm surprised glibc even makes it possible.

We have some documentation for distro vendors describing assumptions made by Steam and its various compatibility frameworks (some, but not all, of this is for the Steam Linux Runtime tool set). It is impossible to avoid making any assumptions at all: some things are just necessary for interoperability. It is possible to adjust or reduce some of these assumptions over time, as we have done in the past for the benefit of distributions like NixOS, Guix and Exherbo, but we cannot spend an unlimited amount of time on distributions with few users and unusual requirements: our time is limited, and if we don't spend it on things that will benefit the most people, that's doing a disservice to Steam's user-base as a whole (every hour I spend helping you is an hour in which I'm not doing work that benefits other Steam users).

Complete source code for the Steam Linux Runtime tool set is available, so it is possible to see precisely what it does, or even compile your own; but please understand that we cannot provide support for modified versions.

I'm being as accommodating as I can, but the more difficult you make it, the less help you will get.

[smcv]

Alternatively, if your operating system is incompatible with the assumptions that Steam makes and you are unwilling to change that, you could consider running Steam inside a container environment that is more similar to a typical Linux environment: perhaps by using Flatpak and the unofficial Flatpak app, or perhaps by using a container environment that is more similar to a typical FHS distribution (this is what NixOS does).

[sylware]

@smcv allright, I got myself an empty and valid (new hash format) /var/cache/ldconfig/ld.so.cache

I managed to run dota2 in the sniper container! I had to remove the "force compatibility" in dota2 properties: it was force-selecting the soldier container. BTW, the sniper container is still failing without a ld.so.cache (the code should just skip it).

Then, now when I try to start dota2, it is properly in the sniper container, but I get nothing, no errors, like the dota2 process is exiting without doing anything.

[smcv]

@jn64:

Please report a separate issue: the root cause cannot be the same for you, because for you, the game is launching in the sniper container as intended but not working correctly. I think libbz2.so.1.0: cannot open shared object file: No such file or directory is probably the key thing in your report, so please mention that in the issue title.

The information requested in https://github.com/ValveSoftware/steam-runtime/blob/master/doc/reporting-steamlinuxruntime-bugs.md will probably be necessary in your case.

[smcv]

I had to remove the "force compatibility" in dota2 properties: it was force-selecting the soldier container

Yes, that's a known Steam Client issue. Ideally it shouldn't be possible to force that runtime for a game that is intended to run in sniper, but right now it is possible, even though it won't work.

the sniper container is still failing without a ld.so.cache

At the moment, that is expected: almost every other Linux distribution has a ld.so.cache, so the situation where there isn't one has never been tried. (NixOS/Guix are the one exception, but they run Steam in a more-FHS-like container that does have a ld.so.cache.)

The path does not necessarily have to be /var/cache/ldconfig/ld.so.cache, and in fact /etc/ld.so.cache is more usual. I think /var/cache/ldconfig/ld.so.cache is just the last one we try, so it will have been the one that appears in the error message.

If you don't have glibc's /sbin/ldconfig, then this will definitely not work, so please set that up before proceeding. The container runtime framework requires ldconfig for various annoying reasons. We cannot rely on LD_LIBRARY_PATH: the same as every other seemingly obvious thing that we're not doing, we tried that already and it didn't work (in this case because some games incorrectly overwrite the entire LD_LIBRARY_PATH variable, breaking the search path that we had carefully set up for them).

If there are other things about your OS (which OS is it?) that don't match https://gitlab.steamos.cloud/steamrt/steam-runtime-tools/-/blob/main/docs/distro-assumptions.md, and they're something that is easily resolvable, then resolving those now will avoid wasting a lot of your time later.

After that, the best next step here would be to collect a log from the container runtime framework, for which see https://github.com/ValveSoftware/steam-runtime/blob/master/doc/reporting-steamlinuxruntime-bugs.md. The short version is: please run Steam with STEAM_LINUX_RUNTIME_LOG=1 and STEAM_LINUX_RUNTIME_VERBOSE=1. After trying to run Dota 2, you should find a log in steamapps/common/SteamLinuxRuntime_sniper/var/slr-latest.log: please attach it or upload it as a gist. This will tell us more about what is going on.

[sylware]

@smcv distro-assumptions.md is literaly useless, pointing on the index of the msdn library would be the same: "You have to respect all those gigantic specs, down to the bit level (and includes the same bugs)".

To stay reasonable, I need to know exactly what's needed, what's done.

I guess you have some verbose logging which reports everything which is done to create the container user image, haven't you? Going thru those logs may already be enough to let me setup something to please your code.

(I can still disable the container for dota2, fix the now broken dota2 linux runtime for my distro and play).

[sylware]

@smcv I had a look at your log to understand how to please your code.

This is what I was whining in a previous post.

You are using the linux "user namespace" as a user level chroot that to reconstruct a "recent" linux userland.

The problem, as I stated in my post above: whatever workaround it brings, you will need to cherry pick the drivers from the host system to inject them properly into your new userland.

You have re-coded the entire glibc/ELF dependency system (ld.so.cache, DT_NEEDED, and I guess DT_RUNPATH and the obsolete DT_RPATH,etc), that to rebuild the ELF dependency graph of the drivers binaries, come on... And you expect that will be enough for all drivers, or you will need to have specific knowledge of the internals all the drivers (vulkan/GL/etc) out there, all the time, for instance how to look for some internal datafiles AND dynamically loaded modules required by some drivers, and this is presuming those drivers give a way to let user programs like your container discover all that.

This is not reasonable.

alsa seems to be one of such driver (datafiles and modules), it seems that for this reason you must force the host system to have pulseaudio/pipewire as a gateway from your container alsa to the host alsa.

With this perspective, I am sorry, I cannot condone this container system.

[sylware]

@smcv

I missed an important point: I guess you are also, in some way, rewriting the ELF symbol versions of the cherry picked drivers binaries to avoid conflicts with the container runtime glibc. For instance my drivers expect some glibc 2.33 symbols and the sniper runtime is glibc 2.31. Or did you modify the ELF glibc loader to ignore such version??

[Espionage724]

Can whatever changed just be undone and kept on a beta branch... for testing? I just want to play Dota 2, and had it in the most ideal configuration that I thought wouldn't be interfered with for simple bot matches. It was working fine with system libraries and no Steam. Fork it as some private beta, give me a password, and as long as that stays on the server for as long as Steam and Dota 2 are in beta-testing, fine :p

Whatever changed had no positive benefit on my system and left me unable to play Dota 2, and Steam was already pushing it with workarounds I was willing to deal with up until the context menu mess (which was pushed to stable users in the broken mess it was).

Is there a clear reason why this had to happen? Did someone wake up and decide it was a fine idea to push an obviously untested new requirement to Dota 2 just because? Why was this not on a beta branch?

[sylware]

@Espionage724

If you don't use that runtime exactly, VAC anti-cheat will stop working and you'll get VAC bans. I guess STEAM_RUNTIME=0 does not work with dota2 anymore (you cannot even disable proton with dota2 installed anymore).

So it seems that one of the moves from "the hidden agenda" of those technical decisions with an obscene technical cost, is actually VAC "anti-cheat" covering the runtime. Then cheaters will move to the userland drivers valve have to pull into their container runtime for direct rendering. A very good thing from the container runtime: they provide an opengl software renderer if no DRI hardware driver is found (like on my system). But I can see the container has the intel i915 dri hardware driver. So valve may plan to provide hardware userland drivers which are able to avoid conflicts with the host userland drivers used by the host windows system (x11 or wayland).

The right way: if end users can get anti-cheats, so a dedicated anti-cheat team too. Then, this is a cat and mouse process: breaks those cheats until barely detectable AI based cheats are made available to end users (like what did happen for FPS). I have strong doubt such small AIs can do anything pertinent in dota2, the "visual scene" is probably too complex for them.

Don't forget: anti-cheat tech is DRM tech. To make users willingly accept DRM tech, do sell them as anti-cheat tech...

And we know what will happen: valve container runtime will diverge after some time from mainstream and massive elf/linux distros, game devs will have to build binaries for valve runtime, which won't run natively even on those distros.

@smcv

There is a way to define which GNU symbol version to target while generating ELF objects. The binutils assembly ".symver" directive. But I would have to patch LLVM(dri+vulkan), ACO(dri+vulkan), and gcc static libstdc++/static libgcc to target the lowest versions as possible: this would be an act of faith, unless explicitely tested and validated by gcc/glibc/llvm/valve devs. Or, in order to work with drivers which are more recent than valve container runtime, valve will need to patch its container ELF runtime code to dynamically link, in an act of faith, the driver with a different version.

This is not reasonable.

BTW, the container shell scripts use the long options for the "timeout" command, is it possible to use the short options instead (-s instead of --signal, I run busybox timeout).

[Espionage724]

Does anyone else have a differing reason why this was implemented? That would be truly absurd to have a game physically on my computer with single-player locally-hosted bot matches become literally unplayable with an untested update that doesn't explain a benefit but only happened... because of anti-cheat?

I've heard of others having games become unplayable because of DRM, but I never imagined it to happen on Linux from Valve of all companies. But maybe I'm jumping ahead and there's another explanation?

Btw great 10 year anniversary Dota 2! This was certainly a surprise :p

[smcv]

No, the primary reason for Dota 2 and other games to be using a container runtime is the same as the reason Proton uses a container runtime: to make it more likely to work, in a consistent way across many Linux distributions, without its developers being limited to only being able to rely on a 10 year old library stack; and to make it continue to work many years into the future, without randomly breaking as a result of changes in the underlying operating system.

There is a lot of public information available about the container runtime: this is a good starting point.

The more space in this issue is filled with complaining about Valve's technical choices, the lower the signal-to-noise ratio will be; and the lower the signal-to-noise ratio gets, the less likely it is that anything constructive is going to happen as a result.

[smcv]

@sylware:

I guess you are also, in some way, rewriting the ELF symbol versions of the cherry picked drivers binaries to avoid conflicts with the container runtime glibc. For instance my drivers expect some glibc 2.33 symbols and the sniper runtime is glibc 2.31.

No, that wouldn't work: even if we somehow ignored the ELF symbol-versioning mechanism, your drivers could still require symbols that were new in glibc 2.33 and didn't exist in 2.31 (like for example pthread_attr_setsigmask_np() seems to be new in 2.32). That's why the container runtime infrastructure uses either your glibc, or the glibc from the container, whichever is newer. That, in turn, is why it needs to know some quite specific things about how and where your glibc is installed, as described in the distro assumptions document.

BTW, the container shell scripts use the long options for the "timeout" command, is it possible to use the short options instead

Maybe. I don't think Valve intends it to be a supported scenario to run Steam on a system where the command-line utilities are not GNU coreutils, and a lot of native Linux games are likely to fail to run in this situation (because their startup scripts assume coreutils behaviour, and would continue to do so even if Steam and the Steam Runtime didn't), so that change is not going to be a high priority; but if you're willing to be constructive and reasonable then this is the sort of thing that can potentially be adjusted. In the longer term I want to stop using the timeout utility at all (it has also caused some weird failure modes when Steam is run as a Snap app), but again, it has not been a priority so far.

[smcv]

It was working fine with system libraries and no Steam

As far as I'm aware (but please bear in mind that I am not a Dota 2 developer), that was never a supported way to run Dota 2. It might have worked on some systems, by luck.

[Espionage724]

No, the primary reason for Dota 2 and other games to be using a container runtime is the same as the reason Proton uses a container runtime: to make it more likely to work, in a consistent way across many Linux distributions, without its developers being limited to only being able to rely on a 10 year old library stack; and to make it continue to work many years into the future, without randomly breaking as a result of changes in the underlying operating system.

That's a bit ironic in my situation, but Dota 2 was using a container runtime already wasn't it? From what I understand, an update changed Dota 2 from one working runtime to another, for no public reason that I'm aware of. If that's the case, why was Dota 2's runtime changed specifically? It certainly couldn't be to make it work on more systems :p

Does Dota 2 even work at all on Linux right now (not through Proton)? https://github.com/ValveSoftware/Dota-2/issues/2394#issuecomment-1644779043

[sylware]

@Espionage724, dota2 has been running without a container within the "classic" steam runtime (setup with LD_LIBRARY_PATH) for 10 years.

@smcv I augmented busybox timeout with those options. I did a few runs looking at the logs: I could not see where my glibc libs (2.33) where pulled or not in the container, but pulling the dri driver (radeonsi) and and vulkan driver (radeon) fail. I don't see x86_64-linux-gnu-capsule-capture-libs logging anything about why though. Any way to enable more logs? The alternative would be a static analysis of your code and runtime analysis via a custom modified x86_64-linux-gnu-capsule-capture-lib. (and I presumed the failures happens there as this is the only hint I could find in the logs).

[smcv]

I did a few runs looking at the logs: I could not see where my glibc libs (2.33) where pulled or not in the container, but pulling the dri driver (radeonsi) and and vulkan driver (radeon) fail. I don't see x86_64-linux-gnu-capsule-capture-libs logging anything about why though.

Please attach a log, with at least STEAM_LINUX_RUNTIME_LOG=1 and STEAM_LINUX_RUNTIME_VERBOSE=1 in the environment. CAPSULE_DEBUG=tools might also be helpful (but is very verbose).