Persistent crashes with cached shaders with Mesa

[ghost]

Environment

Operation System	Flatpak Version	Desktop Environment
Fedora 28	0.99.3	Gnome 3.28.2

$ flatpak run com.valvesoftware.Steam -d

Running Steam on org.freedesktop.platform 18.08 64-bit
STEAM_RUNTIME is enabled automatically
Pins up-to-date!
Installing breakpad exception handler for appid(steam)/version(1533766730)
Installing breakpad exception handler for appid(steam)/version(1533766730)
Installing breakpad exception handler for appid(steam)/version(1533766730)
sh: lspci: command not found
Installing breakpad exception handler for appid(steam)/version(1533766730)
surface_linux.cpp (2095) : Assertion Failed: None of the authentication protocols specified are supported
Assert( Assertion Failed: None of the authentication protocols specified are supported ):surface_linux.cpp:2095

Installing breakpad exception handler for appid(steam)/version(1533766730)
assert_20180814090255_6.dmp[161]: Uploading dump (out-of-process)
/tmp/dumps/assert_20180814090255_6.dmp
[0814/090256.298400:WARNING:crash_reporting.cc(254)] Failed to set crash key: UserID with value: 0
[0814/090256.298490:WARNING:crash_reporting.cc(254)] Failed to set crash key: BuildID with value: 1533764000
[0814/090256.298500:WARNING:crash_reporting.cc(254)] Failed to set crash key: SteamUniverse with value: Public
[0814/090256.298509:WARNING:crash_reporting.cc(254)] Failed to set crash key: Version with value: 20180808213320
[0814/090256.298517:WARNING:crash_reporting.cc(254)] Failed to set crash key: ProductName with value: cefwebhelper
[0814/090256.298530:WARNING:crash_reporting.cc(254)] Failed to set crash key: Vendor with value: Valve
[0814/090256.373070:ERROR:gpu_process_transport_factory.cc(1029)] Lost UI shared context.
Installing breakpad exception handler for appid(steam)/version(1533766730)
Installing breakpad exception handler for appid(steam)/version(1533766730)
Installing breakpad exception handler for appid(steam)/version(1533766730)
CAppInfoCacheReadFromDiskThread took 81 milliseconds to initialize
Installing breakpad exception handler for appid(steam)/version(1533766730)
Installing breakpad exception handler for appid(steam)/version(1533766730)
Installing breakpad exception handler for appid(steam)/version(1533766730)
Installing breakpad exception handler for appid(steam)/version(1533766730)
Installing breakpad exception handler for appid(steam)/version(1533766730)
Installing breakpad exception handler for appid(steam)/version(1533766730)
Installing breakpad exception handler for appid(steam)/version(1533766730)
Installing breakpad exception handler for appid(steam)/version(1533766730)
Installing breakpad exception handler for appid(steam)/version(1533766730)
[0814/090256.651946:WARNING:crash_reporting.cc(254)] Failed to set crash key: UserID with value: 0
[0814/090256.651990:WARNING:crash_reporting.cc(254)] Failed to set crash key: BuildID with value: 1533766730
[0814/090256.652004:WARNING:crash_reporting.cc(254)] Failed to set crash key: SteamUniverse with value: Public
[0814/090256.652016:WARNING:crash_reporting.cc(254)] Failed to set crash key: Version with value: 20180808213320
[0814/090256.652027:WARNING:crash_reporting.cc(254)] Failed to set crash key: ProductName with value: cefwebhelper
[0814/090256.652039:WARNING:crash_reporting.cc(254)] Failed to set crash key: Vendor with value: Valve
CApplicationManagerPopulateThread took 299 milliseconds to initialize (will have waited on CAppInfoCacheReadFromDiskThread)
Installing breakpad exception handler for appid(steam)/version(1533766730)
Installing breakpad exception handler for appid(steam)/version(1533766730)

** (steam:117): WARNING **: 09:02:57.155: Unknown device type 14

** (steam:117): WARNING **: 09:02:57.155: Could not create object for /org/freedesktop/NetworkManager/Devices/1: unknown object type
assert_20180814090255_6.dmp[161]: Finished uploading minidump (out-of-process): success = yes
assert_20180814090255_6.dmp[161]: response: CrashID=bp-661ee177-edfc-42a8-83d6-091d02180813
assert_20180814090255_6.dmp[161]: file ''/tmp/dumps/assert_20180814090255_6.dmp'', upload yes: ''CrashID=bp-661ee177-edfc-42a8-83d6-091d02180813''
[0814/090258.745031:ERROR:web_plugin_impl.cc(38)] Widevine registration is not supported after context initialization
[0814/090259.761795:ERROR:web_plugin_impl.cc(38)] Widevine registration is not supported after context initialization
[0814/090300.657057:ERROR:web_plugin_impl.cc(38)] Widevine registration is not supported after context initialization
Assert( CClientPipe::BWriteAndReadResult: BWaitResult failed, disconnected ):../common/pipes.cpp:766

/home/fedora/.var/app/com.valvesoftware.Steam/data/Steam/steam.sh: line 876:   117 Segmentation fault      (core dumped) $STEAM_DEBUGGER "$STEAMROOT/$STEAMEXEPATH" "$@"
crash_20180814090302_2.dmp[241]: Uploading dump (out-of-process)
/tmp/dumps/crash_20180814090302_2.dmp
crash_20180814090302_2.dmp[241]: Finished uploading minidump (out-of-process): success = yes
crash_20180814090302_2.dmp[241]: response: Discarded=1
crash_20180814090302_2.dmp[241]: file ''/tmp/dumps/crash_20180814090302_2.dmp'', upload yes: ''Discarded=1''

[ghost]

@matthew-cline I'm just a normal user not a linux expert and maybe wrong but, It seem i already install a require below on line 14.

org.freedesktop.Platform/x86_64/18.08 system,runtime

1 com.transmissionbt.Transmission/x86_64/stable     system,current
2 com.valvesoftware.Steam/x86_64/stable             system,current
2 com.visualstudio.code.oss/x86_64/stable           system,current
3 io.github.GnomeMpv/x86_64/stable                  system,current
4 org.gnome.Boxes/x86_64/stable                     system,current
5 org.gnome.Geary/x86_64/stable                     system,current
6 org.jdownloader.JDownloader/x86_64/stable         system,current
7 org.libreoffice.LibreOffice/x86_64/stable         system,current
8 org.mozilla.FirefoxDevEdition/x86_64/master       system,current
9 org.freedesktop.Platform.Compat.i386/x86_64/18.08 system,runtime
10 org.freedesktop.Platform.Compat32/x86_64/1.6      system,runtime
11 org.freedesktop.Platform.VAAPI.Intel/x86_64/18.08 system,runtime
12 org.freedesktop.Platform.ffmpeg/x86_64/1.6        system,runtime
13 org.freedesktop.Platform/x86_64/1.6               system,runtime
14 org.freedesktop.Platform/x86_64/18.08             system,runtime
15 org.freedesktop.Sdk/x86_64/1.6                    system,runtime
16 org.freedesktop.Sdk/x86_64/18.08                  system,runtime
17 org.gnome.Platform/x86_64/3.24                    system,runtime
18 org.gnome.Platform/x86_64/3.28                    system,runtime
19 org.gtk.Gtk3theme.Adapta-Eta/x86_64/3.22          system,runtime

[matthew-cline]

To fix the problem I tried locally emptying out all my installed flatpak stuff and the reinstalling flatpak Steam from scratch. First I had to install runtime/org.freedesktop.Platform/x86_64/18.08 to get things working at all, since that hadn't been pulled in as a dependency. Then I installed org.freedesktop.Sdk/x86_64/18.08 and org.freedesktop.Platform.ffmpeg/x86_64/1.6, but still no go. The system journal is indicating that the crash is happening with a steam process whose comm is si_shader, so maybe it's a graphics driver issue? My machine is an AMD Radeon, and what I have installed is:

Ref                                               Options       
com.valvesoftware.Steam/x86_64/stable             system,current
org.freedesktop.Platform.Compat.i386/x86_64/18.08 system,runtime
org.freedesktop.Platform.VAAPI.Intel/x86_64/18.08 system,runtime
org.freedesktop.Platform.ffmpeg/x86_64/1.6        system,runtime
org.freedesktop.Platform/x86_64/18.08             system,runtime
org.freedesktop.Sdk/x86_64/18.08                  system,runtime

I'm using flatpak version 0.99.3 on Fedora 28 with KDE 5.13.3

[nanonyme]

Ping @valentindavid

[nanonyme]

@matthew-cline are you installing Steam through command line or some GUI?

[matthew-cline]

@nanonyme Installed Steam via the flatpak command line.

[nanonyme]

I wonder whether that migration case to newer runtime on upgrade just works poorly in flatpak currently

[valentindavid]

It might be an issue with Mesa. I would recommend you to remove $HOME/.var/app/com.valvesoftware.Steam/cache/mesa_shader_cache.

[nanonyme]

My hunch based on reading is Steam is having issues connecting to X11 in original report. The container should be letting connections through (that has not been touched) as long as context about what to contact to isn't missing. Does xrandr work inside the app @vibolvireak ?

[ghost]

@nanonyme I may really happy to help if you provide more understandable guide on step by step how to perform your action because as i remark on my first comment that i'm just only a normal user so i don't know how to produce your step according to xrandr etc.

Your question seem to be hard for me :disappointed_relieved:

[valentindavid]

@vibolvireak What was the result of cleaning the mesa shader cache?

[ghost]

@valentindavid @nanonyme sorry my bad after cleaning as you suggest with

rm -r $HOME/.var/app/com.valvesoftware.Steam/cache/mesa_shader_cache

Confirmed: now steam start fine no error and i can back to play my game.

Thank you!

[valentindavid]

I had this problem sometimes. But I always thought it happened with downgrading Mesa only (I have been switching version of runtime back and forth). But I suppose there is a real bug in Mesa here and we should try to reproduce it simply and report to upstream. There are probably others not using flatpak getting the same issue when upgrading distribution. I recommend we keep this issue open and we work on that.

[nanonyme]

Hmm, that actually makes me wonder if it's even a good idea to persist the cache at all if it can result in this kind of issues. Of course currently it's speeding up game launches

[valentindavid]

Could steam-wrapper cache the version of Mesa (both 32 and 64 bits) and cleans up the cache each time it detects a new version?

[nanonyme]

I don't know but the logic would need to work in nVidia proprietary as well. Guess we could write some deployment timestamp in post install and track its changes in steam-wrapper?

[nanonyme]

I guess that wouldn't in practice work except for these major runtime upgrades

[nanonyme]

@vibolvireak do you happen to have Intel GPU? Looks like shader cache was only recently enabled there

[valentindavid]

Then it was probably AMDGPU. I think it would be nice in general to clean the cache. And yes. The stamp is fine. With certain version of the kernel, you can get total gpu freeze when downgrading Mesa or LLVM. Never got that when upgrading though.

We could use: glxinfo | grep "OpenGL renderer string: " for the stamp.

[valentindavid]

Also "OpenGL core profile version string"

[ghost]

@nanonyme @valentindavid I'm using Radeon RX 550.

Remark

No i can't play Dota 2.

When start the game the screen will look like this. I'm not yet test other game yet because i can't purchase it now. But i'm sure this will the mesa issue.

[valentindavid]

@vibolvireak I do not have the same issue and I have a Radeon card. Are there any interesting messages in the console?

If not, make it more verbose:

flatpak run --env=MESA_DEBUG=1 --env=LIBGL_DEBUG=1 com.valvesoftware.Steam

[valentindavid]

You could also try:

• Disable shader pre-caching in Steam's settings.
• Rename .var/app/com.valvesoftware.Steam to start from a clean install.

[ghost]

@valentindavid running command $ flatpak run --env=MESA_DEBUG=1 --env=LIBGL_DEBUG=1 com.valvesoftware.Steam now steam unable to start again and only show message

$ flatpak run com.valvesoftware.Steam
Running Steam on org.freedesktop.platform 18.08 64-bit
STEAM_RUNTIME is enabled automatically
Pins up-to-date!

[nanonyme]

If deleting shader cache again helps, I'm going to suspect there's something reproducibly breaking

[valentindavid]

@vibolvireak Try to remove the cache again.

If it does not work then, just reset.

flatpak run com.valvesoftware.Steam --reset

Then run Steam again.

[ghost]

Well remove .var/app/com.valvesoftware.Steam to start install steam from fresh and go to setting to disable sharder pre-caching but the game still not start

Here is the error while starting dota 2

Opted-in Controller Mask: 70
Game update: AppID 570 "", ProcID 293, IP 0.0.0.0:0
>>> Adding process 293 for game ID 570
GameAction [AppID 570, ActionID 1] : LaunchApp changed task to WaitingGameWindow with ""
ERROR: ld.so: object '/home/fedora/.var/app/com.valvesoftware.Steam/data/Steam/ubuntu12_32/gameoverlayrenderer.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS32): ignored.
shell-init: error retrieving current directory: getcwd: cannot access parent directories: Bad file descriptor
ERROR: ld.so: object '/home/fedora/.var/app/com.valvesoftware.Steam/data/Steam/ubuntu12_32/gameoverlayrenderer.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS32): ignored.
shell-init: error retrieving current directory: getcwd: cannot access parent directories: Bad file descriptor
chdir: error retrieving current directory: getcwd: cannot access parent directories: Success
pid 295 != 294, skipping destruction (fork without exec?)
ERROR: ld.so: object '/home/fedora/.var/app/com.valvesoftware.Steam/data/Steam/ubuntu12_32/gameoverlayrenderer.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS32): ignored.
chdir: error retrieving current directory: getcwd: cannot access parent directories: Success
ERROR: ld.so: object '/home/fedora/.var/app/com.valvesoftware.Steam/data/Steam/ubuntu12_32/gameoverlayrenderer.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS32): ignored.
GameAction [AppID 570, ActionID 1] : LaunchApp changed task to Completed with ""
>>> Adding process 294 for game ID 570
>>> Adding process 296 for game ID 570
ERROR: ld.so: object '/home/fedora/.var/app/com.valvesoftware.Steam/data/Steam/ubuntu12_32/gameoverlayrenderer.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS32): ignored.
ERROR: ld.so: object '/home/fedora/.var/app/com.valvesoftware.Steam/data/Steam/ubuntu12_32/gameoverlayrenderer.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS32): ignored.
Using breakpad crash handler
Game update: AppID 570 "", ProcID 297, IP 0.0.0.0:0
Setting breakpad minidump AppID = 570
Forcing breakpad minidump interfaces to load
Looking up breakpad interfaces from steamclient
Calling BreakpadMiniDumpSystemInit
Looking up breakpad interfaces from steamclient
Calling BreakpadMiniDumpSystemInit
Steam_SetMinidumpSteamID:  Caching Steam ID:  76561198082899156 [API loaded yes]
Steam_SetMinidumpSteamID:  Setting Steam ID:  76561198082899156
Setting breakpad minidump AppID = 373300
>>> Adding process 297 for game ID 570
>>> Adding process 298 for game ID 570
Attempting to create OpenGL context: v3.3 (Core profile)...success.
Attempting to create OpenGL context: v3.3 (Core profile)...success.
DumpContextInfo: OpenGL vendor X.Org
DumpContextInfo: OpenGL renderer Radeon RX 550 Series (POLARIS12, DRM 3.25.0, 4.17.12-200.fc28.x86_64, LLVM 6.0.1)
DumpContextInfo: Using OpenGL context version 4.5
DumpContextInfo: Context supports GLSL version 4.50
 failed to dlopen "/home/fedora/.var/app/com.valvesoftware.Steam/data/Steam/steamapps/common/dota 2 beta/game/bin/linuxsteamrt64/librenderdoc.so" error=/home/fedora/.var/app/com.valvesoftware.Steam/data/Steam/steamapps/common/dota 2 beta/game/bin/linuxsteamrt64/librenderdoc.so: cannot open shared object file: No such file or directory
 failed to dlopen "librenderdoc.so" error=librenderdoc.so: cannot open shared object file: No such file or directory
Installing breakpad exception handler for appid(gameoverlayui)/version(20180808213232)
Installing breakpad exception handler for appid(gameoverlayui)/version(1.0)
ERROR: ld.so: object '/home/fedora/.var/app/com.valvesoftware.Steam/data/Steam/ubuntu12_64/gameoverlayrenderer.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS64): ignored.
crash_20180814223812_1.dmp[320]: Uploading dump (out-of-process)
/tmp/dumps/crash_20180814223812_1.dmp
>>> Adding process 319 for game ID 570
Installing breakpad exception handler for appid(gameoverlayui)/version(1.0)
/home/fedora/.var/app/com.valvesoftware.Steam/data/Steam/steamapps/common/dota 2 beta/game/dota.sh: line 104:   297 Segmentation fault      (core dumped) ${STEAM_RUNTIME_PREFIX} ${GAME_DEBUGGER} "${GAMEROOT}"/${GAMEEXE} "$@"
Installing breakpad exception handler for appid(steam)/version(1533766730)
Game removed: AppID 570 "", ProcID 297 
No cached sticky mapping in ActivateActionSet.Installing breakpad exception handler for appid(gameoverlayui)/version(1.0)
crash_20180814223812_1.dmp[320]: Finished uploading minidump (out-of-process): success = yes
crash_20180814223812_1.dmp[320]: response: CrashID=bp-ab9678c2-f880-45b5-8e79-c854b2180814
crash_20180814223812_1.dmp[320]: file ''/tmp/dumps/crash_20180814223812_1.dmp'', upload yes: ''CrashID=bp-ab9678c2-f880-45b5-8e79-c854b2180814''
pid 320 != 319, skipping destruction (fork without exec?)
Installing breakpad exception handler for appid(steam)/version(1533766730)
Installing breakpad exception handler for appid(steam)/version(1533766730)
Assert( CClientPipe::BWriteAndReadResult: BWaitResult failed, disconnected ):../common/pipes.cpp:766

crash_20180814223824_2.dmp[329]: Uploading dump (out-of-process)
/tmp/dumps/crash_20180814223824_2.dmp
[0814/223824.892285:WARNING:crash_reporting.cc(254)] Failed to set crash key: UserID with value: 0
[0814/223824.892370:WARNING:crash_reporting.cc(254)] Failed to set crash key: BuildID with value: 1533764000
[0814/223824.892376:WARNING:crash_reporting.cc(254)] Failed to set crash key: SteamUniverse with value: Public
[0814/223824.892382:WARNING:crash_reporting.cc(254)] Failed to set crash key: Version with value: 20180808213320
[0814/223824.892387:WARNING:crash_reporting.cc(254)] Failed to set crash key: ProductName with value: cefwebhelper
[0814/223824.892392:WARNING:crash_reporting.cc(254)] Failed to set crash key: Vendor with value: Valve
[0814/223824.910217:ERROR:gpu_process_transport_factory.cc(1029)] Lost UI shared context.
[0814/223824.936009:WARNING:crash_reporting.cc(254)] Failed to set crash key: UserID with value: 76561198082899156
[0814/223824.936057:WARNING:crash_reporting.cc(254)] Failed to set crash key: BuildID with value: 1533766730
[0814/223824.936076:WARNING:crash_reporting.cc(254)] Failed to set crash key: SteamUniverse with value: Public
[0814/223824.936093:WARNING:crash_reporting.cc(254)] Failed to set crash key: Version with value: 20180808213320
[0814/223824.936107:WARNING:crash_reporting.cc(254)] Failed to set crash key: ProductName with value: cefwebhelper
[0814/223824.936126:WARNING:crash_reporting.cc(254)] Failed to set crash key: Vendor with value: Valve
[0814/223826.937375:ERROR:web_plugin_impl.cc(38)] Widevine registration is not supported after context initialization
[0814/223828.020554:ERROR:web_plugin_impl.cc(38)] Widevine registration is not supported after context initialization
crash_20180814223824_2.dmp[329]: Finished uploading minidump (out-of-process): success = yes
crash_20180814223824_2.dmp[329]: response: Discarded=1
crash_20180814223824_2.dmp[329]: file ''/tmp/dumps/crash_20180814223824_2.dmp'', upload yes: ''Discarded=1''
[0814/223829.038917:ERROR:web_plugin_impl.cc(38)] Widevine registration is not supported after context initialization
[0814/223830.025507:ERROR:web_plugin_impl.cc(38)] Widevine registration is not supported after context initialization
Setting breakpad minidump AppID = 769
Steam_SetMinidumpSteamID:  Caching Steam ID:  76561198082899156 [API loaded no]
[0814/223830.070864:WARNING:crash_reporting.cc(254)] Failed to set crash key: SteamUniverse with value: Public
[0814/223831.020457:ERROR:web_plugin_impl.cc(38)] Widevine registration is not supported after context initialization
Found unsupported CDM version 1.4.9.1088, using last known good version 1.4.8.1008

[nanonyme]

Try flatpak run --env=MESA_GLSL_CACHE_DISABLE=true com.valvesoftware.Steam

[nanonyme]

If disabling cache removes this noise, we can set that in the json. You can always set local override for it if you want it back on

[ghost]

Finally running flatpak run --env=MESA_GLSL_CACHE_DISABLE=true com.valvesoftware.Steam can get the game to work.

[valentindavid]

I suppose this is a bug with Mesa. Maybe some incompatibility with your kernel version.

You can save the environment if you want. So you can launch Steam from a launcher.

flatpak override --env=MESA_GLSL_CACHE_DISABLE=true com.valvesoftware.Steam

[nanonyme]

@valentindavid can you handle JSON change? I'm not currently close to a computer

[nanonyme]

This is breaking at least two other person's Steam, probably more

[valentindavid]

@nanonyme I am not sure we should disable it for everybody. Let's see first how often this happens.

[ghost]

@valentindavid @nanonyme no worry i will keep it unchange until you have a fix and make sure a next release is fixed. Thank you guy :)

[valentindavid]

@matthew-cline Do you still have the issue even after cleaning up the shader cache?

[matthew-cline]

@valentindavid Removing the shader cache fixed everything.

[nanonyme]

Did you also suffer of the issue where this came back or was removing cache once enough to get rid of this for good?

[matthew-cline]

Removing it once was enough, I've quit and restarted Steam multiple times since then.

[nanonyme]

I'll try to figure out a way to do cache flushing soon but this is expected to collide with the other work with #60 since it'll most likely touch same file. To avoid unnecessary work with merge conflicts I'll try to get that sorted first. For now cache is completely disabled (this especially so since I've been considering rewriting the entire steam-wrapper into Python that execs Steam at the end given our logic is getting quite complex for a batch script)

[nanonyme]

@valentindavid any idea if Steam's own shader caching (cloud-based) could be interfering?

[valentindavid]

@nanonyme, @vibolvireak said that disabling Steam's shader cache was not helping, So I do not think this is the problem.

Of course there could be issues there one day. But I could not think we can do much about it.

[nanonyme]

I'm having issues verifying that the cloud sync fixes work, I'm probably going to need another day on that front still at least. After that will fully focus on this again

[nanonyme]

@valentindavid do you think we can rely on things being stable enough within a runtime version? I didn't yet find any reasonable way to detect GL version changes but I did locate a /lib/os-release with contents

NAME=Freedesktop.org VERSION="18.08 (Flatpak runtime)" VERSION_ID=18.08 ID=org.freedesktop.Platform

[nanonyme]

Apparently also on some Flatpak versions this works

flatpak-spawn cat /.flatpak-info

[valentindavid]

@nanonyme If an extension is used, the runtime version is not what we want. I would recommend we make a small C program that just query the OpenGL and Vulkan drivers. I think glGetString for OpenGL. Not sure about Vulkan.

[nanonyme]

@valentindavid I thought this was only a problem specifically with Mesa shader cache (and a possible invalidation bug on their part). Shouldn't that always come directly through runtime?

[nanonyme]

My current idea is using /.flatpak-info which looks like an ini file that has app-path, runtime-path and runtime-extensions. That covers extensions too and you don't need to mess with guessing stuff from GL strings

[valentindavid]

@nanonyme There were .GL extensions of updated version of Mesa for 1.6. And we will have the same for 18.08.

[nanonyme]

Ah. Well, you can still detect changes in them from changed runtime-extensions string.

[nanonyme]

@valentindavid @vibolvireak shader cache is now re-enabled by default. The logic for wiping the cache is currently still quite dumb, it just flushes the cache on every start of Steam.