search

ValveSoftware / steam-runtime

A runtime environment for Steam applications
Other
1.21k stars 86 forks source link

Steam Linux - Adobe substance not launch #701

Open adx-fr opened 3 weeks ago

adx-fr commented 3 weeks ago

Hello,

After asking for steam support for “Adobe Substance 3D Painter 2023” doesn’t launch.

Teamthe support returned me :

“I am sorry to hear that you are experiencing an issue running Substance 3D Painter 2023 using our Steam Play Proton feature for Linux.

Please submit all issues or bugs occurring when using this feature, to the Valve - Proton Github bug database using a GitHub account. This database is regularly reviewed by the Valve development team.”

Steam Support Chris”

I tested run to steam application and to the terminal but same result, it appear substance image after open steams application and nothing.

In the terminal :

steam.sh[6467]: Running Steam on debian 12 64-bit steam.sh[6467]: STEAM_RUNTIME is enabled automatically setup.sh[6505]: Steam runtime environment up-to-date! steam.sh[6467]: Steam client's requirements are satisfied CProcessEnvironmentManager is ready, 6 preallocated environment variables. [2024-11-07 18:31:53] Startup - updater built Nov 5 2024 21:33:18 [2024-11-07 18:31:53] Startup - Steam Client launched with: '/home/adx/.steam/debian-installation/ubuntu12_32/steam' '-srt-logger-opened' 'steam://run/2199970' 11/07 18:31:53 minidumps folder is set to /tmp/dumps 11/07 18:31:53 Init: Installing breakpad exception handler for appid(steam)/version(1730853027)/tid(6578) [2024-11-07 18:31:53] Loading cached metrics from disk (/home/adx/.steam/debian-installation/package/steam_client_metrics.bin) [2024-11-07 18:31:53] Using the following download hosts for Public, Realm steamglobal [2024-11-07 18:31:53] 1. https://client-update.fastly.steamstatic.com, /, Realm 'steamglobal', weight was 900, source = 'update_hosts_cached.vdf' [2024-11-07 18:31:53] 2. https://client-update.akamai.steamstatic.com, /, Realm 'steamglobal', weight was 100, source = 'update_hosts_cached.vdf' [2024-11-07 18:31:53] 3. https://client-update.steamstatic.com, /, Realm 'steamglobal', weight was 1, source = 'baked in' [2024-11-07 18:31:53] Verifying installation... [2024-11-07 18:31:53] Verification complete UpdateUI: skip show logo Steam logging initialized: directory: /home/adx/.steam/debian-installation/logs

XRRGetOutputInfo Workaround: initialized with override: 0 real: 0xf03d7780 XRRGetCrtcInfo Workaround: initialized with override: 0 real: 0xf03d6050 steamwebhelper.sh[6589]: Using supervisor /home/adx/.steam/root/ubuntu12_32/steam-runtime/amd64/usr/bin/steam-runtime-supervisor steamwebhelper.sh[6589]: Starting steamwebhelper under bootstrap sniper steam runtime via /home/adx/.steam/debian-installation/ubuntu12_64/steam-runtime-sniper.sh steamwebhelper.sh[6589]: Using CEF sandbox (try with -no-cef-sandbox if this fails) steamwebhelper.sh[6589]: Starting steamwebhelper with Sniper steam runtime at /home/adx/.steam/debian-installation/ubuntu12_64/steam-runtime-sniper/_v2-entry-point exec ./steamwebhelper -lang=en_US -cachedir=/home/adx/.steam/debian-installation/config/htmlcache -steampid=6578 -buildid=1730853027 -steamid=0 -logdir=/home/adx/.steam/debian-installation/logs -uimode=7 -startcount=0 -steamuniverse=Public -realm=Global -clientui=/home/adx/.steam/debian-installation/clientui -steampath=/home/adx/.steam/debian-installation/ubuntu12_32/steam -launcher=0 -use_xcomposite_workaround -no-restart-on-ui-mode-change --valve-initial-threadpool-size=6 --valve-enable-site-isolation --enable-smooth-scrolling --disable-gpu-compositing --disable-gpu --password-store=basic --log-file=/home/adx/.steam/debian-installation/logs/cef_log.txt --disable-quick-menu --disable-features=SpareRendererForSitePerProcess,DcheckIsFatal,ValveFFmpegAllowLowDelayHEVC Steam Runtime Launch Service: starting steam-runtime-launcher-service Steam Runtime Launch Service: steam-runtime-launcher-service is running pid 6819 bus_name=com.steampowered.PressureVessel.LaunchAlongsideSteam Desktop state changed: desktop: { pos: 0, 0 size: 4480,1440 } primary: { pos: 0, 0 size: 2560,1440 } Caching cursor image for left_ptr, size 24x24, serial 1764, cache size = 0 BuildCompleteAppOverviewChange: 132 apps BuildCompleteAppOverviewChange: 132 apps chdir "/home/adx/.steam/debian-installation/steamapps/common/Substance 3D Painter 2023" Game Recording - would start recording game 2199970, but recording for this game is disabled Adding process 6947 for gameID 2199970 qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "" even though it was found. This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Available platform plugins are: minimal, offscreen, xcb.

Game Recording - game stopped [gameid=2199970] Removing process 6947 for gameID 2199970”

I have removed steam application and "Adobe substance" but same result.

Can you help me, I need this application for my work.

Thanks for your prompt return.

My system OS : Debian 12 Bookworm (actual OS) OS : Fedora 41 (tested but same result) Steam Version : 1730853027 Kernel : 6.1.0-26-amd64 (tested same result) Kernel : 6.10.11+bpo-amd64 Desktop : GNOME v.43.9 CPU : Intel Core i9-10850K GPU : GeForce RTX 4070 Ti SUPER

smcv commented 3 weeks ago

As a short-term workaround, running Steam with steam -compat-force-slr off should return to previous behaviour (but this is unlikely to remain possible permanently).

@touraill-adobe, @Froyok, is this the same failure mode that you mentioned on a different issue? This issue is about Substance 3D Painter 2023, and you mentioned Substance 3D Painter 2024, but if that's an update of essentially the same product a year later, it seems reasonably likely that they're related.

To investigate this further, we will need a log file for the scenario where it fails. https://github.com/ValveSoftware/steam-runtime/blob/master/doc/reporting-steamlinuxruntime-bugs.md#essential-information

@kisak-valve, do you want to move this to ValveSoftware/steam-runtime?

smcv commented 3 weeks ago

If this is a Qt problem, setting environment variable QT_DEBUG_PLUGINS=1 would also provide helpful information. You can do that at the same time as STEAM_LINUX_RUNTIME_LOG=1 and/or STEAM_LINUX_RUNTIME_VERBOSE=1.

smcv commented 3 weeks ago

As a wild guess based on the dependencies of libQt6XcbQpa.so.6 on my Debian system, this might be because the Steam Runtime has never guaranteed that it will provide libxcb-icccm.so.4, libxcb-image.so.0, libxcb-keysyms.so.1 or libxcb-render-util.so.0.

touraill-adobe commented 3 weeks ago

@touraill-adobe, @Froyok, is this the same failure mode that you mentioned on a different issue?

Yes it is the same problem, basically all recent versions of the Substance 3D apps no longer run after the client update (unless you launch the client with -compat-force-slr off).

The errors we are getting are

touraill-adobe commented 3 weeks ago

Some additional context: while we were initially building against the Steam runtime, we stopped doing so several years ago because the runtime was getting way too old for our needs. So we started building on a more recent Ubuntu and made that our only supported target. Since we bundle most of our dependencies, things mostly worked out.

I'm not entirely sure why the change broke everything, I still haven't caught up with the latest evolutions of the runtime, by I read that the legacy runtime relied solely on LD_LIBRARY_PATH while the new ones are full-fledged containers, which could explain why some of our non-bundled dependencies are no longer found I suppose.

touraill-adobe commented 3 weeks ago

In the future we might try to go back to building within the Steam runtime since it now seems to be well maintained, but what solutions do we have for our older applications (or the ones we might release in the meantime)? Currently we told our users to run their client with the "don't force slr flag" but you seem to hint that this flag will probably go away in a future release of the client.

Wouldn't it be possible to let developers specify that their game should run natively, outside of any Steam runtime (leaving to them the responsibility of which distributions they would support)? Just launch the executable and that's it, without containers or LD_LIBRARY_PATH? Or at least to propose the legacy runtime in the compatibility settings next to the new runtimes, both on the client and on Steamworks?

touraill-adobe commented 3 weeks ago

I'm seeing here that you would be willing to update the runtime with some libraries to restore compatibility, that would be a nice solution I suppose. Is there anything I can try locally to check what are the missing pieces in the sniper runtime? If I can somehow add missing libraries one by one we could have an overview of what it would take to make the runtime compatible?

touraill-adobe commented 3 weeks ago

Here is the full log in case it helps, I didn't use the verbose flag as it would require some curation on the output but there is not much more information than just "xcbcursor0 is missing".

(And thanks for pointing me to those handy environment variables, I was looking for a way to troubleshoot some completely unrelated issues I have with a Windows game I run through Proton ^^).

adx-fr commented 3 weeks ago

Hello,

It works, thank you very much for your help

De : Luc Touraille @.> Envoyé : jeudi 7 novembre 2024 21:06 À : ValveSoftware/steam-runtime @.> Cc : adx-fr @.>; Author @.> Objet : Re: [ValveSoftware/steam-runtime] Steam Linux - Adobe substance not launch (Issue #701)

@touraill-adobehttps://github.com/touraill-adobe, @Froyokhttps://github.com/Froyok, is this the same failure mode that you mentioned on a different issue?

Yes it is the same problem, basically all recent versions of the Substance 3D apps no longer run after the client update (unless you launch the client with -compat-force-slr off).

The errors we are getting are

/home/luc/.local/share/Steam/steamapps/common/Substance 3D Designer 2024/Adobe Substance 3D Designer: error while loading shared libraries: libbrotlidec.so.1: cannot open shared object file: No such file or directory

qt.qpa.plugin: From 6.5.0, xcb-cursor0 or libxcb-cursor0 is needed to load the Qt xcb platform plugin. qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "" even though it was found. This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

— Reply to this email directly, view it on GitHubhttps://github.com/ValveSoftware/steam-runtime/issues/701#issuecomment-2463113427, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BMXYWYCMURYG7TRGWFTBRYTZ7PB23AVCNFSM6AAAAABRL53ED2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINRTGEYTGNBSG4. You are receiving this because you authored the thread.Message ID: @.***>

smcv commented 3 weeks ago

It works, thank you very much for your help

I'm going to interpret this as: the workaround in https://github.com/ValveSoftware/steam-runtime/issues/701#issuecomment-2463044682 was successful, but there is still a dependency problem that needs to be solved by Adobe and/or Valve for future-proofing.

smcv commented 3 weeks ago

the legacy runtime relied solely on LD_LIBRARY_PATH while the new ones are full-fledged containers

Yes, this. In the new container runtime, your application can't see libraries from the host system in its search path (unless they are part of the recursive dependencies of the user-space graphics drivers, which for technical reasons we have to borrow from the host in most cases).

This is good because it means the application can't make assumptions about libraries that it hopes are installed on the host system for other reasons, which maximizes the probability that an application that works in 2024 will still work in 2034, even after the host system has undergone incompatible changes (for example replacing libtiff.so.5 with an incompatible libtiff.so.12 or whatever number it has reached by then). It also means that an application that works successfully on a typical distro like Ubuntu will probably also work on more obscure distros like Exherbo or NixOS.

However, for existing applications that already make assumptions about host-system libraries, it can cause regressions like this one. Arguably we're still providing better consistency between distributions, but unfortunately the consistent situation in this case is "doesn't work", which obviously isn't really what anyone wants.

you would be willing to update the runtime with some libraries to restore compatibility

It depends on the library, really. If the required library is small, has no dependencies that we don't already ship, and is under a "normal" permissive license, then we can probably just add it. If it's large, has large dependencies, or is under a strange license that lawyers don't want to have to think about, then we might have to say no.

The libxcb-* family of libraries are an example of something that we can probably just add - I don't set the policy, so I can't promise anything, but they haven't been a problem in the past.

The Qt 6 family is an example of something that we would not add without some significant thought, because it's rather large: we don't want to let the size of the runtime get out of control, because it's a disk space cost that every Steam-on-Linux user needs to pay.

In the future we might try to go back to building within the Steam runtime

Please do - that's what it's there for. The recommended version for new code is Steam Runtime 3 'sniper', for which you can get a SDK (as a Docker/Podman/Toolbx/OCI container, or a chroot tarball) from https://gitlab.steamos.cloud/steamrt/sniper/sdk/.

There is also a "Platform" OCI image available from https://gitlab.steamos.cloud/steamrt/sniper/platform which you can use for compatibility testing. This only provides a rather old version of Mesa (so you'll probably only get software rendering) but hopefully that's enough for a simple smoke-test that verifies that your application can launch and load its mandatory plugins. The only libraries and versions that you can rely on having access to at runtime are the ones in the Platform image - on end-user systems, you will often see newer versions of libraries like Mesa and glibc, but those are an implementation detail and not part of the runtime's ABI guarantee.

The intention is to have a series of newer container runtimes in future, "the same shape" as sniper but based on newer Debian releases. At the moment we have an internal-only branch based on Debian 12, and another based on Debian 13 prereleases. The Debian-12-based branch isn't available for general use at the moment because we don't want to take up more disk space than we need to, and it's possible that we'll skip past Debian 12 and go straight to a Debian-13-based runtime instead - we'll have to see what the timing of the Debian 13 freeze is like.

while we were initially building against the Steam runtime, we stopped doing so several years ago because the runtime was getting way too old for our needs. So we started building on a more recent Ubuntu and made that our only supported target. Since we bundle most of our dependencies, things mostly worked out.

Yes, unfortunately this is a familiar story from various app/game developers. Steam Runtime 1 'scout' is very, very old (based on Ubuntu 12.04), but we can't upgrade it without breaking backwards-compatibility, and until relatively recently we were not ready to replace it with a container-based environment that mitigates the backwards-compatibility concerns. But, one of the flaws of the old LD_LIBRARY_PATH-based runtime is that it was very easy for developers who are doing this to pick up dependencies on implementation details of the host OS, either accidentally or intentionally.

For new software that you are willing and able to recompile, the intention is to encourage app/game developers to use Steam Runtime 3 'sniper'. For example, Valve has done this for Counter-Strike 2 and other flagship titles. I'm guessing you might want to do this for at least the 2024 versions of your apps?

For legacy software that is shipped as a set of frozen binaries and not going to be recompiled, the Steam Linux Runtime 1.0 (scout) container is our backwards-compatibility story. For example, Valve is currently still using this for the older Half-Life and Portal games. It's intended to approximate the result of running the old LD_LIBRARY_PATH-based scout runtime on a Debian 10 host system, which is what was current at the time we started this container work.

If your app was compiled in a newer environment anyway (which environment?), then you might find that sniper is a better fit. sniper is based on Debian 11 from 2021, so it's generally newer than Ubuntu 20.04 (but older than 22.04).

Is there anything I can try locally to check what are the missing pieces in the sniper runtime?

The general answer is to identify all of your ELF executables and libraries, parse their DT_NEEDED headers (for example with objdump -T -x $exe_or_library | grep NEEDED or eu-readelf -d $exe_or_library | grep NEEDED), discard duplicates (sort -u), discard anything that you bundle with the application (like Qt 6), and tell us what's left.

Alternatively, if it's possible to launch your application in a Docker/Podman/Toolbx container with software rendering, you could try running it in the Platform image from https://gitlab.steamos.cloud/steamrt/sniper/platform. The advantage of that approach is that you can easily experiment with adding new packages (enter the container as root and do an apt install in the usual Debian/Ubuntu way), and then iterate until you have what you need, and send us a wishlist of missing libraries when you have the app working.

It is also possible for an app/game developer to edit the SLR runtime in-place for local experiments (see https://gitlab.steamos.cloud/steamrt/steam-runtime-tools/-/blob/main/docs/slr-for-game-developers.md#running-in-a-sdk-environment, https://gitlab.steamos.cloud/steamrt/steam-runtime-tools/-/blob/main/docs/slr-for-game-developers.md#running-in-a-modified-platform-or-sdk-environment) and prepare a wishlist of missing libraries that way. Please don't suggest this to your users as a workaround, though! This is definitely in "unsupported tinkering" territory.

If you're willing to switch an application to be run under SLR 3.0 (sniper), the official definition of what is in the sniper runtime is: sniper ABI. Most of those libraries are at the versions found in Debian 11, with a few key libraries like SDL and Vulkan-Loader backported from Debian 12 or newer (inspect the Platform or SDK OCI image to find specific versions).

For legacy applications that run under SLR 1.0 (scout), officially the only thing you can rely on is what's in the scout ABI, mostly at the versions found in Ubuntu 12.04 (which is ancient), although it does have a significant number of backported packages from Debian 10 or newer. However, SLR 1.0 isn't actually a completely strict Steam Runtime 1 'scout' container, because we found that in practice, a lot of game developers have been compiling their games in a non-scout environment. Instead, it's actually a Steam Runtime 2 'soldier' container (based on Debian 10), with a LD_LIBRARY_PATH-based compatibility layer to make it fully scout-compatible. This means that 99% of the libraries from the soldier ABI are available, generally at versions matching Debian 10 (but, again, some key libraries like SDL and Vulkan-Loader have been backported) - the intention is to approximate the user experience of running the old LD_LIBRARY_PATH runtime on a minimal Debian 10 system, on the basis that "most" older games ran successfully in that environment in practice.

smcv commented 3 weeks ago

Here is the full log in case it helps, I didn't use the verbose flag as it would require some curation on the output but there is not much more information than just "xcbcursor0 is missing".

Specifically, the information we need is the missing SONAME, libxcb-cursor.so.0. It's interesting that your Qt build needs this, when the one on my Debian system doesn't - perhaps a newer version of Qt dropped that particular dependency?

Unfortunately, this is only the first missing library dependency: if your libqxcb.so also needs some other missing library, like perhaps libxcb-icccm.so.4, then we won't find that out until after libxcb-cursor.so.0 has been added.

I think what I'm going to do as a first step is to see what the size impact would be if we add most of the libxcb-* libraries as one big batch, which might save us some time compared with trying to isolate a completely minimal set.

touraill-adobe commented 3 weeks ago

I think I can try ~linking~ copying libraries in our app install to see which ones are missing.

touraill-adobe commented 3 weeks ago

Adding only libxcb-cursor.so.0 makes Substance 3D Designer 2024 launch again; we'll need to do additional testing to see if there are other issues with other libraries we load dynamically, but that is promising!

touraill-adobe commented 3 weeks ago

Thank you so much for the very detailed summary of the current state of things and options we have, this is very very much appreciated :heart:. As a side note, we were discussing internally about how we missed all those latest developments on the SLR, and learned about the breaking update only after it was released; what would be the best channel to keep up-to-date with those changes, is there a newsletter/mailing list we should subscribe to?

touraill-adobe commented 3 weeks ago

Sustance 3D Painter 2024 has more missing dependencies: libsharpyuv.so.0, libjbig.so.2.1, libLerc.so.4, libwebp.so.7, libxkbfile.so.1, libtiff.so.5, libxcb-cursor.so.0 (again, only considering what it takes to get the app running, not any libraries that would be lazy loaded).

smcv commented 3 weeks ago

libxcb-cursor.so.0

I imagine this one should be reasonably straightforward. I'll see what happens if I add the whole libxcb- family.

libxkbfile.so.1

I'm a little surprised that you need this when you're (presumably!) not implementing a desktop environment; but it's small, so we might be able to add this too. I'll look into it.

libtiff.so.5

This is present in both soldier and sniper, and has been for a long time, so I'm surprised you're not finding it. Can I see a Help -> Steam Runtime Diagnostics from an affected system, please? (Attached here, or as a gist, or emailed to smcv at collabora.com)

libwebp.so.7

This is not part of either the soldier or sniper ABI, which includes the older libwebp.so.6 instead... and libwebp doesn't have ELF symbol-versioning, so if there is ever a situation where libwebp.so.7 and libwebp.so.6 get pulled into the same process space (possibly via indirect dependencies), your application is probably going to crash.

Sorry, we are unlikely to add libwebp.so.7 for this reason. You could consider bundling a copy, at your own risk.

If the dependency is in a plugin, you might be able to mitigate the crash risk by dlopening it with RTLD_LOCAL|RTLD_DEEPBIND.

Or, if you are able to recompile in the sniper SDK against the sniper ABI so that you get a dependency on the older libwebp.so.6, that would ensure that everything is consistent.

In the longer term, if we do a new runtime based on Debian 12 (Steam Runtime 4, 'medic' exists as an internal beta but is not currently available to the public) or Debian 13 (Steam Runtime 5 exists as an internal alpha, but we certainly can't usefully release that to the public until Debian 13 has frozen), those will include libwebp.so.7.

libwebp is a nice example of what I was saying about your app being at the mercy of the wider OS ABI: the fact that it has broken ABI in the past suggests that it will do so again in the future. When it does, today's version of your app will stop working with the legacy LD_LIBRARY_PATH runtime, because it will still be assuming the host OS has libwebp.so.7, but a sufficiently new host OS will only have libwebp.so.8 (or newer).

libsharpyuv.so.0

This is associated with newer versions of libwebp, so, sorry, unlikely to be added to soldier or sniper for the same reasons.

libjbig.so.2.1

Sorry, I don't know a library of this name and neither does apt-file search. Are you sure that's the required SONAME? Do you have an example of a host OS where it can be loaded successfully when using the legacy LD_LIBRARY_PATH runtime?

The libjbig that we know about is libjbig.so.0 from jbigkit, which is GPL, therefore linking it into proprietary software is a licensing concern, and we've patched our libtiff to exclude it on that basis. Although if this is the library you had in mind, I see that it's possible to negotiate commercial licenses, and maybe Adobe has done that? Or maybe Adobe has a different libjbig.so.2.1?

I think you'll need to bundle your own copy of this library.

libLerc.so.4

This wasn't added to Debian until after Debian 11, so isn't trivially available in sniper (and I'm somewhat surprised the app works reliably if this is a mandatory dependency, since it seems like a feature that host OSs might just skip). We also disable this feature in our libtiff to minimize dependencies, because it seems to be a GIS-related file format that isn't popular in other contexts. In the absence of demand for this format from game developers, I think the answer is probably: sorry, if you need this, you'll have to bundle it.

touraill-adobe commented 3 weeks ago

libxkbfile.so.1

I'm a little surprised that you need this From what I can tell this is a dependency of libQt6WebEngineCore.so.6. And I didn't answered earlier but libxcb-cursor.so.0 is a dependency of the Qt platform plugin libqxcb.so.

I think I must have rushed my investigation on Painter 2024 earlier, I redid some tests and only had to copy libxcb-cursor.so.0 and libxkbfile.so.1 (presumably I was not running in the sniper runtime?)... Sorry about that, I think we (Adobe) need to take some time to do a more thorough study across the various apps and versions, discuss the options with the various teams involved and come back to you then. We have the -compat-force-slr off workaround for now so let's take the time to devise the correct solution.

Also we are much aware that by leaving the runtime, we gave up on compatibility with many distributions. This is a trade-off we accepted at the time, and communicated to our users; we will definitely reevaluate this choice in the light of the new runtimes/SDKs you are providing, which seem really neat. So we have no expectations that our older apps will continue running on newer distributions (at least not without tweaks), and we don't expect Steam to somehow make those apps forward compatible, since we decided to build outside the runtime, with full knowledge of the facts.

Some additional information I'd like to have to discuss with the teams before coming back to you:

And again, thanks for the help! :bow:

smcv commented 3 weeks ago

presumably I was not running in the sniper runtime?

It's possible that you were affected by https://github.com/ValveSoftware/steam-for-linux/issues/11448, a regression in which some titles are run in an incorrect runtime, which I found today while investigating this issue. I hope that a Steam developer can address that issue soon (I cannot).

You can tell for sure which runtime you are in by looking at ~/.steam/steam/logs/console_log.txt, or the process tree and arguments (pstree -alp, systemd-cgls).

Steam Linux Runtime 1.0 (scout) looks like this in console_log.txt. In this example I was running Tomb Raider 2013, and I've broken up a long line into shorter pieces to make it a bit clearer:

[2024-11-05 13:42:29] /bin/sh\0-c\0 /home/desktop/steam-client-beta/ubuntu12_32/steam-launch-wrapper -- /home/desktop/steam-client-beta/ubuntu12_32/reaper SteamLaunch AppId=203160 -- '/home/desktop/SteamLibrary/steamapps/common/SteamLinuxRuntime_soldier'/_v2-entry-point --verb=waitforexitandrun -- '/home/desktop/SteamLibrary/steamapps/common/SteamLinuxRuntime'/scout-on-soldier-entry-point-v2 -- '/home/desktop/SteamLibrary/steamapps/common/Tomb Raider/TombRaider.sh'\0

The key facts to look for are that it uses steamapps/common/SteamLinuxRuntime_soldier to launch a container, and then runs steamapps/common/SteamLinuxRuntime/scout-on-soldier-entry-point-v2 inside that container to provide full compatibility with the legacy LD_LIBRARY_PATH-based environment.

Steam Linux Runtime 3.0 (sniper) looks more like this:

/bin/sh\0-c\0 /home/desktop/steam-client-beta/ubuntu12_32/steam-launch-wrapper -- /home/desktop/steam-client-beta/ubuntu12_32/reaper SteamLaunch AppId=404410 -- '/home/desktop/SteamLibrary/steamapps/common/SteamLinuxRuntime_sniper'/_v2-entry-point --verb=waitforexitandrun -- '/home/desktop/SteamLibrary/steamapps/common/Endless Sky/endless-sky'\0

The key facts to look for are that this time, it uses steamapps/common/SteamLinuxRuntime_sniper to launch the container, and then it does not run steamapps/common/SteamLinuxRuntime/scout-on-soldier-entry-point-v2 inside the container.

For comparison, the legacy LD_LIBRARY_PATH runtime looked like this, with a steam-launch-wrapper and a reaper, but no compatibility tool between the reaper and the app/game itself:

/bin/sh\0-c\0 /home/desktop/steam-client-beta/ubuntu12_32/steam-launch-wrapper -- /home/desktop/steam-client-beta/ubuntu12_32/reaper SteamLaunch AppId=302380 -- '/home/desktop/SteamLibrary/steamapps/common/Floating Point/Floating Point.x86'\0

If you want to check this programmatically from inside your application, the facts to look at are:

[1] typically /home/.../ubuntu12_32/steam-runtime but please don't rely on this [2] typically /home/.../steamapps/common/SteamLinuxRuntime/var/steam-runtime but please don't rely on this

smcv commented 3 weeks ago

libxkbfile.so.1

From what I can tell this is a dependency of libQt6WebEngineCore.so.6.

Aha, that makes sense. I'm hoping to add this library to future versions of all container runtimes so that you can rely on having it around.

And I didn't answered earlier but libxcb-cursor.so.0 is a dependency of the Qt platform plugin libqxcb.so.

I'm also hoping to add this, and a bunch of the other libxcb-* family of libraries for completeness (if we can prevent a future request by already having the right libraries in place, so much the better).

I think I must have rushed my investigation on Painter 2024 earlier, I redid some tests and only had to copy libxcb-cursor.so.0 and libxkbfile.so.1

If those are all you need, then adding them to the runtime will be sufficient. If you find that you need more libraries, please ask and we can discuss the right way to address them (either adding them to the runtime, or asking you to bundle them with your app).

If necessary we can do this iteratively, smallest/easiest/least-controversial libraries first, until everything is resolved.

what is the deprecation plan for compat-force-slr off?

would it be worth considering to add [an option to use the legacy LD_LIBRARY_PATH runtime] to the "Linux Runtime" settings on Steamworks so that we can force this behavior for our apps without user intervention?

Sorry, I can't answer those policy questions: my role here is to make each of the various runtimes work as well as possible, and it's up to Valve/Steam developers which ones they do or don't offer to app developers and users. @TTimo might be able to shed some light on this, though?

My personal recommendation, if you can, would be to build your app as purely x86_64 (no 32-bit code) in the Steam Runtime 3 'sniper' SDK, and flag it to run under Steam Linux Runtime 3.0 (sniper). That is the "modern" runtime environment that we are increasingly aiming to recommend for new titles.

I realise that for your particular app, this will probably not be possible until we can add a few more libraries to sniper. Adding the first few of those is in progress.

if you do plan to remove [the ability to select the legacy LD_LIBRARY_PATH runtime], how about having a "native" mode that runs the app outside of any runtime, for developers that want to use Steam as a selling and distribution platform but not as a runtime environment (taking care of compatibility issues themselves)?

Again, whether that is offered or not is not up to me. It's an option that is available for Valve to consider, if they want to.

I would request that if this is added, it should not be referred to as "native", because that can mean several different things in the context of Linux gaming and it's better if we can avoid that ambiguity. We often use "native" in the context of "native Linux executables, not a Windows executable via Wine/Proton/emulation", because I don't think there's any other good name for that; but others often use "native" to mean some different thing like "host OS libraries, not the Steam Runtime" (I think that's what you're requesting here), or "host OS packaging system, not Flatpak/Snap/AppImage", or "non-sandboxed, not Flatpak/Snap/Firejail", or even "using the OS's GUI look-and-feel, not an unrelated UI toolkit". Sometimes this comes with an implicit value-judgement where "more native" is assumed to be better (which is not necessarily true).

From the point of view of the Steam Runtime, I would describe the environment you're suggesting here as having two major aspects: the host OS's library stack (with no special compatibility layers), and no container boundary. That is not actually something that Steam has ever officially offered: even in 2013 when Steam first became available on Linux, all games ran in the LD_LIBRARY_PATH runtime environment, to provide cross-distro compatibility with a then-current version of Ubuntu.

There are two compatibility issues with having your app use such a thing. The first is that on distros that do not match the one you support (older, newer, or just different), some libraries and external executables can be incompatible because they are older or newer, making ABI compatibility a moving target over time; it seems you might be addressing that by officially requiring a single, specific distro. The second, which is more subtle, is that you're also relying on the list of libraries and external executables that are preinstalled by the distro: for example, you might be able to assume that users of a relatively complete GNOME or KDE desktop will always have libxkbfile.so.1, but users of a more minimal desktop environment might not have anything that pulls in that library.

702 is an interesting example of the "list of executables" problem: typical GNOME or KDE systems will have xrandr, but a heavily-Wayland-based environment like Sway might not, so LWJGL's assumption that it could run the xrandr program was already troublesome even before the recent changes.

smcv commented 3 weeks ago

Addition of libxkbfile.so.1 and libxcb-cursor.so.0 is queued up for a future beta.

Note that because of the way these components get released internally, this change will show up as an update to the Steam Linux Runtime 2.0 (soldier) tool. This is not the same thing as using betas of the Steam Client or the Steam Linux Runtime 1.0 (scout) tool.

You can opt-in to beta versions of SLR 2.0 by locating it in your Library, then following instructions similar to https://help.steampowered.com/en/faqs/view/5A86-0DF4-C59E-8C4A - but instead of changing the properties of Counter-Strike 2, you would change the properties of SLR 2.0 to use the client_beta branch.

touraill-adobe commented 2 weeks ago

Addition of libxkbfile.so.1 and libxcb-cursor.so.0 is queued up for a future beta.

That's great news, thank you! I can confirm that at least Sampler 2023, Designer 2024 and Painter 2024 now launches successfully in the sniper runtime beta :).

smcv commented 2 weeks ago

For historical native Linux titles (which are assumed to be targeting scout unless configured otherwise), this is believed to be fixed by Steam Linux Runtime 2.0 (soldier) beta build 0.20241111.107902, which was made available on the client_beta branch yesterday. This beta adds libxkbfile.so.1, libxcb-cursor.so.0, and an assortment of other smaller libraries which might help you in your quest. The beta will be copied to the default branch when it has had some more testing.

Note that even though the compatibility tool indicated in the UI is Steam Linux Runtime 1.0 (scout), the change actually happened in Steam Linux Runtime 2.0 (soldier) because of the way the compatibility tools are implemented, so it is Steam Linux Runtime 2.0 (soldier) where you would need to opt-in to the beta branch to get the fix. Using the beta of Steam Linux Runtime 1.0 (scout) is not necessary, but is harmless (it doesn't change often, and is currently identical to the default branch).

For native Linux titles where the developer has specifically opted-in to using sniper, this is fixed in Steam Linux Runtime 3.0 (sniper) beta build 0.20241111.107899, also made available on its client_beta branch yesterday.

To opt-in to a compatibility tool beta, follow instructions similar to https://help.steampowered.com/en/faqs/view/5A86-0DF4-C59E-8C4A, but instead of locating Counter-Strike 2 and changing its Properties, you would do the same for SLR 2.0 and/or SLR 3.0 (whichever one is appropriate, if in doubt do both).

If I am reading SteamDB correctly, Adobe's Substance 3D Designer 2024 and Substance 3D Painter 2024 are currently assumed to be scout titles, so Steam will run them in a Steam Linux Runtime 1.0 (scout) environment, which is a soldier container plus a scout compatibility shim. If Adobe would prefer Steam to run them in the newer sniper container by default, then you should configure app mappings for them via the partner web interface. If this is done correctly, they should show up as configured similarly to Counter-Strike 2 (search that page for app_mappings and you'll see what I mean).

Similarly, the retired title that is labelled as either Substance 3D Sampler 2023 or Sampler 2024 (depending where I look) is also currently assumed to be a scout title. If you want to run it as a sniper title, please configure it accordingly in the partner web interface.

If you find that you need other libraries, we can discuss whether they should be added to a suitable runtime or bundled with your app. It might be better if we take that discussion to a new issue report per batch of libraries, so that each batch is immediately actionable.

smcv commented 2 weeks ago

For historical native Linux titles (which are assumed to be targeting scout unless configured otherwise), this is believed to be fixed by Steam Linux Runtime 2.0 (soldier) beta build 0.20241111.107902, which was made available on the client_beta branch yesterday. This beta adds libxkbfile.so.1, libxcb-cursor.so.0, and an assortment of other smaller libraries which might help you in your quest. The beta will be copied to the default branch when it has had some more testing.

This change has now been copied to the default branch.

SteamLinuxRuntime_soldier/VERSIONS.txt for a version that fixes this issue will say depot 0.20241111.107902 or newer.

If Adobe would prefer Steam to run [various apps] in the newer sniper container by default, then you should configure app mappings for them via the partner web interface

This doesn't seem to have changed: these titles are still being treated as targeting Steam Linux Runtime 1.0 (scout). If Adobe's intention is for the newer Steam Linux Runtime 3.0 (sniper) container to be used, then action will still required from Adobe to set that up.

If you find that you need other libraries, we can discuss whether they should be added to a suitable runtime or bundled with your app. It might be better if we take that discussion to a new issue report per batch of libraries, so that each batch is immediately actionable.

I think it might be best if we close this issue now, and an Adobe developer can open a new issue as a continuation of this one if you find that more libraries are necessary.

touraill-adobe commented 2 weeks ago

Sorry I missed your message from last week.

This change has now been copied to the default branch.

This is great news, we will do a new round of testing outside the beta, thanks!

This doesn't seem to have changed: these titles are still being treated as targeting Steam Linux Runtime 1.0 (scout).

Yes, we didn't make the switch on Steamworks before the release of the beta runtime as it would have prevented the -compat-force-slr off workaround from working (if I understand correctly). We will do it now that the change is available in the default branch.

I think it might be best if we close this issue now, and an Adobe developer can open a new issue as a continuation of this one if you find that more libraries are necessary.

We noticed some problem related to libssl, we will open a new issue if need be once we have better understanding of the issue (Qt is complaining about a version mismatch).

Thanks for your help!

smcv commented 2 weeks ago

we didn't make the switch [to sniper] on Steamworks before the release of the beta runtime as it would have prevented the -compat-force-slr off workaround from working (if I understand correctly). We will do it now that the change is available in the default branch.

Ah, yes, I can see that being a reason to not switch to sniper.

You are correct to think that -compat-force-slr off has no effect on sniper titles: they always run in the sniper container, and there is no alternative to that. It's only scout titles that have a choice between a legacy LD_LIBRARY_PATH-based runtime and a newer container runtime.

libxkbfile.so.1, libxcb-cursor.so.0 and friends are included in sniper depot 0.20241111.107899 and newer. A version of sniper that includes these libraries appeared in the default branch for the first time yesterday, at around the same time as the corresponding soldier release.

We noticed some problem related to libssl

If it helps, OpenSSL in sniper is libssl.so.1.1 from the OpenSSL 1.1 branch, currently at version 1.1.1w-0+deb11u2 (we use the version from Debian 11 LTS, without further changes). soldier has an older version, 1.1.1n-0+deb10u6 from Debian 10 LTS.

Confusingly, there is also libssl3.so, which is part of Mozilla NSS, currently at version 2:3.61-1+deb11u4 in sniper (also from Debian 11 LTS) or 2:3.42.1-1+deb10u8 in soldier (also from Debian 10 LTS).

After you have moved these titles to run in sniper, recompiling your copy of Qt in a sniper container should give you a binary that will not have warnings about version mismatches.