Space Trash Scavenger (1759350)

[AlSqPg]

Compatibility Report

• Name of the game with compatibility issues: Space Trash Scavenger
• Steam AppID of the game: 1759350

System Information

• GPU: Steam Deck (OS 3.4.11 20231005.1)
• Video driver version: Steam Deck (OS 3.4.11 20231005.1)
• Kernel version: Steam Deck (OS 3.4.11 20231005.1) 5.13.0-valve24-1-neptune
• Link to full system information report as Gist: Very vanilla Steam Deck (OS 3.4.11 20231005.1) (no addons or alterations)
• Proton version: Proton 8, Proton Experimental (and a few subversions within), however it sometimes works with proton 7 (with proton 7 it either crashes, freezes for a minute then reboots, or works...)

I confirm:

• [X] that I haven't found an existing compatibility report for this game.
• [X] that I have checked whether there are updates for my system available. steam-1759350.zip

Symptoms

I am developing a unity built PC game which is releasing soon into early access. This game runs well on PC, and until recently it has worked well on Deck. Following a particular update, the build suddenly fails on Deck alone. It crashes when loading the world in Proton 8 (specifically Proton 8.0-4, as well as experimental). Strangely, I found it sometimes works in Proton 7, though it usually crashes there as well. With a lot of testing, I have found that when the texture of a model (a scaffold) is loaded the game crashes - though only when this object is visible on screen. There is no stacktrace or log from the Unity engine about any issue, the FPS drops to 0, and then eventually the whole process just dies, and it seems to take out Steam itself too. Testers tell me when launched in 'game' mode on the Steam Deck the whole OS will reboot. When launched in 'desktop' mode on the Steam Deck then the Game and Steam processes both die (and Steam restarts a few seconds later on its own). It works on normal linux installs on desktops (from arch to ubuntu) in proton 8, works on Windows fine, the issue only seems to be encountered on the Steam Deck. Prior versions of the game before that scaffold work fine.

This same issue has been consistently reported by multiple SteamDeck testers.

Reproduction

This game is currently unreleased but entering early access shortly. If any Valve employee should see this and help, perhaps you can install and test it (or you can contact me privately). Just install the game, set compatibility mode to Proton 8 or Proton Experimental, launch it, click the disclaimer that this is early access (left stick to move cursor, 'A' to select), click New Game (optionally click skip cinematics to speed up the next part, then click Start, the game will go to a cloudy loading screen that states "Initialising..." at the top-left and a spinning loading indicator in the lower-right, then the loading indicator freezes, then a couple seconds later the steam deck reboots.

[AJuujarvi]

@AlSqPg I started taking a look at this, and I cannot seem to reproduce this after checking 8 different world generations. I checked with Proton 8.0-4 and Proton Experimental and I am able to get in game. I looked around everywhere and didn't get a crash, and was able to do actions in game. Is it possible I didn't get the scaffold you mentioned loaded in any of the worlds I did? If so, a seed for a world that has this would be helpful.

My settings for in game were default settings. If I need to change any of these to trigger a crash, please let me know.

[AlSqPg]

Hi there. Thanks for talking a look, much appreciated. Well! You are right. What great news! I seem to have inadvertantly fixed the issue over the last week - and had not checked in on the Deck since.

The fix must have come in when I took a closer look at the textures across the game. Firstly, we found that compiling on unity with "half sized" textures across the board resolved it.

Following that I went back to full size, but realised there are many large textures including at least one 8k one, and these were always preloaded by unity. I downsized some of these, so there were no 8k textures, and then also especially focussed on downsizing the many large 4k ambient occlusion and metallic maps in the game that didn't need that detail. (But I hadnt got a chance to test this one on Deck.)

Could it have been on specific "corrupt" texture? Out of memory somehow? Not sure... and not sure why it was behaving like it was, but happy to hear its working.

Are you interested to see the build where it was broken from last week? I can put one on a test branch if so if its of value to understand why it took out Proton.

[AJuujarvi]

@AlSqPg Ahh I am happy to hear that it was figured out! Talking with a few folks, I don't think there would be much value from the build in this case, but the offering is very much appreciated!

In my testing, I did find an issue that is unknown right now on if it is a game bug or a Proton bug that I would like to mention for you just in case. It's something that we haven't seen before on our end. With the default controller config of "Gamepad", typing into a text field is very difficult. I will bring up the keyboard to type, press a single button which does get registered in the text field, but then I am not able to type any more. I notice when I press a button on the keyboard, the game recenters the reticule to the middle of the screen. Seemingly it refocuses it then at the same time, which is why it can't register any more inputs until I close the keyboard and refocus on the text field.

[AlSqPg]

Thanks - I will need to take a look at this. I think it's our systems for controller input swapping from keyboard to controller and back again causing this. I'm going to try making steam input action mapping mosue keyboard on the deck to see if it resolves that.

[AlSqPg]

@AJuujarvi Hi again. Well today I have some bad news. The issue has returned! Yesterday I made a new build (0.333), no major changes, just minor bug fixes. I'm looking at the commit and I didnt even change any textures or add files to the project compared with 0.332, which I believe was the build you found to work.

Out of the blue, the very strange 1 fps bug has returned on Deck. Again, it is only while looking at particular objects in the scene.

This is inline with how it first came about: an innocuous change to the project, and sudden unexplained breaking on Deck only.

Note, on PC it works fine, I also hear it runs fine in a linux environment on PC, I also heard it works when you force it to run in opengl). Repro steps are as before. It would be awesome if you can take a look. It's so strange. There seems almost nothing I can do my side to resolve this.

[AJuujarvi]

@AlSqPg Oh no! Do you have a seed I can test on to replicate this reliably? Or is it every generated environment?

[AlSqPg]

@AJuujarvi I think most if not all. Start new game, leave seed default, select skip cinematics. This should reproduce the issue. If you look down, the fps will go back up, but look forwards and it drops to 1.

[AJuujarvi]

@AlSqPg Awesome, thank you! Unfortunately I am out today due to illness so I will have to check tomorrow on Tuesday. I just happened to see the email update from Github and I didn't want to leave you hanging. :)

[AlSqPg]

@AJuujarvi Thanks, catch up then - hope you feel better!

[AlSqPg]

@AJuujarvi Hiya - hope all is well and that you are feeling improved. Did you get a chance to check it out? Sorry to chase, but we're launching into early access in 8 days so would super handy to understand the issue a bit more if possible. Cheers!

[AJuujarvi]

@AlSqPg Hi! I wasn't able to repro at all, and I had someone else take a look when they had the chance. They were able to repro and send me a save file. With this save file I can kinda repro, I can get it down to 15fps pretty consistently. They are writing something up for a developer over here to look at this morning in fact as the repro wasn't until on the tail end of the workday yesterday. As soon as we can get a fix out if it is feasible, I will let you know.

The only repro we got was intermittent low fps, we never got a crash on Deck.

[AlSqPg]

@AJuujarvi Thanks so much for continuing to look at this. My tester finds it reproduces every time (main branch), with default seed, on a 512gb deck. This is default proton, start new game > default seed in game > skip cinemetics. When looking up into the corner of the scene to particular asteroid area, it drops to <1 FPS.

I have a video here also: https://1drv.ms/v/s!AmJHGPq8b9FIhLxnVJe43XjQj1PWrA?e=hy6nFJ

It's beyond strange it is not consistent! And it comes and goes with builds. Hope this video assists.

[AJuujarvi]

@AlSqPg Shot in the dark, could your tester try and see if they get the low fps while the Deck is in Desktop mode?

[AlSqPg]

@AJuujarvi Looks to be the same issue in desktop mode https://1drv.ms/v/s!AmJHGPq8b9FIhLxow-KpBJyJvWPMeQ?e=EkXeyo

[AlSqPg]

@AJuujarvi Just in case it helps, a second tester with same 512gb Deck hardware, Proton Experimental currently (so the latest 8) has the same identical experience as the vid, (and to confirm this main game, not the beta or press branch, nor the playest app).

[gofman]

@AlSqPg Thanks for getting in touch with us!

We looked at the issue a bit, I could reproduce 1fps problem right after starting new game with default seed, after turning around and looking somewhere in the sun direction.

The problem on Deck is likely triggered by high video memory pressure from the game, which pushes GPU driver internals to an unfortunate operation mode when submitting GPU tasks. The reported VRAM usage after load is ~8GB, which is way too high.

I spotted that there is a number of suspiciously high resolution textures created: there are 260 2d textures of 4096x4096 size created by the end of loading into the game. To confirm the memory pressure is a real problem here I hacked texture creation in d3d11 dxvk implementation to create such textures with 2048x2048 size, and that reduced VRAM usage to ~5GB and fixed the 1fps slowdown issue here (at least at the same spot at game start). Those textures are of the following formats:

• DXGI_FORMAT_BC1_UNORM_SRGB (159 textures);
• DXGI_FORMAT_BC3_UNORM (112);
• DXGI_FORMAT_BC3_UNORM_SRGB (8)
• DXGI_FORMAT_A8_UNORM (1).

There is also some number of 2048x2048 textures.

The solution would probably be to reduce the textures size based on the rendering resolution (or some explicit texture detail setting) during texture load in game (screen resolution is just 1280x800 on Deck, and maybe even on desktop having, e. g, 1920x1080 monitor such texture resolution might be excessive?). Besides solving this specific issue with 1fps, doing so might increase overall performance and reduce the potential out of memory issues going further in game. I am not sure we have any sane way to solve or workaround this in Steam Deck GPU driver stack.

Not related to the problem being discussed, but I spotted that the game enumerates dxgi outputs each frame. While that has way lesser impact on performance, this is still rather expensive operation (even on Windows, while much more so currently in Proton). If it is possible to avoid that it would be probably good for game performance.

[AlSqPg]

@gofman Hi there - thanks so much for taking a look at this. Ok I can see that is a large amount of VRAM usage. I'm going to have a look at this and see what I can do about reducing it. Thanks again for checking it out, this information will be very useful in finding a solution.

[gbl]

Chiming in - Not sure if this is still relevant as the bug has been untouched for a while. I'm running Space Trash Scavenger on Linux - not Steam Deck, but a real Linux Desktop - newest version, of the game, and Proton Experimental (selected automatically, no manual override). After starting the game and entering a world, the process used 1.1 GB; after an hour of playing, it was at 1.2 GB. So, the problem seems fixed to me.

[AlSqPg]

Chiming in - Not sure if this is still relevant as the bug has been untouched for a while. I'm running Space Trash Scavenger on Linux - not Steam Deck, but a real Linux Desktop - newest version, of the game, and Proton Experimental (selected automatically, no manual override). After starting the game and entering a world, the process used 1.1 GB; after an hour of playing, it was at 1.2 GB. So, the problem seems fixed to me.

The issue only ever happens on Steam Deck. To avoid the problem, we have have two builds on Steam, default, and a reduced vram build that you can select in launch options for lower spec sysetems or Steam Deck users and this allows it to run.