Feature Idea: Deduped Steam Libraries

duskmoss commented 7 months ago

It seems likely to cause problems to use the same steam library folder for multiple clients, at least at the same time.

So something that could be useful is running dedupe remove against the container folders for various clients. If mounted on BTRFS this could save a few gigabytes in the base image and many more in game downloads.

It would also be cool if there could be a main Steam container that installs all the games and is running when any steam containers for clients are open so a new client can install by steams feature to transfer from it over the "local-network". Which in this case can be a really quick bridge between containers.

Hard part on the main Steam container part might be identifying what a client steam container has installed to then installing on the "master" container. Maybe we can just look at "steamapps" grab the ids from the list of appmanifest_.acf files and then send those to steamcmd? But I'm not sure if a headless steam manage steamcmd implements the local network transfer stuff. Perhaps with a full fledged steam client we can just copy the files in? but I'm not sure if we'd need to trigger it to recognize them and offer them to other clients somehow.

ABeltramo commented 7 months ago

Not sure if this is something that we could add to Wolf or it just needs to be properly setup on the host..

I'm not sure if a headless steam manage steamcmd implements the local network transfer stuff. Perhaps with a full fledged steam client we can just copy the files in?

The steam image runs the full client, so everything should be available as is

duskmoss commented 7 months ago

It could be something that Wolf doesn't do, but its a lot of state that specifically is useful to wolf, and seems to have advantages to be tied in to its lifecycle. Integrating some sort of plan here to avoid wasted diskspace and redownloading games for each streaming device will also make users less likely to just throw a single drive in mounts and put steam library on there. But maybe that would just work well enough, it feels like it's asking for race conditions messing everything up in a multi user environment.

So for the goal of 1) reducing disk usage 2) reducing need to redownload:

1) Something needs to run "duperemove" after certain periods.

This part of the feature is fairly easy to do outside, just have a cron job that dedupes the wolf folder once a day. However it makes a lot of sense to me if if Wolf schedules it when a container shuts down, to dedupe files added during that session.

2) This one is a lot tricker to set up, and is more software than just configuration. Just running a steam client on the host doesn't do anything unless all games which will be installed by clients are pre installed there. And there's no always running steam for hosts which are fully headless. The idea here would be to have a steam container that isn't tied to a client, and some plumbing so that after clients install games in their steam containers, it installs them so that a different client won't have to download from the net again.

| The steam image runs the full client, so everything should be available as is

Right, but the steamcmd comment wasn't about a image spun up for a client. Those do already transfer from local network, and if you have Steam running on the host they will transfer from there, though not as fast as could be done between containers on a shared internal bridge.

Building on that, if wolf can have with a container that's always on, and can be scripted to install games, it could be seamless that people don't have to download games multiple times if multiple moonlight devices play the same game on the wolf server. Another thing that could potentially be scripted externally. But here there is a risk partly installed games being transferred depending on implementation Wolf doing it when a client shuts down would make it somewhat atomic.

SteamCMD is easier to script which is why I wonder if it can do the local network transfers. Another image like is spin up for clients would for sure work to serve games to the others, but I'm not sure sosohow to script it installing games. And wolf probably would need a way to connect to it's display to administer it (set it to allow all users on the local network to transfer games at least)

Been thinking about a potentially easier path for 2) even if it does have some disadvantages:

Another approach could be adding a LanCache container to wolfs network. I stopped running one at home because I had flakiness with steam deciding to bypass the container, but I hope it would be more reliable in the virtual network.Another downside is that update churn can cause extra disk space spent on no longer relevant files. It also wouldn't dedupe as cleanly as a "main" steam container approach because it stores compressed chunks and not the specific files. On the plus side it wouldn't need any sort of system to recognize what clients installed and install them on a master container. It would be a lot less moving parts, just some docker network design.

There would still be advantages to having LanCache run by wolf and not set up externally. Mostly so it can be part of an internal bridge, because networking between containers is much faster that way than on the default bridge, DNS is handled better, and wolf can ensure DNS is set on the steam containers to use LanCache without a user having to mess with their local networks settings.

ABeltramo commented 7 months ago

The more I read about this the more I think this is getting outside the scope of Wolf but it's a very interesting discussion and you brought up a lot of good points, there's lots to digest so here are a few random counter points:

RE duperemove: I don't like having to implement filesystem specific code (nor forcing such a personal choice to users), this should be scripted outside Wolf. We could easily add a pre/post session script hook to facilitate that.
The solution of having a Steam "main" account has the main drawback that you need an account that owns all the possible games that the "secondary" accounts will need, AFAIU SteamCMD needs a valid login (that possibly bought the game already) to download/update games.
- This really feels like a hack and easily error prone, having to orchestrate all that seems like asking for troubles..
LanCache + Deduplication (btrfs/zfs/any other fs that supports this) seems to really be the right solution for this. With LanCache you download only once, on-demand and most importantly transparently to all clients (and not just Steam); with deduplication you avoid any wasted disk space and just share the same blocks between clients.
- This should be setup outside Wolf, based on user preference and actual requirements; there's no one-size fits all solution for this.
- As for the networking and internal bridges concerns this should be already possible by overriding the default docker params in Wolf config.toml (you can manually override NetworkingConfig, NetworkMode and so on already in there).
- Besides, this is something that different users will want to tweak (ex: a LanCache somewhere in the network that might be shared with other machines/desktops or a local one in the same machine that runs Wolf)

I'd be very interested in digging further into LanCache + Deduplication even if it's just going to end up being documentation and not actual code. I always wanted to checkout LanCache ony my homelab this might be the right time to actually do it!

duskmoss commented 7 months ago

Thanks for reading and engaging!

I don't like having to implement filesystem specific code (nor forcing such a personal choice to users), duperemove, isn't filesystem specific it's a external utility that uses linux ioctl s that some filesystems (btrfs, xfs, in progress with zfs) support. Simply exits if the files it's pointed at don't support the ioctl.

I do like the idea of hooks and they will be useful for other stuff. I'm still tempted to say it should be something on by default for people but I can live with it not being :)

You need an account that owns all the possible games that the "secondary" accounts

mm yeah. I suppose mostly assuming it was different client devices but same steam account, but this would be a hard part. Hmm.

I'd be very interested in digging further into LanCache + Deduplication even if it's just going to end up being documentation and not actual code.

I think with Lancache even if it's just publishing a docker compose example of it would massively help people out. Running it on a different machine will be much slower without multi gig networking.

As for the networking and internal bridges concerns this should be already possible by overriding the default docker params in Wolf config.toml (you can manually override NetworkingConfig, NetworkMode and so on already in there).

I think something may still need to be done internally for dns, to make it easy to keep the traffic to the internal docker network if running one there.

I might still explore scripting the main steam container some, because I don't trust lancache to work reliably :) If it ends up feeling easy I might have you take a look and see if you want to integrate. Or just figure out how to interface it with the hooks you add.

duskmoss commented 7 months ago

I'm realizing the thing I'd actually more want to hack on is probably something to make reusing containers easier. I don't particularly need or want each device I connect from to have its own set of containers.

Perhaps a user based collection. Or manual selection. Any thoughts for me on forking, vs preparing a PR or pointers on where I might want to start in the code?

ABeltramo commented 7 months ago

I'm realizing the thing I'd actually more want to hack on is probably something to make reusing containers easier. I don't particularly need or want each device I connect from to have its own set of containers.

This is a valid point and I've been thinking about it for quite some time, I believe this is part of a more general feature: share sessions (aka: co-op for multiple users or as a specific case for a single user share the container between devices).

I think there are different possible approaches to this:

Change Wolf so that it supports a notion of users and will "assign" containers to them. This will practically have to be manually configured in the config.toml or, possibly, configured at runtime over a webui
Make a GUI that when you start a connection with the Moonlight clients will show up the list of connected/possible users (like the Netflix first page) and then let you pick the games for that user and ultimately starts and controls the app container.

Whilst the second approach might be a bit more complex it feels that it's the best way of achieving this:

A completely independent game and user launcher so that Wolf can be focused just on the Moonlight protocol and streaming
From the user perspective you don't have to open up a browser to set things up (some Moonlight clients aren't even able to do that: xbox, switch, ...) but you have a nice and easy interface that you can interact with your joypad
Wolf could send messages over this GUI, ex: when first downloading a game it could show the progress bar instead of a black page like it is right now

Thoughts?

vinibutturi commented 6 months ago

A completely independent game and user launcher so that Wolf can be focused just on the Moonlight protocol and streaming

That would be really cool 🤩

If I may add my 2 cents, this is how I see it:

Instead of having all the games showing up on moonlight, we just got one game/app which will be a launcher.
- From the launcher we could start other apps/games, manage/select users, and join sessions with another player.
- The launcher would also be responsible for calling any pre/post script we might want to hook to the apps/games

is that similar to what you're thinking or am I dreaming too much?

ABeltramo commented 6 months ago

A completely independent game and user launcher so that Wolf can be focused just on the Moonlight protocol and streaming

That would be really cool 🤩

If I may add my 2 cents, this is how I see it:
* Instead of having all the games showing up on moonlight, we just got one game/app which will be a launcher.

* From the launcher we could start other apps/games, manage/select users, and join sessions with another player.

* The launcher would also be responsible for calling any pre/post script we might want to hook to the apps/games
is that similar to what you're thinking or am I dreaming too much?

That's exactly what I have in mind, it's not a simple task, but it's definitely doable! Anyone wants to help out? 😅

vinibutturi commented 6 months ago

That's exactly what I have in mind, it's not a simple task, but it's definitely doable! Anyone wants to help out? 😅

I would love to help, but I'm gonna need some guidance on all the tech involved. I got 10+ years experience with ERP development (X++ language) so I'm not starting from scratch, but this is completely new game to me.

ABeltramo commented 6 months ago

I would love to help, but I'm gonna need some guidance on all the tech involved. I got 10+ years experience with ERP development (X++ language) so I'm not starting from scratch, but this is completely new game to me.

The good news is that this will likely be a separate project from Wolf, so it can be done in any language (probably not C++).
It could even be just a simple GUI program that will run fullscreen, and that will communicate back with Wolf when a selection of user+app has been made.

I have to define the requirements a bit better..

vinibutturi commented 6 months ago

I would love to help, but I'm gonna need some guidance on all the tech involved. I got 10+ years experience with ERP development (X++ language) so I'm not starting from scratch, but this is completely new game to me.

The good news is that this will likely be a separate project from Wolf, so it can be done in any language (probably not C++). It could even be just a simple GUI program that will run fullscreen, and that will communicate back with Wolf when a selection of user+app has been made.

I have to define the requirements a bit better..

yeah, keep it as a separated project is the way to go. What do you think about moving this conversation to discord? More convenient and we can brainstorm it further :)

rty813 commented 5 months ago

My method is to mount the .steam directory. If it's just for personal use, you can use this solution.

a-priestley commented 5 months ago

my toml block for steam:

[[apps]]
title = "Steam"
start_virtual_compositor = true

[apps.runner]
type = "docker"
name = "WolfSteam"
image = "ghcr.io/games-on-whales/steam:edge"
mounts = [
    "/mnt/games:/mnt/games",
    """
/home/apriestley/.steam/steam/userdata/19414271/config/shortcuts.vdf:/home/retro/.steam/steam/userdata/1941427\
1/config/shortcuts.vdf\
""",
    "/home/apriestley/.steam/steam/compatibilitytools.d:/home/retro/.steam/steam/compatibilitytools.d",
    """
/home/apriestley/.steam/steam/config/libraryfolders.vdf:/home/retro/.steam/steam/steamapps/libraryfolders.vdf\
""",
]
env = [
    "PROTON_LOG=1",
    "RUN_GAMESCOPE=true",
    "GOW_REQUIRED_DEVICES=/dev/input/* /dev/dri/* /dev/nvidia*",
]
devices = []
ports = []
base_create_json = """
{
  "HostConfig": {
    "IpcMode": "host",
    "CapAdd": ["SYS_ADMIN", "SYS_NICE", "SYS_PTRACE", "NET_RAW", "MKNOD", "NET_ADMIN"],
    "SecurityOpt": ["seccomp=unconfined", "apparmor=unconfined"],
    "Ulimits": [{"Name":"nofile", "Hard":10240, "Soft":10240}],
    "Privileged": false,
    "DeviceCgroupRules": ["c 13:* rmw"]
  }
}
\
"""

The relevant bits are in the mount directive. /mnt/games contains my libraries for every platform (or no platform). The entry with shortcuts.vdf contains configs specifying any non-steam games I've added in steam on the host machine (your steam user ID will be different), and the entry with libraryfolders.vdf is the config for specifying extra library directories, such as when you might keep your titles installed on a separate storage drive like myself. The mount for the library itself is mapped one-to-one - it's identical on the host and client, as that's how the vdf files are configured. As an extra, I have an entry there for the compatabilitytools.d directory, so that I can pass in custom tools from the host, such as ge-proton or steamtinkerlaunch.

This isn't deduplication, but it works well.

EDIT:

I also have written this guide for setting up proper block-level deduplication using bees on top of btrfs, exposing storage shares over iSCSI. For this setup, the host and client are on the same machine. Potentially, each separate wolf user can own a discrete backstore pointed at the same disk image. It's been a while since I've experimented with this, but it came with its fair share of issues (kernel I/O faults in my case), which may or may not have been due to some faulty memory I had installed in my machine at the time. I'm betting that there are improvements to be made.

games-on-whales / wolf

Feature Idea: Deduped Steam Libraries #69