Switch workshop to ISteamUGC

[neico]

To quote the implementation changelog:

Added a new Workshop item content API in ISteamUGC that is easy to use, allows multiple files per item without any size limits. It uses the same content system that handles regular content depots, resulting in faster and smaller downloads due to delta patching. Subscribed workshop items will be placed in unique subfolders in the install folder, so the game doesn't need to fetch them using ISteamRemoteStorage anymore. The new API is not backwards compatible with old items created with ISteamRemoteStorage.

About the last sentence, no you don't need to re-upload all those workshop files, to quote the docs:

Q: Is it possible to update an item published using ISteamRemoteStorage with ISteamUGC? Yes, however once updated to ISteamUGC, the ISteamRemoteStorage API will no longer work with that workshop item.

You could either keep some backwards compatible code on the first release with ISteamUGC or do a automated update of all items once using a tool.

As you can read from the changelog, it eliminates the 200MB limit on the workshop files, reducing the amount of workshop files to download (take the stargate (CAP) or GMod Tower for example), which in turn reduces the amount of search paths gmod needs to look trough upon loading, reducing loading times and increasing overall performance (depending on the situation but for most cases).

For those with steamworks access: https://partner.steamgames.com/documentation/ugc

Quote from steamworks page:

Future ISteamUGC Direction

While the fundamentals of ISteamUGC are now implemented, the team is continuing to expand the functionality of the API. Potential future work includes:

• Manual override of the order items are downloaded in to avoid large downloads that could block a game starting
• Simplify the process for dedicated servers interacting with workshop items
• Adding version information in the API for a workshop item
• Selectively download individual files from uploaded workshop items

[Jvs34]

:rainbow: also apparently workshop now supports category filters, so it'd be nicer to see the saves crap filtered to one of them instead of cluttering everything up. http://bit.ly/15TDuiR

Also to be fair garry kind of implemented server downloading bad, so maybe a full rewrite would be neat.

[neico]

I think all server owners can agree with that last part @Jvs34. (it's probably one of the most annoying things for server owners since the workshop got added) But valve plans to cover that in the future at least, quote from the ISteamUGC docs:

Simplify the process for dedicated servers interacting with workshop items

Maybe someone should go hunting for issue and requests and reference to them here to give this some more significance, and to make it easier to close the correct issue's once it's implemented.

I guess #327 could be one because of the Subscribed workshop items will be placed in unique subfolders in the install folder part

182 is partly addressed due to delta updates

[MattJeanes]

Something that may be worth noting (from the steamworks page):

[robotboy655]

But we never remove any workshop items from the API? It can only be done from the mod page by its author/uploader.

[MattJeanes]

Yes I'm aware but it means that people can't delete their own addons, which would be a missing feature if we upgraded to the new API. Apart from that though I'm all for it, and I'm sure they'll add that ability in time.

edit: unless I misunderstood and users are still able to delete their own content, just not from the API

[GGG-KILLER]

Any updates on this?

[neico]

Cross-reference to indicate progress on this request:

https://twitter.com/FacepunchBot/status/794140301745070080

[JustinCase224]

could this allow for downloading and applying items in-game without having to restart

[robotboy655]

So here's the progress on this:

• Dedicated servers and clientside subscriptions are already on this new system for a while
• In this next update, "WorkshopDL" - Downloading of addons when joining a server will also be on this new system
• In this new update srcds can also mount ISteamUGC addons (previously it only used the API, but didn't mount the new system addons, which don't exist on workshop anyway yet)

You can already upload ISteamUGC workshop items to the workshop and they should work properly. (via SteamCMD) Keep in mind old API will not be able to interact with these new addons. GMPublish will be updated later, not in this update as I want to make sure it all works before forcing everyone to convert their addons to the new system, which is one way.

After the update I will probably update one or two of my addons to use the new system to test it.

Here are the benefits of addons uploaded with SteamUGC API:

• The filesize shown on the Workshop Item page will properly reflect the actual addon file size.
• It will take less space on disk (currently Steam always keeps the compressed file downloaded + We have the uncompressed file in GMod's folders)
• There is no file size limit on uploading, meaning more strict moderating on multi part addons as well
• You can use SteamCMD to upload addons

With all of that, here's the plans for the (hopefully near) future:

• Move game,MountGMA to this new system too (remove steamworks.Download and make an alternative)
• Add new tags and better validate them when game mounts content. Things like hasLua if the addon has Lua files, and not mounting the addon if it has lua folder but not hasLua tag. Also possibly disallowing Lua in ServerContent addons.
• If I can make it happen, once we make the switch completely, game will not mount newly uploaded addons using the old API at all to encourage everyone to use the new, better format. This means any old uploading tools that aren't a wrapper for gmpublish.exe will cease to work.
• Moving the "legacy" workshop addon files into a single folder, away from addons/, etc. The SteamUGC uploaded addons will never appear anywhere in the GMod's folder.

[WinterPhoenix]

Quick question, what do you mean by ISteamUGC uploaded addons not appearing in GMod's folder?

Would this mean that no matter how you get it, whether it's by Subscribing, resource.AddWorkshop, steamworks.Download, etc, they'd all go to the same place, somewhere handled by Steam?

[robotboy655]

Yes, which also means Steam will delete the addon files automatically (those that you aren't subscribed to) when they haven't been used for over 2 weeks, or at least that's what we've been told.

steamworks.Download haven't been converted to the new system yet.

[WinterPhoenix]

That sounds amazing. Okay, then here's another question. Would the following be possible with the new system?

1. Players join the server, download core required content addons to be able to play using resource.AddWorkshop
2. After spawn, use steamworks.Download and game.MountGMA to download and mount the rest of the content in the background
3. If the player rejoins, resource.AddWorkshop correctly detects that we've used steamworks.Download to download the content already, and doesn't redownload separate versions
4. Subscribing to the Workshop Addons skips all of this, but correctly detects anything already downloaded by either resource.AddWorkshop or steamworks.Download

In the current system, the trouble is mainly that all 3 of these ways to download Workshop addons go to separate folders, so if you happen to use all 3, surprise, you've got 3 copies of the same flippin' GMA. And none of the methods can tell that another method already downloaded them. Except for resource.AddWorkshop detecting that which you're already subscribed to.

Case in point: Subscribing: garrysmod/addons/ resource.AddWorkshop: garrysmod/downloads/server/ steamworks.Download: garrysmod/cache/workshop/

The point for all of this is to get players into the server as fast as possible so they can start playing. Nobody likes sitting on a loading screen forever waiting for all the content to download all at once.

[robotboy655]

For 3. no, it won't detect anything, but it won't redownload anything either, unless you add new stuff.

And no, it doesn't currently download 3 copies of the addons, (steamworks.Download probably does though due to how it works), but yeah a fairly big benefit of this new system is that the files are all handled for us, so no wasted space. It will work best of all existing addons update to the new format after we completely switch to the new system so we don't have to keep the compressed versions of all the addons.

[WinterPhoenix]

Could the future steamworks.Download be changed in some way to make a system like this possible? Where we could realistically and usefully download content in the background, while they're playing on the server, that can be used by the other methods?

It would be extremely useful in my eyes to have that flexibility with downloading content, so we don't have to tell players "go subscribe and wait a while" or "wait on this loading screen for a decade while we download everything all at once."

Instead, they could just download while they play

[robotboy655]

But you can already do it?

[WinterPhoenix]

At its core, the problem right now is that steamworks.Download is useless from the eyes of resource.AddWorkshop. If we wanted to do it right now, we'd have to use steamworks.Download and game.MountGMA from the cache, each time.

And this completely breaks down if, say, you're trying to download content (specifically maps) for other servers that you can connect to from the current server.

Because if resource.AddWorkshop doesn't detect the stuff downloaded by steamworks.Download, you're redownloading that content, completely defeating the point of downloading in the background.

And you can't use steamworks.Download + game.MountGMA in a server you can't even join, so.

[robotboy655]

As i said, nothing is ever redownloaded (Unless there is an update of course). If the addon is already downloaded, it will not be downloaded again, even if you call steamworks.Download, or if you are joining a server with an addon resource.AddWorkshop'd that was previously downloaded via steamworks.Download

[neico]

Note: the following a steam implementation detail, this may change anytime steam downloads workshop items to the workshop folder under each library location's steamapps folder, there it's categorized per-appid and one folder for each workshop item (as ISteamUGC workshop items are a folder instead of a single file now) That means that no workshop item ever exists more than once.

Now to the difference:

• a subscribed item stays there for as long as it's subscribed to, which makes it query-able by the game
• to get a workshop item you're not subscribed to and don't want to force on your users requires the use of ISteamUGC::DownloadItem() which to quote the steamworks docs: If the user is not subscribed to the item (e.g. a Game Server using anonymous login), the workshop item will be downloaded and cached temporarily., which is what rubat mentioned earlier about it being cached around 2 weeks (again, implementation detail)

My guess here is that resource.AddWorkshop will make use of said DownloadItem function to handle it, that function uses an asynchronous callback tough, so I'd expect it to be a bit of a hassle to add to the engine's serial loading mechanism so it waits for them to complete downloading before continuing, unless it's made a optional background download (which could be handled by adding an optional bool arg to make it a required workshop item, this could also be emphasized by using DownloadItem's second argument to mark it as a high-priority download, something that can be easily sent over the download stringtable by setting the bool inside the string entry's userdata)

Speaking of which, I expect AddonMaterial to also change slightly internally as you can just query for the preview image URL from steam directly using ISteamUGC.

What I wonder tough, are ISteamUGC addons still using the .gma files? Pretty much all the metadata from that file can be put either inside key-value entries (visible from the website) or a binary MetaData chunk. Using raw files instead of an archive will also make steam's delta patching work better on them. (you could still opt to use steam-friendly chunk'd VPK files then for the very bloated ones) At this point renaming game.MountGMA to game.MountAddon or game.MountWorkshop (function naming seems rather inconsistent) would be needed too. Not to mention that it would break compatibility at a good time, because you'll need to adjust your code for the new behavior anyway.

[robotboy655]

Here's the progress since last progress post:

• Dupes/Saves/Demos were completely moved to ISteamUGC and now support updating. Publishing UI also got a refresh.
• If you are using game.MountGMA you will need to switch to steamworks.DownloadUGC which takes a workshopID, not a fileID. steamworks.Download is deprecated as it cannot access ISteamUGC files. The only reason I am leaving it available is because there's no other way to download workshop icons.
• gmpublish.exe was also moved to the new API (on the Dev branch only until the update).

As of right now, Dev branch is basically completely switched to ISteamUGC.

Things I still want to look into:

• If I can make it happen, once we make the switch completely, game will not mount newly uploaded addons using the old API at all to encourage everyone to use the new, better format. This means any old uploading tools that aren't a wrapper for gmpublish.exe will cease to work.
• Moving the "legacy" workshop subscriptions from addons/ into cache/. This would require subscription list caching though for offline play.
• Add new tags and better validate them when game mounts content. Things like hasLua if the addon has Lua files, and not mounting the addon if it has lua folder but not hasLua tag. Also possibly disallowing Lua in ServerContent addons.
• Potentially scan cache/workshop/ folder monthly or something and delete files that aren't in use for a certain period. This should be already automatically done for you for ISteamUGC workshop content.
• Maybe some UI tweaks for the addon menu - for example subscribing to addons with dependencies showing those dependencies in-game.