search

SteamRE / DepotDownloader

Steam depot downloader utilizing the SteamKit2 library.
GNU General Public License v2.0
1.88k stars 253 forks source link

Use Steam manifest format #476

Open NicknineTheEagle opened 7 months ago

NicknineTheEagle commented 7 months ago

This migrates DepotDownloader to the original manifest format used by Steam, phasing out the pseudo-format from the old days before Steam itself was using protobuf. The output manifest files stored in download dir are compatible with Steam client and match what you can find in Steam's depotcache dir.

NicknineTheEagle commented 6 months ago

Needs SteamKit with https://github.com/SteamRE/SteamKit/pull/1337 merged since very large depots are slow to serialize without it.

xPaw commented 1 month ago

This would be a breaking change. Can we have a backwards compat for the ProtoManifest.LoadFromFile?

And can you make a separate PR for the error related commits.

NicknineTheEagle commented 1 month ago

This would be a breaking change. Can we have a backwards compat for the ProtoManifest.LoadFromFile?

Not sure how to handle it. Option A is converting from ProtoManifest to DepotManifest which would be tricky since the former doesn't have all the data from the latter. Option B is handling both of these types in tandem which would be a pure nightmare.

xPaw commented 1 month ago

the former doesn't have all the data from the latter.

What data that is needed is missing?

xPaw commented 1 month ago

@azuisleet what do you think? I currently can't update to last SK because it changed the chunk checksum from byte[] to uint, and the breaks the proto contract of the ProtoManifest.

@NicknineTheEagle can you rebase and undo the log message changes please.

NicknineTheEagle commented 1 month ago

@NicknineTheEagle can you rebase and undo the log message changes please.

Done.

What data that is needed is missing?

I don't believe anything that is needed by DepotDownloader is missing but ProtoManifest format is missing a few fields from Steam manifest, for example, filename hashes and symlinks. We can convert from ProtoManifest to DepotManifest and the result will be valid for DepotDownloader but probably not valid for other tools that work with Steam manifest format.

xPaw commented 1 month ago

We can convert from ProtoManifest to DepotManifest and the result will be valid for DepotDownloader but probably not valid for other tools that work with Steam manifest format.

We only care about backwards compat on ProtoManifest.LoadFromFile for the cached files, we don't need to save them back.

If .bin exists, load proto, then the proper .manifest, always save .manifest.

NicknineTheEagle commented 1 month ago

We can convert from ProtoManifest to DepotManifest and the result will be valid for DepotDownloader but probably not valid for other tools that work with Steam manifest format.

We only care about backwards compat on ProtoManifest.LoadFromFile for the cached files, we don't need to save them back.

If .bin exists, load proto, then the proper .manifest, always save .manifest.

I don't quite get it. If proto exists, should I load it to check if it's valid then download the proper manifest from Steam servers?

xPaw commented 1 month ago

The proto is loaded from disk to diff it against latest manifest to see which files need to be updated, no?

NicknineTheEagle commented 1 month ago

The proto is loaded from disk to diff it against latest manifest to see which files need to be updated, no?

So are you proposing to use them in tandem then? That would still require internally converting ProtoManifest to DepotManifest (just without re-saving, as you said) but that's probably manageable. What should be done if the latest manifest id matches user's current ProtoManifest? Just keep ProtoManifest file untouched?

xPaw commented 1 month ago

Keeping it as is instead of resaving sounds simpler.