search

actionquake / distrib

Official AQtion distribution files
GNU General Public License v2.0
25 stars 13 forks source link

Reduce differences in PKZ files between releases #289

Closed res2k closed 1 year ago

res2k commented 1 year ago

Currently, when a new update is made available on Steam, it typically contains pretty much the complete .pkz files every time, even though only a fraction of the content itself changed.

The culprits are probably the file times recorded in the archives: since git sets the file times to the checkout time, ie in practice the current time when the GH Action(s) run, this is what ends up in the archives. And since ZIP entry information is also recorded interspersed throughout the archive, this results in lots of little changes throughout the file, preventing Steam from effectively detecting "unchanged chunks". (Info on chunking taken from here: https://partner.steamgames.com/doc/sdk/uploading)

To make the file times recorded in the archive consistent between archive creation runs I created a Python script that does the archival, but specifically pulls the used timestamps from the git history.

Additionally, the files are now added sorted by name; the idea is that this avoids files moving around in the archive, as strictly speaking the current order is the one reported by the OS and may change for arbitrary, unknown reasons.

ndit-dev commented 1 year ago

Nice res2k

I don't know to well how zip archives work, so maybe I'm just mumbling here. But would it make more sense to sort the archive by modified date rather than alphabetically. That will always make sure new files end up in the end of the zip-file. A new file early in the alphabetical order would still change the position all the files after it. Or would this not make a difference?

res2k commented 1 year ago

But would it make more sense to sort the archive by modified date rather than alphabetically. That will always make sure new files end up in the end of the zip-file.

It's definitely sensible. It would handle "new" files better, and I don't think it's any worse in case existing files are modified or removed. I've added it to the script. Note that the extent to which that actually makes a difference depends on whether files are typically added, or if it happens more often that files are modified. In the latter case, sorting by time doesn't provide much of a benefit (but it shouldn't hurt either).

darkshade9 commented 1 year ago

Thanks for this, I am adjusting the base so I can test it out without affecting main 👍