Incremental Launcher Updates

[skaldarnar]

Incremental updates (or delta updates) are useful to reduce the transferred data for launcher updates.

Java Web Start supports incremental updates, but not everybody is using JavaWS. Thus, we should see whether/how we can implement incremental updates for the launcher. This becomes more important w.r.t. to bundling the JRE (see #259).

[shartte]

I've been looking around for auto update solutions that provide delta updates and so far I've found this: https://github.com/Mendeley/Update-Installer

Seems to be Linux, MacOS X and Windows and supports the method I'd prefer (client side delta updates).

[msteiger]

This is interesting and might be even more relevant for game updates, actually. Splitting the game into more and smaller libs and modules might make it easier to update it without any external update scripts.

[shartte]

The external auto update solution is necessary in this case because we'd still need a way to self-update (we cannot overwrite the running java DLLs / exe files while they are being used).

There's a way around that by copying the JRE and required updater jars to a "scratch directory" which is then used to apply downloaded updates. For updating the game itself, we will not need such a "hack" because the launcher can update the game while it's not running.

[skaldarnar]

I found this delta-updater and tested it with two stable versions of the launcher (2.0.2/#15 and 3.0.0/#17). It was no problem to create the patch and apply it with the command line tools.

I think it can be used for both launcher and game. When updating the launcher the whole application would have to be copied to a "scratch directory" on which the patch could be applied, like @shartte described above.

To start off, we can generate a patch file with each build w.r.t. to the last successful build. The client would be responsible to "collect" all patches from its current version to latest one and apply them in the correct order. Later we could have some servlet (close to where the build results are stored) that takes the old version number and target version number and generates the delta patch on the fly. If we cache some of these requested patches (e.g., latest one and/or most requested) it would yield faster updates.

What do you think @msteiger , @shartte ?

PS: there is also xdeltaencoder although it seems that this tool needs more work/preprocessing (SO question)

[shartte]

If you can, I'd try to find an out of the box solution. But there really didn't seem to be much that fits the cross-platform bill to be honest.

Regarding true incremental delta updates: You should also calculate the overall size of all delta updates a client needs to download and switch to the full download if the sum of the delta updates actually exceeds the size of the full download (quite possible for nightly builds or similar).

Which version of the game is your idea for btw? For the "real" release builds? In what interval do these come out?

For the nightly build it might actually be easier to just do individual file updates without real deltas.

[skaldarnar]

Regarding true incremental delta updates: You should also calculate the overall size of all delta updates a client needs to download and switch to the full download if the sum of the delta updates actually exceeds the size of the full download (quite possible for nightly builds or similar).

That's the main reason why I would like to have the server build the patches on demand so that we do not need a chain of patch files. I did not test the patch size for the launcher with bundled JRE yet, wanted to do that today.

Which version of the game is your idea for btw? For the "real" release builds? In what interval do these come out?

I talked with @Cervator about this yesterday. It might be a good idea to always download full major releases, and only use the patches for minor/patch updates. I might test the delta patch size for two game versions as well.

Anyways, I wanted to start with the launcher doing incremental updates - if that's in place we can bundle the JRE a have a self-contained app.

[shartte]

That's the main reason why I would like to have the server build the patches on demand so that we do not need a chain of patch files. I did not test the patch size for the launcher with bundled JRE yet, wanted to do that today.

I thought about that and that really opens up a lot of denial of service potential, heh. In addition, you cannot distribute patches using a purely static web-server (think CDN) that way. My initial idea for the patching was that the JRE should be put into a separate package, since it rarely updates.

(Side-Rant: I am not really hot on using Java for desktop applications anymore since packaging them is such an incredible headache, mainly due to their inane licensing requirements. .NET Core here I come!)

[skaldarnar]

I thought about that and that really opens up a lot of denial of service potential, heh.

Good point. Let's try to get the patches between consecutive versions going then.

[skaldarnar]

artifact	old	new	patch size	ratio*	info
Game (RELEASE)	61	62	27mb	3.8	consecutive
Game (RELEASE)	59	62	27mb	3.8
Game (NIGHTLY)	1623	1624	116kb	905	consecutive
Game (NIGHTLY)	1612	1624	1.2mb	87.5
Launcher (RELEASE)	15	17	104kb	39.4	consecutive

* Tested against the unzipped size of the download. Actual ratios are worse, e..g, 26.9 for the launcher.

Stable releases contain a lot of changes, including changed libraries/dependencies and many code changes. Therefore, the patch ration is quite bad.

Nightly releases often consist of single commits or small patches, therefore the patch ration is very good.

[Cervator]

Hi @shartte good to see you still alive :-)

What's with the license requirements for Java desktop stuff? Doesn't ring a bell for me but I'm curious. If the delta patcher itself is Java hopefully that should be cross-platform well enough?

Neat numbers, @skaldarnar !

And yeah my long-term concerns often relate to traffic issues and how we might survive them without having money to spend on anything serious.

[shartte]

What's with the license requirements for Java desktop stuff? You can't strip the Oracle JRE down any further than they'll allow or you lose the license to redistribute it. I guess one could try using OpenJRE builds for redistribution, since those could be stripped down way further without hitting their redistribution license issues.

[Cervator]

Ahh okay. I wouldn't worry too much about that then, especially with Java 9 coming with its modular thing.

I wonder if we're violating something with the Dest Sol bundled JRE. Even if we are I suspect we'll be on a nice modular JRE 9 before it gets noticed, if ever :-)

[shartte]

Are you using the bundled JRE i built? I actually tried to only remove files that were explicitly mentioned as optional in their license.

Oh and Java 9? Think 2017 :P

[Cervator]

DestSol already came with a bundled JRE when we adopted it, I don't remember any details and haven't checked on what parts are there or not. We added a Linux JRE to it at some point, but I'd prefer to eventually use our nice structured approach there as well :-)

Yup on 2017 and it isn't like we're going to get noticed much before or likely even after that :D

[skaldarnar]

Investigate google/archive-patcher

[skaldarnar]

https://github.com/MovingBlocks/TerasologyLauncher/pull/509 removes the createPatch task as it was not used for some time now. Maybe it contains some bits and pieces that can be of help later.