An update zip is manually created by the server owner and placed on any web server. This is typically done by extracting the new servo binaries over the previous update (with a few file exceptions)
The server configuration is manually changed to point to the new update zip.
The server is stopped via the Admin API or an automated restart, and then the update files are manually copied into the server.
Usually, version.txt has to manually be modified on both the update zip and the server to match the server's versioning structure.
The key word you're looking for here is "manually". There are many chances for this process to go wrong (and it has) and a lot of time is typically spent preparing these updates. Other issues include the increasing size of our updates, and the increasing chance that something will either be corrupted or not extract. There are no safeguards when this happens, and the user often has to reinstall the FreeSO client, which the launcher doesn't make too easy.
The solution is to rethink how we do updates entirely. The less files we have to update, the less intrusive it will be to push out corrective updates for both us and our users, especially for minor updates that just change objects or PIFF patches. Catching things that go wrong and re-updating when they do will save a lot of hassle for users that are affected often.
Automated update generation
Updates are based off of the latest master on http://servo.freeso.org . This may be changed to Microsoft Azure in the future.
Development should be done on develop or feature branches, to avoid affecting game updates.
The server also keeps a store of "addons", files that should be added to the binary to make the final version. (eg. server specific objects, plugins) These should be uploadable to the server using the manage api, and named.
A new update should be created from the Admin API. This should include parameters for the servo build number (default: latest master) and addon pack (default: latest uploaded) to use.
Options: db-init on restart, force non-incremental, force incremental on protected (discord-rpc) etc.
The server should build the update by extracting the servo build into a new folder, then extracting the addons over it.
Catalog xml should either have another method for replacement/management or move to being served upon login instead of via updates.
Version number should be generated and put into version.txt for both client and server.
May be interested in maintaining multiple update and feature branches within the same database.
Incremental Updates
Update binaries are getting larger and larger, as they contain all of the files used by the FreeSO client. This includes our additional objects, 3D city, ui etc... The thing is, most of these files do not change between game versions, and most likely never will. If we discover a bug with a single library or object, it doesn't make sense pushing out an update with 128MB of content to fix it.
The solution is to instead only update the files that have changed from the last patch. With the above in mind, that means the following:
When generating and saving update distributions, all files in the distribution are compared with those in the last, and we note which files we added or modified. (deleted may also be tracked)
These files which were changed are put into their own Incremental Update distribution, which contains the difference between this version and the last.
A manifest is stored which contains hashes for all of the files which will be changed. If any are different, then something may be wrong and we can recommend the user do a full update instead. A manifest should also be used for the full distro, though that can't check what was there before.
If a user is a few updates behind, they can simply run back to back incremental updates to get to the latest version. If they are many updates behind (file size exceeds full update), have an invalid game state, or are installing for the first time, the game will download the full update instead.
Note that this means a few things for servo - specifically that for the most optimal calculation of differences we will need to make sure the buildbot does not build clean every time. This will mean it could leave multiple dlls unmodified, meaning they do not need to be included in the incremental update.
New update installation (server)
Revise the watchdog application to support the new update system. (also use .net core like the server)
Decision: generate zip or just in-place patch? I think generating a zip and then applying it will help us figure out if something goes wrong.
The usual exemptions for important files, like NLog.config, scripts and other config.
Should be possible to schedule a server update to occur automatically on next restart.
New update delivery
Updates should be uploaded to a service outside the game server that can scale appropriately. The initial goal for this is Amazon S3, but it should be possible to extend the service to add targets like generic FTP servers.
An API endpoint for getting all recent versions of the client that are available. For each, we need a link to their full update, incremental update, manifest and obviously name. The list should be ordered by time descending, leaving the newest first and letting us work up from the version we are on to the latest.
Should still be compatible with old update method (serving latest FULL update) so people aren't stuck on old versions of the game. Incremental portion and metadata will be added on top.
New update installation
May still want to download inside the game... on osx and linux leaving the game means not displaying a GUI at all, so we couldn't show download progress to the user.
Launch updater from game: find a way to do this without making antivirus freak out so much, and while also releasing all of freeso's dlls.
Launch game via updater: if the game cannot launch for some reason, the user should be able to use update.exe to redownload and install the last attempted update (standalone, from the game's server info ini). The game could attempt to open the updater on failed launch, or there could be a shortcut in the freeso launcher to do this.
Updater self update: right now it renames to update2.exe and the game replaces the original on start. May want to find a cleaner way to do this.
The updater must correctly handle failures when writing files. It is best to complete the update as fully as we can (after at least FreeSO.exe is free), then list any errors afterwards. It should be possible to retry or ignore these errors, and info should be provided to the user about how to re-run the updater if they need to restart their pc to free them.
The updater must also verify that the files are correct via the hashes in manifest.
This issue is still WIP, but it is the current focus of development. As a result the Admin API angular app will probably be cleaned up too.
The current update path is very primitive:
The key word you're looking for here is "manually". There are many chances for this process to go wrong (and it has) and a lot of time is typically spent preparing these updates. Other issues include the increasing size of our updates, and the increasing chance that something will either be corrupted or not extract. There are no safeguards when this happens, and the user often has to reinstall the FreeSO client, which the launcher doesn't make too easy.
The solution is to rethink how we do updates entirely. The less files we have to update, the less intrusive it will be to push out corrective updates for both us and our users, especially for minor updates that just change objects or PIFF patches. Catching things that go wrong and re-updating when they do will save a lot of hassle for users that are affected often.
Automated update generation
developor feature branches, to avoid affecting game updates.Incremental Updates
Update binaries are getting larger and larger, as they contain all of the files used by the FreeSO client. This includes our additional objects, 3D city, ui etc... The thing is, most of these files do not change between game versions, and most likely never will. If we discover a bug with a single library or object, it doesn't make sense pushing out an update with 128MB of content to fix it.
The solution is to instead only update the files that have changed from the last patch. With the above in mind, that means the following:
Note that this means a few things for servo - specifically that for the most optimal calculation of differences we will need to make sure the buildbot does not build clean every time. This will mean it could leave multiple dlls unmodified, meaning they do not need to be included in the incremental update.
New update installation (server)
New update delivery
New update installation
This issue is still WIP, but it is the current focus of development. As a result the Admin API angular app will probably be cleaned up too.