Future updater (club.exe) and update process work.

[skittles1]

Improved in-client update process by @cyberjunk.

Current process:

• Outdated client (by version number) connects to server.
• Client receives download info (host machine, patch directories, patchinfo.txt location).
• Client downloads patchinfo.txt.
• Client loads patchinfo.txt, checks local club.exe hash vs. patchinfo.txt's club.exe hash.
• Client potentially downloads club.exe.
• Client launches club.exe with parameters (download info, exe (self) name, patchinfo.txt local path).
• Club loads patchinfo.txt and downloads any file that doesn't match the local version.
• Club runs the exe file given in parameters (the client that launched it).

  Proposed process:

• Outdated client (by version number) connects to server.
• Client receives download info (host machine, patch directories, patchinfo.txt location, club.exe location).
• Client saves this data to a local file (downloadinfo.txt or similar) with name of self (exe).
• Client downloads club.exe.
• Client launches club.exe with location of the local downloadinfo.txt.
• Club uses downloadinfo.txt to download patchinfo.txt.
• Club loads patchinfo.txt and downloads any file that doesn't match the local version.
• Club runs the exe file listed in downloadinfo.txt (the client that launched it).

  Pros

• Club could restart a failed download using the existing downloadinfo.txt.
• Less work for the client, majority of update process contained in club.exe.
• Remove JSON lib dependency for clients.
• The new file (downloadinfo.txt) could be kept and used for manual updates by running club.exe.

  Cons

• If contents of downloadinfo.txt become invalid (e.g. webserver folder locations change) users could try to use it for manual updates (which would fail). This might not be a significant concern as connecting to server with an outdated client would fix it. The fail scenario could require both outdated downloadinfo.txt contents and a broken local client (unlikely). The Patcher is also available to fix clients in the event that this happens so only the subset of users who are unable to run it would potentially be affected.
• Club.exe always has to be downloaded (~500kb, not really significant).
• [x] Convert clientpatch/updater to create whole-file hashes, and remove filename from hash process.
• [x] Add new server protocol to send patchhost, patchpath, patchcachepath, patchcachefilename, updaterfilename, running alongside old clientpatch protocol (there's plenty of room here).
• [x] Move patchinfo.txt downloading from classic client to club.exe, have client output server data to file.
• [x] Implement the same process in Ogre client.
• [ ] Improve club.exe file downloading (threading, better UI/feedback).

[cyberjunk]

I noticed the MD5 is only generated on the first 1024 bytes of the club.exe file in the changes you made for OgreClient.

Is this the same for the patching all files in club.exe? I don't think this is sufficient enough to catch a changed file. Just think about a file which is changed but not in the first 1024 bytes (easy - like adding to Meridian59.layout file at the end).

Also I didn't get why the filename is/was included. What's the idea behind that?

[cyberjunk]

I didn't review the latest club.exe, but it feels like this is still fully sequential. One Download after the other?

Looking at the small sizes of M59 files and the overhead for HTTP connections, I guess patching could be speeded up a lot by downloading multiple files at once.

Suggestion for club.exe: 1) Use a threadsafe/concurrent "queue (ordered)" or "bag of things (unordered)" collection. 2) Make the mainthread parse the JSON and create a task for each file-entry in the JSON. (task = instance of class/struct, which provides data and code to compare md5, download if different.) 3) Add all the task-object/struct instances to the collection from (1) 4) Fire up N worker-threads which loop and take out a remaining task-object from the collection 5) Work-Loop: (a) Compare MD5 (b) Possibly Download (c) Post-Feedback 6) ...

[cyberjunk]

Overall-Progress (not sure if this is in yet?): Sum up all file sizes from json = 100%. Sum up downloaded/checked file-sizes from json: = current %

[skittles1]

Think we covered most of this in IRC but posting here for reference.

I noticed the MD5 is only generated on the first 1024 bytes of the club.exe file in the changes you made for OgreClient.

Will take care of this ASAP. This is the way it was done in the patcher (for speed), originally it hashed the whole file but was quite slow and then it was changed to 64 bytes which didn't catch all files. I chose 1024 as an amount that caught all classic client file changes, but would prefer hashing the whole file (and we don't really have a choice). We probably need to change this in the patcher also regardless of the performance hit, but in the meantime the server operators can have separate patchinfo.txt files - one for the 1024 byte patcher hashes, one for full file in-client updater hashes.

Also I didn't get why the filename is/was included. What's the idea behind that?

For club's hashing algorithm I just used the one in the patcher hence the filename addition, so I'm unsure why that was added. Any situation I can think of where the filename would matter (e.g. it changes) would cause the file to be downloaded again anyway. I'll remove this unless you can think of a reason to keep it.

I didn't review the latest club.exe, but it feels like this is still fully sequential.

Downloads are sequential, I'll have to learn threading and set up the solution you outlined.

Overall-Progress (not sure if this is in yet?): Sum up all file sizes from json = 100%. Sum up downloaded/checked file-sizes from json: = current %

Currently club first sees the file to download as it iterates through objects in the JSON array, and finishes downloading that before moving on to the next one. I think even in the threaded approach scanning would be happening alongside downloads so a valid % won't be immediately available. The UI does need work and could eventually have "% files scanned", and "Calculating total downloads" which switches to "% of downloads completed" once all files are scanned.

[cyberjunk]

Currently club first sees the file to download as it iterates through objects in the JSON array, and finishes downloading that before moving on to the next one. I think even in the threaded approach scanning would be happening alongside downloads so a valid % won't be immediately available.

You will have to make a first run through all the entries in the JSON file to sum up their file-lengths, but this should be easy...

long max = sum(all file-lengths from json entries)

Then create a bytes-counter to track the finished overall-bytes

long done = 0;

When you iterate through the entries, you increase the "done" bytes value.

• In case the MD5 matches, increase done-byte counter directly by the file-length of the JSON-entry just processed.
• In case the MD5 does not match, increase done-byte counter when receiving bytes or in intervals or at least when download finished.

At any time, your current overall progress is available by the ratio of:

progress = done / max;

Progress will be a value between 0.0 (start) and 1.0 (end) which can easily be used for an overall-progress bar.

PS: A simpler solution would be to use the ratio of files, but due to different sizes this is not as good:

progress = processed files / overall files

[cyberjunk]

Also:

I think even in the threaded approach scanning would be happening alongside downloads

Not sure if I'm getting this right, technically there are 2 thing to be done per file: a) Calculate MD5 b) Possibly download

And there are two approaches: 1) Calculate MD5 of all files first and afterwards start download of some. 2) Calculate MD5 of a file and immediately download before processing next file

I think option 2 is a bit better for a multithreaded solution where a worker gets one file as a worktask.