rminderhoud
commented
6 months ago Here is my assessment of the current situation using FLTK + bita
Pros:
- No update server required, works using HTTP range request + precalculated rolling hashes
- FLTK callback UI system is okay, not ideal but manageable for making a UI
- We don't have to create delta patches, we can just rebuild and deploy client on every release. This makes it a lot easier for deployment and less error prone
- Updater is already cross platform, we just don't distribute cross platform binaries
- Storing the local file hashes in a "manifest" rather than running it every time is fast for no-updates required
Cons:
- bita doesn't work well on text files, we had some files get corrupted
- bita API is a bit messy to work with when trying to build
- Manual updates are a bit tedious since we have to update only one bita archive and manually edit the manifest
- FLTK webview is a pain, on windows not all players have the webview installed and basically makes proton/crossover/wine/etc. not plug-and-play
Some notes regarding your original post:
- bita works on more modern versions of wine, the issue persists with the webview in fltk and egui didn't work in our last test
- steam doesn't require us to use our updater at all, we can launch the game directly and use steam to update. You just publish a build and they deal with the patching for you.
On the bita front, I think our perspective is also sufficiently biased because we incorrectly used bita so the updater is much slower than it needs to be. There are essentially two steps: download new chunks and scanning for chunks. Unfortunately we don't do this concurrently so we have poorer performance than necessary. In my egui refactor this is A LOT faster when properly creating different tasks and maximizing concurrency which is pretty essential since both tasks are IO heavy (network & file). To me the only downside of bita is that it's a bit tricky to use it correctly and creating the archives is an extra step, however to me the trade-off seems worth it to avoid having to make manual deltas between updates and/or run an actual update server. In my initial research I was planning to use the rsync protocol but that required a server so I opted against it. It seems possible to use the rsync algorithm without the protocol (librsync) but it was not obvious to me how to use it. In my research I discovered more "modern" alternatives to the rsync protocol such as zsync which is used by ubuntu, casync, etc. Funnily enough I happened to accidentally discover bita which is modeled after zsync/casync/etc. in that it uses HTTP requests to get new blocks. Ultimately, TODAY I don't see much advantage to be gained dropping bita. It has enough knobs I think we can optimize our usage further and any similar tech even if hand-rolled would require something similar (archive format, chunk dictionary, etc.) Perhaps it would be a bit cleaner/simpler than bita but might as well contribute to bita then. Biggest pain point is you have to roll your own progress code which I did in the egui version.
Don't even get me started on the UI front...Finding an ideal desktop gui toolkit is challenging these days. We used fltk originally due to time constraints and it was quite easy to get going. I regret the webview part, it's a pain. Sounded nice on paper but some windows users don't have the webview component pre-installed and it's basically a big pain to get it installed in wine/proton/etc. For the second version I chose egui since it was Rust native and bita is in Rust. It was a bit hard to do the layout in egui so not sure it was the best choice for this kind of app where we want nice and fancy layouts. Also in our tests it basically did not work in proton on steamdeck so that's what killed that initiative.
I was planning to investigate using slint for the UI here. My initial experience was rather positive. Hot reloading of template files, vscode plugin made rapid iteration quickly, supports multiple backends. I did not get too far but for the updater I think it's not a bad choice if we can confirm it works on wine/proton/etc. I did consider avalonia/qt/etc. but felt tedious to go cross-language.
The updater has been a constant pain point since we've launched.
As such, we're looking for possible solutions to improve the updater. We can split the problem in two: the updater not being platform independent and the updater not correctly patching and not correctly reporting progress.
Updater Platform Independent
In order to make the updater platform independent we need to solve two issues: tokio async somehow doesn't work on wine and the UI using a webview are the two blockers.
Tokio Async on Wine
Tokio async has had issues in the past on wine. That may have been solved, to check. Another possibility would be to try std async, or even switch languages to C# for example (we already have some C# in our internal repos, so it's an acceptable solution to have C# in the updater. Other than C++/rust/C# languages will need to be evaluated before approval).
Webview
The second issue is that the UI currently uses a webview to display news on the updater. The solution here would be to get rid of the webview and fetch the remote resources via HTTP directly, while having a fixed layout in the app. This way we can display news while avoiding to run a whole webview. This can be done in any UI framework really.
Updater not Correctly Patching
We've been using bita for our patching algorithm because of its ease of use on the server side. However we've been having a lot of problems with files not being updated or somehow being truncated after an update. As such, we propose to move away from bita to another algorithm. We currently have three algorithms in mind: per file patching, bsdiff and valve's patching.
Per File Patching
This is the simplest algorithm that has been used since forever. It's also very popular with games and especially MMOs because we can make sure the files are physically in the correct place for speed. Push the modified files on the server, calculate the hashes, then on the client download the hashes, compare with the current files and download each modified file from the server. This implies that the updater can open and modify the binary data blob or that we distribute the modified files outside the binary blob as an "overwrite".
bsdiff
bsdiff is a diff algorithm that operates specifically on binary files. A nice discussion on bsdiff and some game-related opinions in there: https://news.ycombinator.com/item?id=27391255
Valve's patching
The big upside is that we would use a patching method that has been tested and has been shown to be working. It also would make our transition to steam a lot easier if we ever want to do it.
This issue aims to determine and choose solutions for all the above problems. Other possible solutions not proposed here are also welcome.