Add support for Mirrorbrain features

[TheAssassin]

CC AppImageKit, AppImageSpec, @probonopd, @adrianschroeter

Some projects like KDE or openSUSE use Mirrorbrain to publish their releases. To make it easier for such projects to add update information to their AppImages, we could add an update type supporting this platform.

Mirrorbrain differs from the random server by:

• providing a list of mirrors of the files, sorting them by a few factors like "distance" etc.
• provides integrity information, like file hashes, GPG signatures, etc.
• while some of the mirrors support encrypted connections (HTTPS), Mirrorbrain does not (according to the KDE sysadmin team), and advertising those with priority would require patches to Mirrorbrain, and since the project seems unmaintained, that'd require a fork

I already asked in #kde-sysadmin on Freenode about their infrastructure. My initial intention was to make them enforce HTTPS in their mirror infrastructure (in days of Let's Encrypt, this is a valid request), as that also solves the problem "people observing data stream can easily see what files are being requested" (because that's what people don't think of when talking about HTTPS), and obviously, people being too lazy to actually check the files they download, so having the integrity and (server) authenticity checks HTTPS enforces for those initial downloads would be awesome.

I think if we wanted to help them provide updatable AppImages from such an infrastructure, we should add a Mirrorbrain update information type which at least performs the update checks, because I think it Mirrorbrain can redirect automatically if one requests the right URL from the beginning.

Thinking about it again, IIRC, Mirrorbrain returns at least the hashes as HTTP headers, so, to achieve integrity checks based on these (because GPG signatures without verifying the keys aren't anything but pure hash-based integrity checks), we could make use of those by default, if available. This is primarily useful when downloading the .zsync file from an unauthenticated server (after the redirect from the HTTPS enabled mirrorbrain master): When it's downloaded, check its integrity to make sure the right file has been downloaded. This is similar to how APT works -- they sign and verify the central file which contains hashes of all the files in the repository, and we do the same, as the .zsync file contains integrity information (file hash) of the target file, hence one can ensure that the target file's integrity check is reliable after verifying the .zsync file's integrity. @adrianschroeter once said in the Mirrorbrain configs it was possible to serve certain files from the master server, but we can't/shouldn't rely on this to be configured correctly.

What do you guys think? Personally, I believe we have to improve this situation, especially since we're advertising [OBS], whose artifacts storage is regularly redirecting to HTTP-only servers (I think I posted that somewhere in the OBS repository, if anyone's curious, I don't have the link any more), and therefore we should at least fix one of the problems due to the lack of HTTPS.

Side note: I was analyzing the Kdenlive infrastructure initially, having me analyze the behavior. I verified this with openSUSE's installation because that service is hosting a large share of AppImages. Kdenlive don't provide updates yet, this will be subject of a second issue.

[TheAssassin]

P.S.: This does not solve the problem of initial distribution, which is why I asked for HTTPS support in #kde-sysadmin at first (because there, HTTPS's integrity checks and server authentication could quite some value even if this feature proposal was implemented here (for initial downloads, real world users won't ever run sha*sum and make use of the hashes published on Mirrorbrain...).

[nicolas17]

Note that in theory none of that is mirrorbrain-specific. It's all defined by Metalink/HTTP (RFC6249), which references other specifications like Instance Digests in HTTP (RFC3230) for the hashes.

However, I'm not sure if there's any other non-obscure server software providing Digest headers...

[TheAssassin]

@nicolas17 thanks for the follow-up. I was just finding that when curl -v-ing Mirrorbrain hosted files, and saw that in the headers. If it's standards, that's even more awesome.

[probonopd]

Yes, I agree that proper Mirrorbrain support would be a huge plus, especially given that OBS and KDE are using it. However I'd leave the technical details to people who know more about it than I do.

It's all defined by Metalink/HTTP (RFC6249), which references other specifications like Instance Digests in HTTP (RFC3230) for the hashes.

Even better!

[TheAssassin]

So, what we really need, is:

• [ ] support for meta links (RFC 6249) and/or Web Linking (RFC 5988)
• [x] support for HTTP digests (RFC 3230)
• [x] if it is impossible to verify the integrity of the data downloaded from the server one has been redirected to, show a big warning (and maybe even refuse to work)

[TheAssassin]

HTTP instance digests support has been implemented with #58. The update will fail when the instance digest can't be verified. It works fine with AppImages distributed on the openSUSE download infrastructure.

[probonopd]

Please also try with the KDE download infrastructure.

[TheAssassin]

They don't have .zsync files, as you know. But since they use Mirrorbrain as well (and their server yields the same set of instance digest headers), I am pretty confident that it will work.

[probonopd]

They don't have .zsync files, as you know.

Ah, right... hopefully that will start to change soon :-)

[TheAssassin]

Maybe. At least now we can suggest them to implement update information.

I wish we had a secure solution for initial distribution, too, though.

[TheAssassin]

Edited TODO list with the point "Web Linking (RFC 5988)". This is a lot easier to support than meta links, as web linking is based on HTTP headers, just like instance digests.

[probonopd]

The title of this ticket says "Add mirrorbrain update information type" - is this still something we need to do?

[TheAssassin]

Not really, I'll change it.