automatic update support in iroh {one, cloud}

[b5]

we need automatic updates.

the self update crate seems like a good place to start after some initial googling.

• add an auto_update_check_frequency_hours configuration for number of hours to wait between update checks
  • let's use hours as a minimum so folks can't bomb the update server with an accidental bad config
• when set to zero auto update is disabled
• default value for iroh one should be something like every 3 days: 72 (24*3)
• auto update running in iroh cloud can lead to downtime, and should be disabled by default (set to a value of zero)
• we likely shouldn't use auto update for major version changes, as this could break APIs for folks
• we'll need to cache the timestamp of the last check to a file so update checks survive between process restarts
• we'll need to document the explicit request pattern & any telemetry collected when an auto-update request is made
• will need nice log output for updates

[fabricedesre]

I'm skeptical that updates are something you should deal with in iroh itself. I think it would be better to create packaging for your most common deployment targets (ppa for debian, etc.).

But if you decide to go with that, please make that feature gated, because it won't be useful for everyone. For instance:

• people that would rather rely on their OS package manager.
• environments where iroh is running from a read-only partition.

[b5]

Great points @fabricedesre, there's OS / package manager nuance I hadn't considered, and we should be clear about the point of integration for automatic updates to avoid code bloat & potential unwanted network calls.

The main reason auto-update really is our problem surrounds the slow pace of upgrading on the IPFS network itself. This is an incredibly real concern when rapidly iterating on a distributed system. With that in mind, there are lots of places where auto-update doesn't make sense.

Regarding package manager distributions, we should aim to ship package-manager-specific configurations, and in those configurations we can leave auto-update off by default. I think we should ship auto update at the highest level in our stack, just inside CLI crates.

The target I care about this for the most is iroh one running on the desktop, closely followed by iroh cloud. Auto update in the cloud sounds like a horrible thing for uptime, but I've encountered many small teams running IPFS infra that just need to keep the lights on, and will accept a little downtime if it means they have one less human-in-the-loop thing to deal with. This assumes we actually get this feature right, and can gracefully stop, upgrade, and restart iroh services. Not simple.

Regarding running from a read-only partition, I'm of the mind that's a case our auto-update mechanism would have to handle gracefully in all contexts (weather it's running or not).

So I think we do need this, only in some places, and it's hard to get right so it will be a while before we get around to it.

[fabricedesre]

For anything that is supposed to run as unattended as possible, I still think it's better to not use a custom updater and just rely on the host package manager. You can still ship updates as often as needed. Of course that means having a good packaging story, which is not always trivial either.

But your "iroh one running on the desktop" makes me think that maybe you have in mind a product that is more end user oriented, bundling iroh-one with some nice UI? That would be a different story, these kind of products (like web browsers or electron apps) indeed tend to have their own auto-update mechanism.

Lastly, some food for though: I've wondered if ipfs could be an ideal candidate to reduce the amount of data to download for updates: consider you're on a "daily" update channel. It is likely that there are few changes from your current state, so serving the update over IPFS with a smart chunking approach could reduce the number of CID to fetch. I would be very interested in using something like that for Capyloon updates.

[b5]

For anything that is supposed to run as unattended as possible, I still think it's better to not use a custom updater and just rely on the host package manager.

The numbers for the public IPFS network don't support this conclusion, kubo already has a package manager story, and more than 1/4 of the network is still running go-ipfs v0.7.0:

source

But your "iroh one running on the desktop" makes me think that maybe you have in mind a product that is more end user oriented, bundling iroh-one with some nice UI?

We don't have plans to build this kind of thing, but I don't think that's a reason to not have auto-update. I understand this feels weird & very un-unix, but the numbers are the numbers. having 1/4 of the network running p2p software with known vulnerabilities is an issue we can't leave unaddressed.

With that said, automatic updates and a solid package manager story are composable solutions, and having a solid package manager story should come first. We'll put effort into getting this right with at least a select set of package managers on our first release.

[b5]

Lastly, some food for though: I've wondered if ipfs could be an ideal candidate to reduce the amount of data to download for updates: consider you're on a "daily" update channel. It is likely that there are few changes from your current state, so serving the update over IPFS with a smart chunking approach could reduce the number of CID to fetch. I would be very interested in using something like that for Capyloon updates.

💯 yes. kubo already uses IPFS to download updates, and we should do this. I'm not sure we need custom chunking

[fabricedesre]

Lastly, some food for though: I've wondered if ipfs could be an ideal candidate to reduce the amount of data to download for updates: consider you're on a "daily" update channel. It is likely that there are few changes from your current state, so serving the update over IPFS with a smart chunking approach could reduce the number of CID to fetch. I would be very interested in using something like that for Capyloon updates.

💯 yes. kubo already uses IPFS to download updates, and we should do this. I'm not sure we need custom chunking

I don't know for sure either, we need data :) Considering a situation like:

• v1.0.0 is installed from scratch through n0.computer/install.sh
• v1.0.1 is released, the auto-update mechanism downloads it from ipfs: we need to fetch it all.
• v1.0.2 is released: what is the % of blocks fetched for 1.0.1 that are also used in this 1.0.2? Can we optimize that to reduce data transfers for updates?

I guess that depends on the format of the update package - is that more like a tar file or more like a disk image, etc.