Improve the repository format

[ikeycode]

Currently we use a flat index for our repository format, and this works rather well. However, we need to add a couple of layers above this in order to support an "Update forever" approach.

Chiefly, we need to generate a root level index on the repo side which in turn has links to moss format versioned indices. So, for each versioned release of the repo, the tooling repo side would record the highest used moss format version in all of those packages, and in turn set the minimum required moss version as metadata in the root level index.

Client side we would only look at the highest moss version that we can support, filtering out later moss versions to eliminate any unsupported features / fields creeping into the equation. We will need to decide (soon) how we wish to "cut" releases on the repo side, and establish a shared format in libmoss/vessel/moss.

Lastly, we also need to minimise the time spent refreshing indexes. As we need to support dumb-transport / latent mirror cases, we cannot always rely on http headers for "changed since" usage. It is more likely we need to emit a detached checksum (potentially as a moss format files) that includes a new Checksum type, eliminating the need to hex-encode hashes.

Checksum { type, len, data }

We should also work to incorporate the Checksum type as a replacement for PackageHash meta field in future, by renaming the existing enum member to LegacyPackageHash, accounting for a new PackageChecksum that is strongly typed and robust.

Human explanation:

this is a bit of an explanation re versioned repos. tlder if we add new things to the format or change it, you'll need a version of moss capable of reading it so we lockstep each repo version to the moss version this way we can push out a new moss build that will enable incoming features, and cut a release then we can land packages using those new features ahead of that cut point so your client wont see the new packages yet, they'd need to move to the new moss first in the last supported version it has the effect of staggering the update into chunks ensuring the right moss is in use

[ikeycode]

When designing for this issue we also need to keep #50 in mind, in terms of being able to sign indices as well as individual stones.

[ermo]

Some more notes:

Ikey Doherty The problem we have atm is interlinked We absolutely need versioned repos Such that we protect the existing 6 volatile users And migrate from there With that in place we'll have a bigger safety net As we can retrofit to D infra Then even an old iso will install last pkg to hit volatile, and perform staggered updates Ie volatile will need to be preserved and a new dumping repo for builder network ermo So there's a moss side and a services side to this it seems? Ikey Doherty Yep, interlinked But what we will do is a series of clever changes Such that we'll make a vendor file for volatile that sets an enabled field to false And it'll override the local added repo in etc from the iso And that moss update will also drag in a new vendor config for the new version capable repo Along with version support in that moss Basically the first "epoch bump" From thereon we'll be syncing/ releasing explicitly ermo

Ikey Doherty Such that we'll make a vendor file for volatile that sets an enabled field to false

So this is for the config what we distribute with ... moss? Ikey Doherty Yep Our configuration files get merged Without an enabled field in the etc file, it'll be masked Mildly evil but hey. We need to pull volatile out of circulation ermo Ikey Doherty Without an enabled field in the etc file, it'll be masked

Well, in this case, this implementation detail works to our advantage. Clever. (if possibly unintended) Ikey Doherty Alternative would be special handling of unversioned repo Either way, we'll make an upgrade path away from volatile ermo I think we should explicitly disable volatile in a visible vendor config FWIW (to make it clear that this is not to be pulled from) Ikey Doherty I'd agree with that And some surrounding comments Then new iso builds will need to unload their repo definitions after populating So no more of us abusing /etc Anyway, I see it going well xD