Add canonical docs/standards (maybe tooling here) for integration w/OCI

[cgwalters]

We should standardize some of the interactions with composefs and OCI. Today the composefs tooling is very generic, and integration with OCI or other ecosystems is left to do externally (as is happening in e.g. containers/storage).

Embedding containers.composefs-digest as metadata

While this is a broad topic the first example I'd give here is that we should standardize embedding the composefs digest in a container image manifest; much as was done with ostree and embedding it in in the commit metadata.

Something like a standard containers.composefs-digest (bikeshed: label or annotation?). And we should define exactly how a container image is mapped to a composefs tree. Specifically, I would argue here that the embedded digest should be of the merged, flattened filesystem tree - and that's actually how it should be mounted as well (instead of doing it via individual overlayfs mounts) - i.e. we'd do it how ostree does it.

However, it wouldn't hurt to also embed an annotation with the composefs digest for each individual layer (as part of the descriptor metadata) to give a runtime the ability to selectively choose to manage individual layers or not.

Finally of course, it would make sense for us to provide some tooling which does this. It's an interesting question, should there be something like podman build --feature=composefs to auto-inject this? But in the general case we can just provide a simple tool that accepts an arbitrary container image and "re-processes" it to add this metadata.

[cgwalters]

Something like a standard containers.composefs-digest (bikeshed: label or annotation?). And we should define exactly how a container image is mapped to a composefs tree.

I'm still thinking a lot about this. Here's a related PR: https://github.com/containers/composefs/pull/320

To elaborate on this, again what I really want is the signature on an image (for a manifest) to efficiently cover a composefs blob which is "the image".

To recap the proposal from the above PR it's basically that we take the 3 parts from an image and put them in a single composefs blob (with a single fsverity digest):

• /manifest.json # Also has an xattr user.composefs.sha256 with the full/descriptor sha256 digest
• /config.json
• /rootfs # the container root

Then, when building an image (I know this gets a little circular) we support computing the fsverity digest of that whole thing, and inject that digest as an annotation into a copy of the manifest.json with solely that difference which becomes the canonical manifest.json. The version in the cfs image can be transformed back into the canonical version by re-injecting that annotation (this would work well if the manifest is required to be in canonical form (though cc https://github.com/awslabs/tough/issues/810)).

This style of image with the containers.composefs-digest annotation I would call a "composefs verified OCI image" - it allows us again to have a cosign/GPG style signature cover that manifest, which covers the composefs digest, which covers everything else.

I think it's a really desirable property from such a layout that a fetched OCI image is "a single file" (or really, "a single composefs" - we expect a shared backing store of course) and e.g. a "fsck" style operation is just "composefs fsck" which can be efficiently delegated to the kernel with fsverity (lazily or eagerly).

This is ignoring the "tar split" problem of course, for supporting re-pushing images, it'd be nice to have that covered by composefs/fsverity...but...super messy without getting into "canonical tar" territory. At least for verified images.

To be clear of course, for unverified images (i.e. images that we just want to store locally as composefs, that don't have the precomputed annotation) we can stick anything else we want inside that local composefs, including the tar-split data.

TODO external signature design

We want to natively support e.g. https://github.com/sigstore/cosign to sign images that can be verified client side. cosign covers the manifest, which has the composefs fsverity digest of the "artifact composefs" with the manifest and config and all layers. TBD: Standard for location of signatures for composefs-oci.

Question for Miloslav: Does c/storage cache the signature on disk today?

[cgwalters]

I've been prototyping things out more in https://github.com/cgwalters/composefs-oci in the background, and one thing that I think is interesting is I needed to efficiently index back from a composefs to the original manifest descriptor sha256, so I added a user.composefs.sha256 extended attribute on the manifest JSON (stored in the composefs) for the use case of client synthesized composefs blobs.

For server signed composefs, we obviously can't do that because it becomes fully circular with the composefs digest covering the manifest. Maybe instead what we can do is always store the original manifest digest as an xattr on the composefs itself. That would mean it becomes "unverified state", but that's probably fine.

Still also thinking things through more...given that we know we need to also maintain individual layers, I think we should add an annotation on each layer with its composefs digest as well; this seems like a no brainer in general, but it would specifically help align with how c/storage is representing things today.

[cgwalters]

Tangential but interesting here...we could also look at "composefs as a DDI" where the EROFS is in the DDI but the backing store isn't, but would allow covering the erofs with a dm-verity signature in a standardized envelope.

But, we still have the need to represent layers and handle OCI metadata.