API(s) to support modification

tri-adam commented 2 months ago

Spec out one or more API(s) to support the addition of an overlay, and to update the relevant image manifest/config and root index after an overlay is modified (https://github.com/sylabs/singularity/issues/1480). Also see discussion in https://github.com/sylabs/oci-tools/pull/46.

dtrudg commented 2 weeks ago

Assuming we are following the flow for Singularity detailed in https://github.com/sylabs/singularity/issues/1480 we need to be able to perform 2 modification operations for minimal functionality, with an additional 'seal' operation if we wish to allow images with overlays to be finalized and / or pushed to an OCI registry as an OCI image with r/o tar layers.

1. Add EXTFS Overlay

We must be able to add an extfs overlay, of configurable size, to the SIF file. The overlay will be added to the image config as an additional layer with an extfs mediaType. The image config, manifest, and RootIndex will be updated so that all hashes are consistent at the time of insertion of the extfs overlay layer.

To avoid re-writing the entirity of the SIF file, we would need to enforce that all existing image layers are located at the start of the SIF file. Large images can have many and large layers, so we don't want to re-write the whole SIF if at all possible.

For an image with N layers - Before:

descriptor id	type	content
1	OCI.Blob	SquashFS layer
...	OCI.Blob	SquashFS layer
N	OCI.Blob	SquashFS layer
N+1	OCI.Blob	Image Config
N+2	OCI.Blob	Image Manifest
N+3	OCI.RootIndex	Image Index

After the addition of an extfs overlay:

descriptor id	type	content	state
1	OCI.Blob	SquashFS layer	Unmodified
...	OCI.Blob	SquashFS layer	Unmodified
N	OCI.Blob	SquashFS layer	Unmodified
N+1	OCI.Blob	EXTFS overlay layer	New
N+2	OCI.Blob	Image Config	Updated
N+3	OCI.Blob	Image Manifest	Updated
N+4	OCI.RootIndex	Image Index	Updated

2. Update Metadata

As the EXTFS overlay layer is modified, the image config, manifest, and RootIndex will become inconsistent w.r.t. the hash for the extfs overlay layer. Due to concurrency issues, the risk of batch jobs being killed etc. in the SingularityCE use case it is probably not practical to update the image config, manifest, root index after each use of the overlay in r/w mode. However, before we share the image in a registry, we need the hashes to be correct.

We must have an operation that updates the image metadata, rewriting image config, manifest, rootindex appropriately.

Before metadata is updated:

descriptor id	type	content
1	OCI.Blob	SquashFS layer
...	OCI.Blob	SquashFS layer
N	OCI.Blob	SquashFS layer
N+1	OCI.Blob	EXTFS overlay layer
N+2	OCI.Blob	Image Config
N+3	OCI.Blob	Image Manifest
N+4	OCI.RootIndex	Image Index

After metadata is updated:

descriptor id	type	content	state
1	OCI.Blob	SquashFS layer	Unmodified
...	OCI.Blob	SquashFS layer	Unmodified
N	OCI.Blob	SquashFS layer	Unmodified
N+1	OCI.Blob	EXTFS overlay layer	Unmodified
N+2	OCI.Blob	Image Config	Updated
N+3	OCI.Blob	Image Manifest	Updated
N+4	OCI.RootIndex	Image Index	Updated

3. 'Seal' Overlay

When a user has finished modifying an image via an overlay, we may wish to allow them be able to 'seal' the image. This will convert the EXTFS overlay layer into a SquashFS layer.

Sealing an overlay - into a r/o layer - is also a prerequisite for being able to subsequently push the image in tar layer format, compatibile with other runtimes.

Before image is sealed:

descriptor id	type	content
1	OCI.Blob	SquashFS layer
...	OCI.Blob	SquashFS layer
N	OCI.Blob	SquashFS layer
N+1	OCI.Blob	EXTFS overlay layer
N+2	OCI.Blob	Image Config
N+3	OCI.Blob	Image Manifest
N+4	OCI.RootIndex	Image Index

descriptor id	type	content	state
1	OCI.Blob	SquashFS layer	Unmodified
...	OCI.Blob	SquashFS layer	Unmodified
N	OCI.Blob	SquashFS layer	Unmodified
N+1	OCI.Blob	SquashFS layer	New - conversion of extfs layer
N+2	OCI.Blob	Image Config	Updated
N+3	OCI.Blob	Image Manifest	Updated
N+4	OCI.RootIndex	Image Index	Updated

tri-adam commented 2 weeks ago

1. Add EXTFS Overlay

We must be able to add an extfs overlay, of configurable size, to the SIF file. The overlay will be added to the image config as an additional layer with an extfs mediaType. The image config, manifest, and RootIndex will be updated so that all hashes are consistent at the time of insertion of the extfs overlay layer.

To avoid re-writing the entirity of the SIF file, we would need to enforce that all existing image layers are located at the start of the SIF file. Large images can have many and large layers, so we don't want to re-write the whole SIF if at all possible.

For an image with N layers - Before:

descriptor id type content 1 OCI.Blob SquashFS layer ... OCI.Blob SquashFS layer N OCI.Blob SquashFS layer N+1 OCI.Blob Image Config N+2 OCI.Blob Image Manifest N+3 OCI.RootIndex Image Index After the addition of an extfs overlay:

descriptor id type content state 1 OCI.Blob SquashFS layer Unmodified ... OCI.Blob SquashFS layer Unmodified N OCI.Blob SquashFS layer Unmodified N+1 OCI.Blob EXTFS overlay layer New N+2 OCI.Blob Image Config Updated N+3 OCI.Blob Image Manifest Updated N+4 OCI.RootIndex Image Index Updated

I think there are a couple of steps here. At a minimum:

Construct an updated v1.Image with the EXTFS layer and updated image config/manifest. mutate.AppendLayers can be used for this purpose. You would need to construct a v1.Layer containing the EXTFS filesystem, which is probably best done in the SingularityCE code base?
Replace the original v1.Image with the updated v1.Image. Here, we probably need an API in github.com/sylabs/oci-tools/pkg/sif. Perhaps something like:

func Update(path string, ii v1.ImageIndex, opts ...UpdateOpt) error

Update would expect an OCI-SIF at path and replace its rootIndex with ii. The logic for this could be something like:

Generate a list of all OCIBlob digests referenced by ii, and digest of new rootIndex.
Remove all SIF descriptors not in the list of references (including rootIndex if it's been updated.)
For each descriptor in the list of references, check if the blob is already in the SIF. If not, add it to the SIF.

2. Update Metadata

As the EXTFS overlay layer is modified, the image config, manifest, and RootIndex will become inconsistent w.r.t. the hash for the extfs overlay layer. Due to concurrency issues, the risk of batch jobs being killed etc. in the SingularityCE use case it is probably not practical to update the image config, manifest, root index after each use of the overlay in r/w mode. However, before we share the image in a registry, we need the hashes to be correct.

We must have an operation that updates the image metadata, rewriting image config, manifest, rootindex appropriately.

Before metadata is updated:

descriptor id type content 1 OCI.Blob SquashFS layer ... OCI.Blob SquashFS layer N OCI.Blob SquashFS layer N+1 OCI.Blob EXTFS overlay layer N+2 OCI.Blob Image Config N+3 OCI.Blob Image Manifest N+4 OCI.RootIndex Image Index After metadata is updated:

descriptor id type content state 1 OCI.Blob SquashFS layer Unmodified ... OCI.Blob SquashFS layer Unmodified N OCI.Blob SquashFS layer Unmodified N+1 OCI.Blob EXTFS overlay layer Unmodified N+2 OCI.Blob Image Config Updated N+3 OCI.Blob Image Manifest Updated N+4 OCI.RootIndex Image Index Updated

This should be fairly do-able. Again, you'll need to construct an updated v1.Image, this time that contains an updated EXTFS v1.Layer. This should just be a matter of calculating the layer digest, constructing a v1.Layer similarly to the previous step, and then using mutate.SetLayer to replace the old layer.

From there, if you call Update as described above, I think it might just work?

3. 'Seal' Overlay

When a user has finished modifying an image via an overlay, we may wish to allow them be able to 'seal' the image. This will convert the EXTFS overlay layer into a SquashFS layer.

Sealing an overlay - into a r/o layer - is also a prerequisite for being able to subsequently push the image in tar layer format, compatibile with other runtimes.

Before image is sealed:

descriptor id type content 1 OCI.Blob SquashFS layer ... OCI.Blob SquashFS layer N OCI.Blob SquashFS layer N+1 OCI.Blob EXTFS overlay layer N+2 OCI.Blob Image Config N+3 OCI.Blob Image Manifest N+4 OCI.RootIndex Image Index descriptor id type content state 1 OCI.Blob SquashFS layer Unmodified ... OCI.Blob SquashFS layer Unmodified N OCI.Blob SquashFS layer Unmodified N+1 OCI.Blob SquashFS layer New - conversion of extfs layer N+2 OCI.Blob Image Config Updated N+3 OCI.Blob Image Manifest Updated N+4 OCI.RootIndex Image Index Updated

This will again follow a similar flow I think. The tricky part here will be the EXTFS->SquashFS conversion, and I'm not sure if this belongs in this repo or in SingularityCE. I'd lean towards CE for now, as EXTFS seems fairly specific to that at the moment?

Once you have a v1.Layer containing the result of the conversion... it should be a matter of mutate.SetLayer and then Update again?

sylabs / oci-tools