search

containerd / accelerated-container-image

A production-ready remote container image format (overlaybd) and snapshotter based on block-device.
Apache License 2.0
397 stars 72 forks source link

[Userspace Convertor] Overlaybd Layer Versioning for Deduplication #250

Open estebanreyl opened 8 months ago

estebanreyl commented 8 months ago

What is the version of your Accelerated Container Image

v1.0.2

What would you like to be added?

Layer deduplication needs to account for the underlying overlaybd-tool versions to avoid conversion mismatches.

Why is this needed for Accelerated Container Image?

As layer deduplication is currently unaware of any versioning for the underlying overlaybd tools, whenever overlaybd tools are upgraded, this can result in a situation where deduplication will try to reuse layers that are no longer compatible during conversion. This is also relevant for #249 and the non-user-space convertor, I think. We have been tracking this manually to prevent issues on our side, but a more programmatic approach is probably necessary in light of a few format issues we have discovered recently like https://github.com/containerd/overlaybd/issues/301 and running with root vs not doing so. Overall, we want to discuss how to obtain versioning from overlaybd-tools to account for breaking changes and take this into account for the layer deduplication.

Are you willing to submit PRs to contribute to this feature?

yuchen0cc commented 8 months ago

It's reasonable and important to aware changes of reproducible conversion.

First, we add a CI for userspace convertor to avoid unexpected inconsistency. https://github.com/containerd/accelerated-container-image/blob/main/.github/workflows/ci-userspace-convertor.yml (in #253)

Second, we make sure root & non-root will produce the same product. https://github.com/containerd/overlaybd/pull/303

For versioning for convertor, we are planning to use an option to get consistency version. Maybe --version or --consistency-version. The version should be a combined version of convertor and underlay overlaybd tools. If it's an unavoidable change (like #251), we update the version. @estebanreyl what do you think of this ?

estebanreyl commented 6 months ago

Sorry for my delay with replying, I completely missed your reply and forgot to check. The versioning suggestion and addition of the CI seem completely reasonable to me and great additions. As for what flag to use I am inclined to use the --version and internally have a consistency-version group to which a version belongs which is actually used for the deduplication. This way we can update the version independent of the deduplication logic and still maintain the deduplication. As for how to combine the overlaybd version into this that's where I am less clear. Is there a way today to determine the version of the tools once they are binaries? If so, we can note what --version goes with a specific overlaybd version, when possible, to manage the deduplication. I'd be happy to create a PR for expressing this initially over the next few days to have a more precise discussion if one is not already in the works on your end.