search

confidential-containers / cloud-api-adaptor

Ability to create Kata pods using cloud provider APIs aka the peer-pods approach
Apache License 2.0
44 stars 71 forks source link

podvm: component caching #1816

Open stevenhorsman opened 2 months ago

stevenhorsman commented 2 months ago

With the move to use kata-container's main branch we also get access to their component caches, so could potentially significantly simplify our podvm image build code and improve the performance (as the kata-agent is a big source of the time it takes and the pause image bundle could allow us to remove skopeo and umoci). There are a few issues with this that we should discuss first.

  1. Kata currently only tags the latest version, and we need pinned versions, to avoid having a creeping onward release. I have a draft PR https://github.com/kata-containers/kata-containers/pull/9550 that should help with this
  2. For specifically the guest-components - kata only caches the version that it integrates into it's rootfs https://github.com/kata-containers/kata-containers/blob/2400a4d249dc7f4e0d2f78aec8492c58e2c7a256/versions.yaml#L226-L230 I thinking that in general we are trying to keep in step and implement the same things as local hypervisor, but are we happy to be tied to stay in sync/only on versions that kata has integrated?
  3. We currently support the ability via our https://github.com/confidential-containers/cloud-api-adaptor/blob/main/src/cloud-api-adaptor/versions.yaml file to build the kata-agent and guest-components from any repo, so if users want to try out forks they can do. If we only moved to the oras cache then we could use this file to point to the registry and tag that we get the component from, but that would mean that users had to do prepare the components in the right format and oras push of any dev changes, which might be prohibitive. Instead if we could optimise the https://github.com/confidential-containers/cloud-api-adaptor/blob/main/src/cloud-api-adaptor/podvm/Dockerfile.podvm_binaries / https://github.com/confidential-containers/cloud-api-adaptor/blob/main/src/cloud-api-adaptor/podvm/Makefile.inc so that a user could "pre-build" their binaries to somewhere in the src/cloud-api-adaptor/podvm/files/ directory (as Pradipta is doing with the resources/binaries-tree in the docker provider) that we'd skip the cache pulling that could provide a less arduous dev cycle. Alternatively we first check if the version file is a cached image and if not pull it and build it like today, but that adds complexity rather than removing it.
mkulke commented 2 months ago

IMO it will be good to dogfood the upstream builds where feasible, but I agree w/ 1): we need to be able to reference builds with their revision.

Another upside is also that the bespoke build logic/params that we currently thread through several layers of Dockerfiles and Makefiles can be removed from the project.

I would still recommend we have the option of manually building guest binaries out-of-tree and be able to use them in image build, possibly a simple set of make targets that will only download a binary if they are not present yet is sufficient.