Build container images without requiring docker-in-docker or privileged access

[acceleratesage]

Is your feature request related to a problem? Please describe.

The goal is to be able to containerise pants CI builds without requiring docker-in-docker or privileged access. Ideally I'd want to be able to run pants and build container images on container as a service platforms like AWS Fargate or Azure Container Instances.

Describe the solution you'd like

I'd like to leave the solution space open for the Pants team to figure something out that fulfils the need and fits with how Pants does things. Personally I do like Bazel's approach, however that may not fit with Pant's philosophy of integrating with existing tools instead of offering custom logic.

Describe alternatives you've considered

Other approaches are:

• Bazel's rules_docker allows building images without docker installed or root access by not using RUN instructions but instead adding deterministically built image layers to existing base images. This option seems to offer the most reproducible / deterministic builds at the cost of not using Dockerfiles.
• Google's Kaniko allows building images from Dockerfiles in a Kubernetes cluster or container in userspace. See e.g. this AWS blog post how to use it to build containers in AWS Fargate.
• Docker BuildKit allows rootless building of docker images based on Dockerfiles. It still requires setting seccomp=unconfined and apparmor=unconfined which may not be supported on CaaS platforms.

Additional context

• The slack thread that led to this ticket

[kaos]

Taking inspiration from Bazel's rules_docker, I'm investigating the use of two tools, umoci and skopeo, to create and manage docker images without the need for a docker daemon.

Another option also worth exploring is buildah.

Another related interesting tool is kiwi which supports building any kind of image (Live CD/USB media/Virtual disk etc) in addition to docker images (kiwi uses umoci and skopeo to support the docker image format). That would be a completely new backend, however..

[acceleratesage]

@kaos Just FYI, I no-longer have need for this feature, as I have moved on from using pants to something else. Would you want me to close the ticket or should I leave it open?

I think there were others interested in this, however I do not want to request this without actually needing it at the moment.

[kaos]

Ah, ok. Thanks. I think we can leave this open. The feature is still useful to have, and there'll be others that may use it.

[Kaelten]

This would be a potential blocker for me to go all in with pants. Even if I used pants to build the pex files, I don't have docker-in-docker in CI, that's powered by kaniko in our kubernetes cluster.

[djuarezg]

Without this, building pants images in an unprivileged CI environment becomes impossible, and given you should not have privileged mode enabled as a rule of thumb when dealing with shared CI infra, this is actually pretty relevant to implement.

[rbuckland]

The folks at kaniko' advice is that kaniko is run from it's own docker image, and is not really a "tool" to operate inside another container. This thread does shed some light on why that is the case.

https://github.com/GoogleContainerTools/kaniko/issues/1757

Goal

• allow pants, inside of a docker image, to build and publish docker images

Options With that said, a build backend in pants, for docker, which is "non dind" is very important.

• umoci and skopeo,
  • umoci - building images
  • skopeo - publishing
• kiwi (as mentioned above)
  • which uses umoci and skopeo
• buildah

Potentially Discounted Other Options

• podman

  • it needs privileges in the running container (ref: https://www.redhat.com/sysadmin/podman-inside-container)

• Docker BuildKit allows rootless building of docker images

  • looks like it needs seccomp,apparmor disables
  • /dev/fuse mount (not sure the use case)

• kaniko - as mentioned above

[tgolsson]

Regarding umoci + skopeo as a tool; I maintain a custom pants plugin for that: pants-backend-oci. It's far from feature complete but we use it in production in our ML pants repo and it works for our use-cases. Supports both running in rooted and rootless modes, though rootless requires some manual setup depending on your uid/gid mappings.