feat: support for different OCI build tools backend

[hellkrusher]

Feature request

Build the docker file with podman build or buildah.

Motivation

Docker is a paid product for most commercial uses. podman or buildah are open source.

Other

No response

[yubozhao]

Just curious, @hellkrusher. Is this a requirement for your org?

What do you think about this? @ssheng

[parano]

Had a quick chat with @aarnphm about this - I think Yatai already uses Podman in our image builders on kubernetes, but requires some hacking into the Bento's files. Ideally we should support it via the bentoml containerize CLI command and allow users to choose the backend.

[rajatjatana]

This would be a nice feature to have. My organization is currently trying to use bento (with yatai) and shifting away from docker in favour of podman.

[ssheng]

Makes sense to support different OCI builders. We can consider adding an argument in containerize to allow selection of building. @hellkrusher @rajatjatana How urgent is this for your organization?

[hellkrusher]

@ssheng, We currently do not have a way to containerize using bentoml due to the docker restriction. podman has an alias for docker buildx but removing the buildx check from _internal/utils/buildx.py only gets so far: $ bentoml containerize --no-cache --verbose knn:latest Building docker image for Bento(tag="knn:szd4wsq5pcoq2usu")... Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. error running container: error from /usr/bin/runc creating container for [/bin/sh -c rm -f /etc/apt/apt.conf.d/docker-clean; echo 'Binary::apt::APT::Keep-Downloaded-Packages "true";' > /etc/apt/apt.conf.d/keep-cache]: time="2022-09-02T16:22:53-04:00" level=error msg="container_linux.go:370: starting container process caused: error adding seccomp filter rule for syscall bdflush: permission denied" : exit status 1 ERRO[0000] did not get container create message from subprocess: EOF Error: error building at STEP "RUN rm -f /etc/apt/apt.conf.d/docker-clean; echo 'Binary::apt::APT::Keep-Downloaded-Packages "true";' > /etc/apt/apt.conf.d/keep-cache": error while running runtime: exit status 1 Failed building docker image: Command '['docker', 'buildx', 'build', '--progress', 'auto', '--tag', 'knn:szd4wsq5pcoq2usu', '--file', 'env/docker/Dockerfile', '--load', '--no-cache', '.']' returned non-zero exit status 1.

[tomomonta]

Just wanted to bump this.

We are in the same boat as rajatjatana. Want to use bento with yatai, but we need to use Podman.

[aarnphm]

Alright. It seems like folks want this feature. I will take a look at this.

[aarnphm]

Multiple OCI builders support for containerize

• Author: @aarnphm

Abstract

The ability to support different OCI builders allows users to use build tools that comply with their organisation's requirements and compliance.

Background

There are a few different OCI build tools that support buildkit:

• buildctl - the BuildKit client
• docker buildx - Buildkit support on Docker
• podman - Think of it as a docker replacement
• buildah - A lower-level tools to facilitate OCI image

Even though podman uses buildah under the hood, their container representation is different.

Proposal

Containerize will be using a new tool called bentokit, which will build OCI-compliant bento images.

bentokit build bento:tag --builder kaniko --opt ... --opt ...

supported builder includes [docker|podman|buildah|buildctl|kaniko]

Containerize will use bentokit under the hood.

Features that containerize will offer:

1. Specify the given builder for containerize via BENTOML_BUILDER:

BENTOML_BUILDER=podman bentoml containerize iris_classifier:latest

docker will be the default builder if not specified.

2. To pass in specific options/flags for given build tools, containerize will now unify all options via --opt:

bentoml containerize iris_classifier:latest --opt cache-to=registry/repo/cache --opt security-opt=appamor=unconfined

3. We will also allow user to pass options for given builder via BENTOML_BUILDER_OPTIONS:

BENTOML_BUILDER_OPTIONS="--shm-size=64m --cache-to=registry/repo/cache" BENTOML_BUILDER=podman bentoml containerize iris_classifier

Rationale

The ability to control builder and options via environment would help in certain K8s settings, where it is easier to control behaviour via an environment variable.

Unifying --opt variable allow containerize to be OCI tools agnostic, meaning it should work with any OCI tools backend users wish to use.

What this means with current containerize options?

This means that all --<options> under bentoml containerize will be removed, which means this will be a breaking change to bentoml containerize.

Implementation

bentokit be implemented in Go and create a binary that will be included within bentoml distribution. (wip)

bentokit will provide a build python API:

from bentoml.tools import bentokit

bentokit.build(bento_tag: Tag, builder="kaniko", **kaniko_options): ...

bentokit.build(bento_tag: Tag, builder="podman", **podman_options): ...

bentokit.build(bento_tag: Tag, builder="docker", **docker_options): ...

Direction we can take for bentokit?

Open for suggestion.

1. Dynamic build a container:

with bentokit.build(bento_tag):
    bentokit.Add("./model", "/src/model")
    bentokit.Run('[[ -f /src/model ]] && echo "Hello"')

[yubozhao]

WDYT about the name bentobuilder?

[aarnphm]

WDYT about the name bentobuilder?

The API would be a bit weird with bentobuilder? bentobuilder build?

I'm also thinking to extend a frontend for this tool as well, something like

# syntax=bentoml/bentokit:main

[aarnphm]

Hi @tomomonta, is it possible for you to run buildctl with podman? buildctl is the standalone client to build image with BuildKit?