Add --config option to the bentoml containerize command

[ssheng]

Is your feature request related to a problem? Please describe.

With the recent configuration revamp, building the config file into the Docker image is the last missing link. Without such option, users will have to mount a shared volume through the docker run -v option and setting the BENTOML_CONFIG environment variable to point to the config file in the shared volume, to pass a config file into a docker container.

Describe the solution you'd like

I want to explore adding a --config option to the bentoml containerize command to build a config file into a docker image during the bento containerization phase.

Currently, the image is built from the bentoml/model-server base image specified by the Dockerfile stored in the bento bundle. If the --config option is specified in the containerization command, I am thinking of building a new image layer on top of the image built from containerization, to include the specified configuration file and environment variable. The Dockerfile will probably look as simple as below.

FROM bento_image
COPY --chown=bentoml:bentoml [config_file] ./bentoml.yml
ENV BENTOML_CONFIG ./bentoml.yml

[yubozhao]

For other CLI commands, the --config option will be used to load a different set of BentoML configuration for that command run, right?

Maybe we can be more explicit with the option name, like --config-file-for-docker-image or something.

[ssheng]

No, we don't have the --config option for other CLI commands anymore. Config file is set with environment variable only now.

[parano]

@ssheng I slightly prefer to allow users to set a default config file in Bento bundle during "save" and allow override in runtime, e.g. via docker run -e .. -v .., let's discuss more

[ssheng]

@parano What you suggested was the first implementation I considered, however, it has a limitation. When a user saves a config file in the bento bundle. They will expect the config file to be applied when the bundle is served during bentoml serve or docker run. The configuration is overridden during BentoML initialization, we don't know at that time what bundle, or if any bundle, is going to be loaded.

[ssheng]

After discussing with the BentoML team, the final decision was to not add the containerization feature specifically for configuration. The vision is to manage BentoML configuration separately from the bento bundles. Coupling should not happen until the deployment time which implies that the configuration should not be containerized into the same image as the bento bundle. The team envisions that the Yatai service will be responsible for managing configuration for deployment.

[TheHeadlessHydra]

Our organization has begun using bentoML for production model serving, and this feature is actually a pretty big pain point for us.

I can understand wanting to keep configuration separate from bundling, but this separation does not work well in containerization frameworks like kubernetes. In our case we run a custom containerization framework similar to kubernetes, but the same problem arises for both.

We can't mount files or expect files to exist or get passed to a container at the run time of a container. Environment variables? Yes. Entire files? No. Containers are expected to include all the files they need to run, so a feature like that is not common for containerization frameworks (especially given containers can run on very different hardware on each run and get moved around a lot). For containerization frameworks, we need all files bundled into the image.

We are investigating working around this with multi-step dockerfiles, or creating a base docker container from the bundled bentoML container, but all of them would be workarounds. Preferably, there is a way to pass a file into the bundle step so that the container includes the file before being run.

If you really want to keep configurations separate from bundling, offering an alternative approach via environment variables would be acceptable as well, as most containerization farmeworks work well with that.