search

bentoml / BentoML

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!
https://bentoml.com
Apache License 2.0
7.18k stars 792 forks source link

bug: `bentoml build ... && bentoml containerize ...` fails when using a Dockerfile template #3439

Closed phitoduck closed 1 year ago

phitoduck commented 1 year ago

Describe the bug

bentoml containerize ... throws a FileNotFound error when trying to find the Jinja template Dockerfile added to the bento when running bentoml build ....

It looks like the bentofile.docker.dockerfile_template option has a bug where bentoml containerize ... expects the file to be in a different place than bentoml build ... places it.

$ BUILD_CONTEXT="./api/" \
    BENTOML_DEBUG=TRUE bentoml build --bentofile ./api/bentofile.yaml $BUILD_CONTEXT \
   && \
   DOCKER_BUILDKIT=1 \
   BENTOML_DEBUG=TRUE \
   bentoml containerize $BENTO_SERVICE_NAME:$BENTO_SERVICE_TAG \
        --docker-image-tag $BENTO_SERVICE_NAME:#BENTO_SERVICE_TAG

BentoML will not install Python to custom base images; ensure the base image '763104351884.dkr.ecr.us-east-1.amazonaws.com/tensorflow-training:2.10.0-cpu-py39-ubuntu20.04-ec2' has Python installed.

██████╗░███████╗███╗░░██╗████████╗░█████╗░███╗░░░███╗██╗░░░░░
██╔══██╗██╔════╝████╗░██║╚══██╔══╝██╔══██╗████╗░████║██║░░░░░
██████╦╝█████╗░░██╔██╗██║░░░██║░░░██║░░██║██╔████╔██║██║░░░░░
██╔══██╗██╔══╝░░██║╚████║░░░██║░░░██║░░██║██║╚██╔╝██║██║░░░░░
██████╦╝███████╗██║░╚███║░░░██║░░░╚█████╔╝██║░╚═╝░██║███████╗
╚═════╝░╚══════╝╚═╝░░╚══╝░░░╚═╝░░░░╚════╝░╚═╝░░░░░╚═╝╚══════╝

Successfully built Bento(tag="keras-ctr-service:sve4knewvoabllg6").

+ DOCKER_BUILDKIT=1
+ BENTOML_DEBUG=TRUE
+ bentoml containerize keras-ctr-service:latest --docker-image-tag keras-ctr-service:latest
Building OCI-compliant image for keras-ctr-service:sve4knewvoabllg6 with docker

The path is /var/folders/cn/6x3hstc532b_gdnvrvn0p7600000gp/T/tmpezd_ib5mfsTempFS/template.Dockerfile
Traceback (most recent call last):
  File "/Users/eric/repos/ben/ctr-thumbnail-heatmap-api/venv/bin/bentoml", line 8, in <module>
    sys.exit(cli())
  File "/Users/eric/repos/ben/ctr-thumbnail-heatmap-api/venv/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/eric/repos/ben/ctr-thumbnail-heatmap-api/venv/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/eric/repos/ben/ctr-thumbnail-heatmap-api/venv/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/eric/repos/ben/ctr-thumbnail-heatmap-api/venv/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/eric/repos/ben/ctr-thumbnail-heatmap-api/venv/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/eric/repos/ben/ctr-thumbnail-heatmap-api/venv/lib/python3.8/site-packages/bentoml_cli/utils.py", line 268, in wrapper
    return func(*args, **kwargs)
  File "/Users/eric/repos/ben/ctr-thumbnail-heatmap-api/venv/lib/python3.8/site-packages/bentoml_cli/utils.py", line 239, in wrapper
    return_value = func(*args, **kwargs)
  File "/Users/eric/repos/ben/ctr-thumbnail-heatmap-api/venv/lib/python3.8/site-packages/bentoml_cli/utils.py", line 196, in wrapper
    return func(*args, **kwargs)
  File "/Users/eric/repos/ben/ctr-thumbnail-heatmap-api/venv/lib/python3.8/site-packages/bentoml_cli/utils.py", line 55, in wrapper
    return _f(*args, **transformed)
  File "/Users/eric/repos/ben/ctr-thumbnail-heatmap-api/venv/lib/python3.8/site-packages/bentoml_cli/containerize.py", line 561, in containerize
    result = container.build(
  File "/Users/eric/repos/ben/ctr-thumbnail-heatmap-api/venv/lib/python3.8/site-packages/simple_di/__init__.py", line 139, in _
    return func(*_inject_args(bind.args), **_inject_kwargs(bind.kwargs))
  File "/Users/eric/repos/ben/ctr-thumbnail-heatmap-api/venv/lib/python3.8/site-packages/bentoml/container.py", line 411, in build
    return _internal_build(bento_tag, backend, features=features, **kwargs)
  File "/Users/eric/repos/ben/ctr-thumbnail-heatmap-api/venv/lib/python3.8/site-packages/simple_di/__init__.py", line 139, in _
    return func(*_inject_args(bind.args), **_inject_kwargs(bind.kwargs))
  File "/Users/eric/repos/ben/ctr-thumbnail-heatmap-api/venv/lib/python3.8/site-packages/bentoml/_internal/container/__init__.py", line 194, in build
    context_path, dockerfile = clean_context.enter_context(
  File "/Users/eric/.pyenv/versions/3.8.4/lib/python3.8/contextlib.py", line 425, in enter_context
    result = _cm_type.__enter__(cm)
  File "/Users/eric/.pyenv/versions/3.8.4/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/Users/eric/repos/ben/ctr-thumbnail-heatmap-api/venv/lib/python3.8/site-packages/bentoml/_internal/container/__init__.py", line 156, in construct_containerfile
    dockerfile = generate_containerfile(
  File "/Users/eric/repos/ben/ctr-thumbnail-heatmap-api/venv/lib/python3.8/site-packages/bentoml/_internal/container/generate.py", line 178, in generate_containerfile
    os.path.dirname(resolve_user_filepath(user_templates, build_ctx))
  File "/Users/eric/repos/ben/ctr-thumbnail-heatmap-api/venv/lib/python3.8/site-packages/bentoml/_internal/utils/__init__.py", line 312, in resolve_user_filepath
    raise FileNotFoundError(f"file {filepath} not found")
FileNotFoundError: file template.Dockerfile not found

The resulting bento created by the successfully run bento build ... command looks like this:

/Users/eric/bentoml/bentos/keras-ctr-service/sve4knewvoabllg6
├── README.md
├── apis
│   └── openapi.yaml
├── bento.yaml
├── env
│   ├── docker
│   │   ├── Dockerfile
│   │   └── entrypoint.sh
│   └── python
│       ├── install.sh
│       ├── requirements.lock.txt
│       ├── requirements.txt
│       └── version.txt
├── models
│   └── tensorflow_ctr_keras
│       ├── cw6ffrsqt23kwasc
│       └── latest
└── src
    ├── bentofile.yaml
    ├── docker-compose.yml
    ├── locustfile.py
    ├── pyproject.toml
    ├── setup.cfg
    ├── setup.py
    ├── src
    │   └── api
    └── template.Dockerfile

To reproduce

Our file structure

api/
├── bentofile.yaml
├── locustfile.py
├── pyproject.toml
├── setup.cfg
├── setup.py
├── src
│   ├── api
│   │   ├── __init__.py
│   │   ├── other modules ...
│   │   └── service.py
└── template.Dockerfile
# bentofile.yaml

service: api.service:svc

exclude:
  - ".gitignore"
  - "*.egg-info"
  - "*cache*"

python:
  # other packages are listed in setup.cfg, installed using the rendered template.Dockerfile (below)
  packages:
    - requests

docker:
  dockerfile_template: template.Dockerfile
  base_image: 763104351884.dkr.ecr.us-east-1.amazonaws.com/tensorflow-training:2.10.0-cpu-py39-ubuntu20.04-ec2
# template.Dockerfile

{% extends bento_base_template %}
{% block SETUP_BENTO_COMPONENTS %}
{{ super() }}

# WORKDIR is set to /home/bento/bentoml/src/

# requirements for openCV
RUN apt-get update && apt-get install ffmpeg libsm6 libxext6 -y

# install the API package
RUN --mount=type=cache,target=/root/.cache/pip cd ./src/ && pip install --editable ./

{% endblock %}

Expected behavior

Running bentoml build ... && bentoml containerize ... should successfully build a bento and docker image.

Environment

Environment variable

BENTOML_DEBUG=''
BENTOML_QUIET=''
BENTOML_BUNDLE_LOCAL_BUILD=''
BENTOML_DO_NOT_TRACK=''
BENTOML_CONFIG=''
BENTOML_CONFIG_OPTIONS=''
BENTOML_PORT=''
BENTOML_HOST=''
BENTOML_API_WORKERS=''

System information

bentoml: 1.0.12 python: 3.8.4 platform: macOS-10.16-x86_64-i386-64bit uid_gid: 502:20

pip_packages
``` absl-py==1.4.0 aiohttp==3.8.3 aiosignal==1.3.1 altair==4.2.0 anyio==3.6.2 -e git+ssh://git@bitbucket.org/bengrouptech/ctr-thumbnail-heatmap-api.git@154f557bef8c977a91e875df401f189859c22d4a#egg=api&subdirectory=api appdirs==1.4.4 appnope==0.1.3 argon2-cffi==21.3.0 argon2-cffi-bindings==21.2.0 arrow==1.2.3 asgiref==3.6.0 asttokens==2.2.1 astunparse==1.6.3 async-timeout==4.0.2 attrs==22.2.0 aws-cdk-lib==2.60.0 aws-cdk.asset-awscli-v1==2.2.49 aws-cdk.asset-kubectl-v20==2.1.1 aws-cdk.asset-node-proxy-agent-v5==2.0.38 aws-cdk.aws-apigatewayv2-alpha==2.60.0a0 aws-cdk.aws-apigatewayv2-authorizers-alpha==2.60.0a0 aws-cdk.aws-apigatewayv2-integrations-alpha==2.60.0a0 aws-cdk.aws-lambda-python-alpha==2.60.0a0 Babel==2.11.0 backcall==0.2.0 backoff==2.2.1 backports.zoneinfo==0.2.1 beautifulsoup4==4.11.1 bentoml==1.0.12 bleach==5.0.1 blinker==1.5 boto3==1.26.48 botocore==1.29.48 Brotli==1.0.9 build==0.10.0 cachetools==5.2.1 cattrs==22.2.0 certifi==2022.12.7 cffi==1.15.1 charset-normalizer==3.0.1 circus==0.18.0 click==8.1.3 click-option-group==0.5.5 cloudpickle==2.2.0 comm==0.1.2 commonmark==0.9.1 ConfigArgParse==1.5.3 constructs==10.1.217 contextlib2==21.6.0 contourpy==1.0.6 cycler==0.11.0 debugpy==1.6.5 decorator==5.1.1 deepmerge==1.1.0 defusedxml==0.7.1 Deprecated==1.2.13 entrypoints==0.4 exceptiongroup==1.1.0 executing==1.2.0 fastjsonschema==2.16.2 Flask==2.2.2 Flask-BasicAuth==0.2.0 Flask-Cors==3.0.10 flatbuffers==23.1.4 fonttools==4.38.0 fqdn==1.5.1 frozenlist==1.3.3 fs==2.4.16 gast==0.4.0 gevent==22.10.2 geventhttpclient==2.0.8 gitdb==4.0.10 GitPython==3.1.30 google-auth==2.16.0 google-auth-oauthlib==0.4.6 google-pasta==0.2.0 googleapis-common-protos==1.58.0 greenlet==2.0.1 grpcio==1.51.1 h11==0.14.0 h5py==3.7.0 -e git+ssh://git@bitbucket.org/bengrouptech/ctr-thumbnail-heatmap-api.git@154f557bef8c977a91e875df401f189859c22d4a#egg=iac&subdirectory=iac idna==3.4 imageio==2.24.0 importlib-metadata==6.0.0 importlib-resources==5.10.2 ipykernel==6.20.1 ipython==8.8.0 ipython-genutils==0.2.0 isoduration==20.11.0 itsdangerous==2.1.2 jedi==0.18.2 Jinja2==3.1.2 jmespath==1.0.1 joblib==1.2.0 jsii==1.73.0 json5==0.9.11 jsonpointer==2.3 jsonschema==4.17.3 jupyter-events==0.6.3 jupyter_client==7.4.9 jupyter_core==5.1.3 jupyter_server==2.0.6 jupyter_server_terminals==0.4.4 jupyterlab==3.5.2 jupyterlab-pygments==0.2.2 jupyterlab_server==2.18.0 keras==2.10.0 Keras-Preprocessing==1.1.2 kiwisolver==1.4.4 libclang==15.0.6.1 locust==2.14.2 Markdown==3.4.1 MarkupSafe==2.1.1 matplotlib==3.3.1 matplotlib-inline==0.1.6 mistune==2.0.4 msgpack==1.0.4 multidict==6.0.4 mypy==0.782 mypy-extensions==0.4.3 nbclassic==0.4.8 nbclient==0.7.2 nbconvert==7.2.7 nbformat==5.7.3 nest-asyncio==1.5.6 networkx==3.0 notebook==6.5.2 notebook_shim==0.2.2 numpy==1.24.1 oauthlib==3.2.2 opencv-python==4.6.0.66 opencv-python-headless==4.7.0.68 opentelemetry-api==1.14.0 opentelemetry-exporter-otlp-proto-http==1.14.0 opentelemetry-instrumentation==0.35b0 opentelemetry-instrumentation-aiohttp-client==0.35b0 opentelemetry-instrumentation-asgi==0.35b0 opentelemetry-proto==1.14.0 opentelemetry-sdk==1.14.0 opentelemetry-semantic-conventions==0.35b0 opentelemetry-util-http==0.35b0 opt-einsum==3.3.0 packaging==23.0 pandas==1.5.2 pandocfilters==1.5.0 parso==0.8.3 pathspec==0.10.3 pexpect==4.8.0 phitoduck-projen==0.1.1 pickleshare==0.7.5 Pillow==9.4.0 pip-requirements-parser==32.0.1 pip-tools==6.12.1 pkgutil_resolve_name==1.3.10 platformdirs==2.6.2 projen==0.66.12 prometheus-client==0.15.0 prompt-toolkit==3.0.36 protobuf==3.19.6 psutil==5.9.4 ptyprocess==0.7.0 publication==0.0.3 pure-eval==0.2.2 pyarrow==10.0.1 pyasn1==0.4.8 pyasn1-modules==0.2.8 pycparser==2.21 pydantic==1.10.4 pydeck==0.8.0 Pygments==2.14.0 Pympler==1.0.1 pynvml==11.4.1 pyparsing==3.0.9 pyproject_hooks==1.0.0 pyrsistent==0.19.3 python-dateutil==2.8.2 python-dotenv==0.21.0 python-json-logger==2.0.4 python-multipart==0.0.5 pytz==2022.7 pytz-deprecation-shim==0.1.0.post0 PyWavelets==1.4.1 PyYAML==6.0 pyzmq==25.0.0 requests==2.28.2 requests-oauthlib==1.3.1 rfc3339-validator==0.1.4 rfc3986-validator==0.1.1 rich==13.0.1 roundrobin==0.0.4 rsa==4.9 s3transfer==0.6.0 schema==0.7.5 scikit-image==0.19.3 scikit-learn==1.2.0 scipy==1.10.0 seaborn==0.12.2 semver==2.13.0 Send2Trash==1.8.0 simple-di==0.1.5 six==1.16.0 smmap==5.0.0 sniffio==1.3.0 soupsieve==2.3.2.post1 stack-data==0.6.2 starlette==0.23.1 streamlit==1.16.0 tensorboard==2.10.1 tensorboard-data-server==0.6.1 tensorboard-plugin-wit==1.8.1 tensorflow==2.10.0 tensorflow-addons==0.19.0 tensorflow-estimator==2.10.0 tensorflow-io-gcs-filesystem==0.29.0 termcolor==2.2.0 terminado==0.17.1 tf-keras-vis==0.8.2 threadpoolctl==3.1.0 tifffile==2022.10.10 tinycss2==1.2.1 toml==0.10.2 tomli==2.0.1 toolz==0.12.0 tornado==6.2 traitlets==5.8.1 typed-ast==1.4.3 typeguard==2.13.3 typing==3.7.4.3 typing_extensions==4.4.0 tzdata==2022.7 tzlocal==4.2 uri-template==1.2.0 urllib3==1.26.14 uvicorn==0.20.0 validators==0.20.0 watchfiles==0.18.1 wcwidth==0.2.5 webcolors==1.12 webencodings==0.5.1 websocket-client==1.4.2 Werkzeug==2.2.2 wrapt==1.14.1 yarl==1.8.2 zipp==3.11.0 zope.event==4.6 zope.interface==5.5.2 ```
phitoduck commented 1 year ago

Interesting observation. I found a hacky way to get around this problem.

After running bentoml build ..., the bento is generated. I made the following change to the bento.yaml file (for me: /Users/eric/bentoml/bentos/keras-ctr-service/sve4knewvoabllg6/bento.yaml) after it was generated.

I changed 

bento.yaml

...
docker:
  ...
  base_image: 763104351884.dkr.ecr.us-east-1.amazonaws.com/tensorflow-training:2.10.0-cpu-py39-ubuntu20.04-ec2
  dockerfile_template: template.Dockerfile
...

to this

bento.yaml

...
docker:
  ...
  base_image: 763104351884.dkr.ecr.us-east-1.amazonaws.com/tensorflow-training:2.10.0-cpu-py39-ubuntu20.04-ec2
  dockerfile_template: src/template.Dockerfile
...

And suddenly bentoml containerize ... worked. This would be annoying to have to do every time we re-run bentoml build ...

sauyon commented 1 year ago

I believe this should be fixed by #3400; that should probably have had a better description.