search

zenml-io / zenml

ZenML 🙏: The bridge between ML and Ops. https://zenml.io.
https://zenml.io
Apache License 2.0
4.07k stars 437 forks source link

[BUG]: BentoML Transformer pipeline deployment failed #1592

Closed adamwawrzynski closed 1 year ago

adamwawrzynski commented 1 year ago

Contact Details [Optional]

wawrzynski.adam@protonmail.com

System Information

ZENML_LOCAL_VERSION: 0.40.1 ZENML_SERVER_VERSION: 0.40.1 ZENML_SERVER_DATABASE: sqlite ZENML_SERVER_DEPLOYMENT_TYPE: other ZENML_CONFIG_DIR: /home/user/.config/zenml ZENML_LOCAL_STORE_DIR: /home/user/.config/zenml/local_stores ZENML_SERVER_URL: sqlite:////home/user/.config/zenml/local_stores/default_zen_store/zenml.db ZENML_ACTIVE_REPOSITORY_ROOT: /home/user/praca/zenml_playground/zenml_examples/bentoml_deployment PYTHON_VERSION: 3.8.16 ENVIRONMENT: native SYSTEM_INFO: {'os': 'linux', 'linux_distro': 'ubuntu', 'linux_distro_like': 'debian', 'linux_distro_version': '22.04'} ACTIVE_WORKSPACE: default ACTIVE_STACK: local_bentoml_stack ACTIVE_USER: default TELEMETRY_STATUS: enabled ANALYTICS_CLIENT_ID: 59f35758-eb34-4b01-9210-8c674c63b351 ANALYTICS_USER_ID: cbefbd23-e385-4873-9193-a27b3bf38a34 ANALYTICS_SERVER_ID: 59f35758-eb34-4b01-9210-8c674c63b351 INTEGRATIONS: ['airflow', 'bentoml', 'huggingface', 'kaniko', 'mlflow', 'pillow', 'pytorch', 'scipy', 'sklearn', 'tensorboard', 'tensorflow'] PACKAGES: {'certifi': '2023.5.7', 'regex': '2023.5.5', 'fsspec': '2023.5.0', 'tzdata': '2023.3', 'pytz': '2022.7.1', 'pytzdata': '2020.1', 'setuptools': '65.6.3', 'cryptography': '41.0.1', 'pip-requirements-parser': '32.0.1', 'pyzmq': '25.1.0', 'flatbuffers': '23.5.26', 'cattrs': '23.1.2', 'pip': '23.1.2', 'packaging': '23.1', 'aiofiles': '23.1.0', 'attrs': '23.1.0', 'gevent': '22.10.2', 'contextlib2': '21.6.0', 'argon2-cffi': '21.3.0', 'argon2-cffi-bindings': '21.2.0', 'isoduration': '20.11.0', 'clickclick': '20.10.2', 'gunicorn': '20.1.0', 'lit': '16.0.5', 'libclang': '16.0.0', 'rich': '12.6.0', 'nvidia-cublas-cu11': '11.10.3.66', 'nvidia-cuda-cupti-cu11': '11.7.101', 'nvidia-cuda-runtime-cu11': '11.7.99', 'nvidia-cuda-nvrtc-cu11': '11.7.99', 'nvidia-nvtx-cu11': '11.7.91', 'nvidia-cusparse-cu11': '11.7.4.91', 'pynvml': '11.5.0', 'nvidia-cusolver-cu11': '11.4.0.1', 'websockets': '11.0.3', 'pyarrow': '11.0.0', 'nvidia-cufft-cu11': '10.9.0.58', 'nvidia-curand-cu11': '10.2.10.91', 'pillow': '9.5.0', 'ipython': '8.12.2', 'nvidia-cudnn-cu11': '8.5.0.96', 'tenacity': '8.2.2', 'jupyter-client': '8.2.0', 'click': '8.1.3', 'python-slugify': '8.0.1', 'ipywidgets': '7.7.5', 'nbconvert': '7.4.0', 'overrides': '7.3.1', 'ipykernel': '6.23.1', 'pip-tools': '6.13.0', 'importlib-metadata': '6.6.0', 'notebook': '6.5.4', 'tornado': '6.3.2', 'docker': '6.1.3', 'multidict': '6.0.4', 'zope.interface': '6.0', 'bleach': '6.0.0', 'importlib-resources': '5.12.0', 'psutil': '5.9.5', 'nbformat': '5.9.0', 'traitlets': '5.9.0', 'pyyaml': '5.4.1', 'cachetools': '5.3.1', 'jupyter-core': '5.3.0', 'decorator': '5.1.1', 'smmap': '5.0.0', 'tqdm': '4.65.0', 'fonttools': '4.39.4', 'transformers': '4.29.2', 'jsonschema': '4.17.3', 'beautifulsoup4': '4.12.2', 'rsa': '4.9', 'pexpect': '4.8.0', 'colorlog': '4.8.0', 'typing-extensions': '4.6.3', 'zope.event': '4.6', 'flask-jwt-extended': '4.5.2', 'apache-airflow-providers-http': '4.4.1', 'flask-appbuilder': '4.1.4', 'gitdb': '4.0.10', 'async-timeout': '4.0.2', 'bcrypt': '4.0.1', 'cmake': '3.26.3', 'protobuf': '3.20.3', 'marshmallow': '3.19.0', 'zipp': '3.15.0', 'filelock': '3.12.0', 'orjson': '3.8.14', 'aiohttp': '3.8.4', 'h5py': '3.8.0', 'asgiref': '3.7.2', 'matplotlib': '3.7.1', 'apache-airflow-providers-docker': '3.7.0', 'anyio': '3.7.0', 'widgetsnbextension': '3.6.4', 'platformdirs': '3.5.1', 'markdown': '3.4.3', 'apache-airflow-providers-ftp': '3.4.1', 'apache-airflow-providers-sqlite': '3.4.1', 'idna': '3.4', 'apispec': '3.3.2', 'python-jose': '3.3.0', 'opt-einsum': '3.3.0', 'oauthlib': '3.2.2', 'apache-airflow-providers-imap': '3.2.1', 'xxhash': '3.2.0', 'gitpython': '3.1.31', 'jinja2': '3.1.2', 'configupdater': '3.1.1', 'charset-normalizer': '3.1.0', 'threadpoolctl': '3.1.0', 'networkx': '3.1', 'prompt-toolkit': '3.0.38', 'argcomplete': '3.0.8', 'python-daemon': '3.0.1', 'wtforms': '3.0.1', 'marshmallow-oneofschema': '3.0.1', 'tritonclient': '2.34.0', 'requests': '2.31.0', 'pycparser': '2.21', 'google-auth': '2.19.1', 'fastjsonschema': '2.17.1', 'pygments': '2.15.1', 'nvidia-nccl-cu11': '2.14.3', 'connexion': '2.14.2', 'babel': '2.12.1', 'datasets': '2.12.0', 'python-dateutil': '2.8.2', 'tensorflow': '2.8.0', 'keras': '2.8.0', 'tensorboard': '2.8.0', 'tf-estimator-nightly': '2.8.0.dev2021122109', 'pyjwt': '2.7.0', 'jupyter-server': '2.6.0', 'flask-sqlalchemy': '2.5.1', 'fs': '2.4.16', 'pyparsing': '2.4.7', 'apache-airflow': '2.4.3', 'soupsieve': '2.4.1', 'jsonpointer': '2.3', 'dnspython': '2.3.0', 'termcolor': '2.3.0', 'flask': '2.2.5', 'werkzeug': '2.2.3', 'mlflow': '2.2.2', 'asttokens': '2.2.1', 'cloudpickle': '2.2.1', 'markdown-it-py': '2.2.0', 'markupsafe': '2.1.2', 'pendulum': '2.1.2', 'itsdangerous': '2.1.2', 'python-json-logger': '2.0.7', 'mistune': '2.0.5', 'linkify-it-py': '2.0.2', 'greenlet': '2.0.2', 'geventhttpclient': '2.0.2', 'flask-caching': '2.0.2', 'kafka-python': '2.0.2', 'tomli': '2.0.1', 'torch': '2.0.1', 'triton': '2.0.0', 'flask-babel': '2.0.0', 'grpcio': '1.54.2', 'urllib3': '1.26.16', 'numpy': '1.24.3', 'six': '1.16.0', 'cffi': '1.15.1', 'wrapt': '1.15.0', 'opentelemetry-api': '1.14.0', 'opentelemetry-sdk': '1.14.0', 'webcolors': '1.13', 'sympy': '1.12', 'pydantic': '1.10.8', 'scipy': '1.10.1', 'backoff': '1.10.0', 'python-rapidjson': '1.10', 'yarl': '1.9.2', 'lazy-object-proxy': '1.9.0', 'send2trash': '1.8.2', 'tensorboard-plugin-wit': '1.8.1', 'alembic': '1.8.1', 'distro': '1.8.0', 'passlib': '1.7.4', 'debugpy': '1.6.7', 'astunparse': '1.6.3', 'blinker': '1.6.2', 'monotonic': '1.6', 'nest-asyncio': '1.5.6', 'pandas': '1.5.3', 'websocket-client': '1.5.2', 'marshmallow-enum': '1.5.1', 'fqdn': '1.5.1', 'apache-airflow-providers-common-sql': '1.5.1', 'pandocfilters': '1.5.0', 'sqlalchemy': '1.4.41', 'appdirs': '1.4.4', 'kiwisolver': '1.4.4', 'analytics-python': '1.4.post1', 'absl-py': '1.4.0', 'cron-descriptor': '1.4.0', 'croniter': '1.3.15', 'pkgutil-resolve-name': '1.3.10', 'frozenlist': '1.3.3', 'mlserver': '1.3.3', 'mlserver-mlflow': '1.3.3', 'setproctitle': '1.3.2', 'email-validator': '1.3.1', 'requests-oauthlib': '1.3.1', 'aiosignal': '1.3.1', 'sniffio': '1.3.0', 'text-unidecode': '1.3', 'mpmath': '1.3.0', 'deprecated': '1.2.14', 'querystring-parser': '1.2.4', 'mako': '1.2.4', 'arrow': '1.2.3', 'scikit-learn': '1.2.2', 'tinycss2': '1.2.1', 'executing': '1.2.0', 'uri-template': '1.2.0', 'joblib': '1.2.0', 'jupyterlab-widgets': '1.1.4', 'keras-preprocessing': '1.1.2', 'flask-wtf': '1.1.1', 'exceptiongroup': '1.1.1', 'deepmerge': '1.1.0', 'bentoml': '1.0.20', 'brotli': '1.0.9', 'contourpy': '1.0.7', 'pymysql': '1.0.3', 'uc-micro-py': '1.0.2', 'sqlalchemy-jsonfield': '1.0.1.post0', 'xformer': '1.0.1', 'mypy-extensions': '1.0.0', 'pluggy': '1.0.0', 'requests-toolbelt': '1.0.0', 'python-dotenv': '1.0.0', 'pyproject-hooks': '1.0.0', 'nbclassic': '1.0.0', 'fastapi': '0.75.2', 'multiprocess': '0.70.14', 'numba': '0.57.0', 'shap': '0.41.0', 'zenml': '0.40.1', 'llvmlite': '0.40.0', 'wheel': '0.38.4', 'sqlalchemy-utils': '0.38.3', 'opentelemetry-instrumentation': '0.35b0', 'opentelemetry-semantic-conventions': '0.35b0', 'opentelemetry-instrumentation-aiohttp-client': '0.35b0', 'opentelemetry-instrumentation-asgi': '0.35b0', 'opentelemetry-util-http': '0.35b0', 'marshmallow-sqlalchemy': '0.26.1', 'httpx': '0.24.1', 'tensorflow-io-gcs-filesystem': '0.24.0', 'tensorflow-io': '0.24.0', 'graphviz': '0.20.1', 'docutils': '0.20.1', 'pyrsistent': '0.19.3', 'httplib2': '0.19.1', 'accelerate': '0.19.0', 'watchfiles': '0.19.0', 'validators': '0.18.2', 'jedi': '0.18.2', 'ecdsa': '0.18.0', 'responses': '0.18.0', 'circus': '0.18.0', 'databricks-cli': '0.17.7', 'uvicorn': '0.17.6', 'httpcore': '0.17.2', 'starlette': '0.17.1', 'terminado': '0.17.1', 'prometheus-client': '0.17.0', 'uvloop': '0.17.0', 'starlette-exporter': '0.16.0', 'torchvision': '0.15.2', 'huggingface-hub': '0.15.1', 'python-nvd3': '0.15.0', 'unicodecsv': '0.14.1', 'h11': '0.14.0', 'tokenizers': '0.13.3', 'lockfile': '0.12.2', 'cycler': '0.11.0', 'python-terraform': '0.10.1', 'build': '0.10.0', 'commonmark': '0.9.1', 'typing-inspect': '0.9.0', 'tabulate': '0.9.0', 'pathspec': '0.9.0', 'cachelib': '0.9.0', 'parso': '0.8.3', 'watchgod': '0.8.2', 'aiokafka': '0.8.1', 'nbclient': '0.8.0', 'pickleshare': '0.7.5', 'schema': '0.7.5', 'defusedxml': '0.7.1', 'ptyprocess': '0.7.0', 'py-grpc-prometheus': '0.7.0', 'jupyter-events': '0.6.3', 'stack-data': '0.6.2', 'flask-login': '0.6.2', 'tensorboard-data-server': '0.6.1', 'click-option-group': '0.5.5', 'gast': '0.5.4', 'webencodings': '0.5.1', 'inflection': '0.5.1', 'flask-session': '0.5.0', 'httptools': '0.5.0', 'pyasn1': '0.5.0', 'google-auth-oauthlib': '0.4.6', 'colorama': '0.4.6', 'sqlparse': '0.4.4', 'jupyter-server-terminals': '0.4.4', 'entrypoints': '0.4', 'evaluate': '0.4.0', 'dill': '0.3.6', 'mdit-py-plugins': '0.3.5', 'pyasn1-modules': '0.3.0', 'click-params': '0.3.0', 'wcwidth': '0.2.6', 'notebook-shim': '0.2.3', 'jupyterlab-pygments': '0.2.2', 'pure-eval': '0.2.2', 'fastapi-utils': '0.2.1', 'prison': '0.2.1', 'google-pasta': '0.2.0', 'ipython-genutils': '0.2.0', 'backcall': '0.2.0', 'matplotlib-inline': '0.1.6', 'simple-di': '0.1.5', 'rfc3339-validator': '0.1.4', 'comm': '0.1.3', 'mdurl': '0.1.2', 'rfc3986-validator': '0.1.1', 'pyre-extensions': '0.0.29', 'xformers': '0.0.20', 'swagger-ui-bundle': '0.0.9', 'sqlmodel': '0.0.8', 'slicer': '0.0.7', 'python-multipart': '0.0.6', 'sqlalchemy2-stubs': '0.0.2a34'} The stack local_bentoml_stack contains components that require building Docker images. Older versions of ZenML always built these images locally, but since version 0.32.0 this behavior can be configured using the image_builder stack component. This stack will temporarily default to a local image builder that mirrors the previous behavior, but this will be removed in future versions of ZenML. Please add an image builder to this stack: zenml image-builder register ... zenml stack update 1bfa7357-73fd-44a1-b0c2-a9feb8532ed8 -i

CURRENT STACK

Name: local_bentoml_stack ID: 1bfa7357-73fd-44a1-b0c2-a9feb8532ed8 Shared: No User: default / cbefbd23-e385-4873-9193-a27b3bf38a34 Workspace: default / d8f66bda-434c-4f5e-a9e5-6a820fa45a38

ORCHESTRATOR: default

Name: default ID: d4669c7f-a5c8-40e6-864c-d487b8e2edc8 Type: orchestrator Flavor: local Configuration: {} Shared: No User: default / cbefbd23-e385-4873-9193-a27b3bf38a34 Workspace: default / d8f66bda-434c-4f5e-a9e5-6a820fa45a38

ARTIFACT_STORE: default

Name: default ID: bea46b8e-cc73-43c3-b42e-af6aff195d42 Type: artifact_store Flavor: local Configuration: {'path': ''} Shared: No User: default / cbefbd23-e385-4873-9193-a27b3bf38a34 Workspace: default / d8f66bda-434c-4f5e-a9e5-6a820fa45a38

MODEL_DEPLOYER: bentoml_deployer

Name: bentoml_deployer ID: 806c2477-e505-480c-a4fc-e64ed626d194 Type: model_deployer Flavor: bentoml Configuration: {'service_path': ''} Shared: No User: default / cbefbd23-e385-4873-9193-a27b3bf38a34 Workspace: default / d8f66bda-434c-4f5e-a9e5-6a820fa45a38

What happened?

I've tried to deploy transformers pipeline to local BentoML deployment but I've got the error (error in Relevant log output). Content of log file:

$ cat /home/user/.config/zenml/local_stores/806c2477-e505-480c-a4fc-e64ed626d194/a7c73553-7e7d-439f-9c57-ded90067834a/service.log
Loading service daemon configuration from /home/user/.config/zenml/local_stores/806c2477-e505-480c-a4fc-e64ed626d194/a7c73553-7e7d-439f-9c57-ded90067834a/service.json
The @step decorator that you use to define your step is deprecated. Check out our docs https://docs.zenml.io for information on how to define steps in a more intuitive and flexible way!
The @step decorator that you use to define your step is deprecated. Check out our docs https://docs.zenml.io for information on how to define steps in a more intuitive and flexible way!
Starting BentoML prediction service as blocking process... press CTRL+C once to stop it.
Running in development mode.
WARNING:bentoml.serve:serve_http_development is deprecated. Please use serve_http_production with api_workers=1 and development_mode=True
INFO:bentoml.serve:Prometheus metrics for HTTP BentoServer from "gpt_pipeline_service:u27xjaqdpc2acbze" can be accessed at http://127.0.0.1:8001/metrics.
INFO:circus:Installing handle_callback_exception to loop
INFO:circus:Registering signals...
INFO:circus:Starting master on pid 14871
INFO:circus:sockets started
INFO:circus:Arbiter now waiting for commands
INFO:circus:api_server started
INFO:bentoml.serve:Starting production HTTP BentoServer from "gpt_pipeline_service:u27xjaqdpc2acbze" listening on http://127.0.0.1:8001 (Press CTRL+C to quit)

The weird thing is that model is correctly deployed and I can see hosted API description and make an inference while zenml model-deplyer models list gives this:

$ zenml model-deployer models list
Using the default local database.
Running with active workspace: 'default' (repository)
Running with active stack: 'local_bentoml_stack' (repository)
┏━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━┓
┃ STATUS │ UUID                                 │ PIPELINE_NAME          │ PIPELINE_STEP_NAME │ MODEL_NAME   ┃
┠────────┼──────────────────────────────────────┼────────────────────────┼────────────────────┼──────────────┨
┃   ⏸    │ a7c73553-7e7d-439f-9c57-ded90067834a │ training_fashion_mnist │ deployer           │ gpt_pipeline ┃
┗━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━┛

You can run this code by slightly modifying bentoml_deployer example. Below you can find modified script files. trainer.py:

#  Copyright (c) ZenML GmbH 2022. All Rights Reserved.
#
#  Licensed under the Apache License, Version 2.0 (the "License");
#  you may not use this file except in compliance with the License.
#  You may obtain a copy of the License at:
#
#       https://www.apache.org/licenses/LICENSE-2.0
#
#  Unless required by applicable law or agreed to in writing, software
#  distributed under the License is distributed on an "AS IS" BASIS,
#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
#  or implied. See the License for the specific language governing
#  permissions and limitations under the License.

import torch
from torch import nn
from torch.utils.data import DataLoader

from zenml.steps import step
import transformers
from transformers import pipeline

@step(enable_cache=True)
def trainer(train_dataloader: DataLoader) -> transformers.pipelines.text_generation.TextGenerationPipeline:
    """Trains on the train dataloader."""
    generator = pipeline('text-generation', model='gpt2')#, device=torch.device("cpu"))
    return generator

bento_builder.py:

#  Copyright (c) ZenML GmbH 2022. All Rights Reserved.
#
#  Licensed under the Apache License, Version 2.0 (the "License");
#  you may not use this file except in compliance with the License.
#  You may obtain a copy of the License at:
#
#       https://www.apache.org/licenses/LICENSE-2.0
#
#  Unless required by applicable law or agreed to in writing, software
#  distributed under the License is distributed on an "AS IS" BASIS,
#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
#  or implied. See the License for the specific language governing
#  permissions and limitations under the License.
from constants import MODEL_NAME

from zenml.integrations.bentoml.steps import (
    BentoMLBuilderParameters,
    bento_builder_step,
)

import os

import bentoml
from bentoml import bentos
from bentoml._internal.bento import bento

from zenml.integrations.bentoml.constants import DEFAULT_BENTO_FILENAME
from zenml.materializers import UnmaterializedArtifact
from zenml.steps import step
from zenml.steps.step_context import StepContext
from zenml.utils import source_utils
from zenml.utils.artifact_utils import load_artifact

@step
def transformer_pipeline_bento_builder_step(
    model: UnmaterializedArtifact,
    params: BentoMLBuilderParameters,
    context: StepContext,
) -> bento.Bento:
    """Build a BentoML Model and Bento bundle.

    This steps takes a model artifact of a trained or loaded ML model in a
    previous step and save it with BentoML, then build a BentoML bundle.

    Args:
        model: the model to be packaged.
        params: the parameters for the BentoML builder step.
        context: the step context.

    Returns:
        the BentoML Bento object.
    """
    # save the model and bento uri as part of the bento labels
    labels = params.labels or {}
    labels["model_uri"] = model.uri
    labels["bento_uri"] = os.path.join(
        context.get_output_artifact_uri(), DEFAULT_BENTO_FILENAME
    )

    # Load the model from the model artifact
    model = load_artifact(model)

    bentoml.transformers.save_model(name=params.model_name, pipeline=model)
    # Build the BentoML bundle
    bento = bentos.build(
        service=params.service,
        version=params.version,
        labels=labels,
        description=params.description,
        include=params.include,
        exclude=params.exclude,
        python=params.python,
        docker=params.docker,
        build_ctx=params.working_dir or source_utils.get_source_root(),
    )
    # Return the BentoML Bento bundle
    return bento

bento_builder = transformer_pipeline_bento_builder_step(
    params=BentoMLBuilderParameters(
        model_name=MODEL_NAME,
        model_type="pytorch",
        service="gpt_service.py:svc",
        exclude=["data"],
        python={
            "packages": ["zenml", "torch", "transformers"],
        },
    )
)

gpt_service.py:

import bentoml

from bentoml.io import Text, JSON
from constants import MODEL_NAME, SERVICE_NAME

runner = bentoml.transformers.get(f"{MODEL_NAME}:latest").to_runner()

svc = bentoml.Service(SERVICE_NAME, runners=[runner])

@svc.api(input=Text(), output=JSON())
async def unmask(input_series: str) -> list:
    result = await runner.async_run(input_series)
    return result

Reproduction steps

  1. pip install zenml==0.40.1 zenml[server]==0.40.1 torch transformers
  2. follow instruction from bentoml_deployment example
  3. TOKENIZERS_PARALLELISM=false python3 run.py --config deploy

Relevant log output

$ TOKENIZERS_PARALLELISM=false python3 run.py --config deploy
The @pipeline decorator that you use to define your pipeline is deprecated. Check out our docs https://docs.zenml.io for information on how to define pipelines in a more intuitive and flexible way!
The @pipeline decorator that you use to define your pipeline is deprecated. Check out our docs https://docs.zenml.io for information on how to define pipelines in a more intuitive and flexible way!
The @step decorator that you use to define your step is deprecated. Check out our docs https://docs.zenml.io for information on how to define steps in a more intuitive and flexible way!
The @step decorator that you use to define your step is deprecated. Check out our docs https://docs.zenml.io for information on how to define steps in a more intuitive and flexible way!
The @step decorator that you use to define your step is deprecated. Check out our docs https://docs.zenml.io for information on how to define steps in a more intuitive and flexible way!
The @step decorator that you use to define your step is deprecated. Check out our docs https://docs.zenml.io for information on how to define steps in a more intuitive and flexible way!
The @step decorator that you use to define your step is deprecated. Check out our docs https://docs.zenml.io for information on how to define steps in a more intuitive and flexible way!
The @step decorator that you use to define your step is deprecated. Check out our docs https://docs.zenml.io for information on how to define steps in a more intuitive and flexible way!
The @step decorator that you use to define your step is deprecated. Check out our docs https://docs.zenml.io for information on how to define steps in a more intuitive and flexible way!
The @step decorator that you use to define your step is deprecated. Check out our docs https://docs.zenml.io for information on how to define steps in a more intuitive and flexible way!
The @step decorator that you use to define your step is deprecated. Check out our docs https://docs.zenml.io for information on how to define steps in a more intuitive and flexible way!
The @step decorator that you use to define your step is deprecated. Check out our docs https://docs.zenml.io for information on how to define steps in a more intuitive and flexible way!
The stack local_bentoml_stack contains components that require building Docker images. Older versions of ZenML always built these images locally, but since version 0.32.0 this behavior can be configured using the image_builder stack component. This stack will temporarily default to a local image builder that mirrors the previous behavior, but this will be removed in future versions of ZenML. Please add an image builder to this stack:
zenml image-builder register <NAME> ...
zenml stack update 1bfa7357-73fd-44a1-b0c2-a9feb8532ed8 -i <NAME>
The BaseParameters class to define step parameters is deprecated. Check out our docs https://docs.zenml.io/user-guide/advanced-guide/configure-steps-pipelines for information on how to parameterize your steps. As a quick fix to get rid of this warning, make sure your parameter class inherits from pydantic.BaseModel instead of the BaseParameters class.
The BaseParameters class to define step parameters is deprecated. Check out our docs https://docs.zenml.io/user-guide/advanced-guide/configure-steps-pipelines for information on how to parameterize your steps. As a quick fix to get rid of this warning, make sure your parameter class inherits from pydantic.BaseModel instead of the BaseParameters class.
The BaseParameters class to define step parameters is deprecated. Check out our docs https://docs.zenml.io/user-guide/advanced-guide/configure-steps-pipelines for information on how to parameterize your steps. As a quick fix to get rid of this warning, make sure your parameter class inherits from pydantic.BaseModel instead of the BaseParameters class.
Registered pipeline training_fashion_mnist (version 6).
The stack local_bentoml_stack contains components that require building Docker images. Older versions of ZenML always built these images locally, but since version 0.32.0 this behavior can be configured using the image_builder stack component. This stack will temporarily default to a local image builder that mirrors the previous behavior, but this will be removed in future versions of ZenML. Please add an image builder to this stack:
zenml image-builder register <NAME> ...
zenml stack update 1bfa7357-73fd-44a1-b0c2-a9feb8532ed8 -i <NAME>
The stack local_bentoml_stack contains components that require building Docker images. Older versions of ZenML always built these images locally, but since version 0.32.0 this behavior can be configured using the image_builder stack component. This stack will temporarily default to a local image builder that mirrors the previous behavior, but this will be removed in future versions of ZenML. Please add an image builder to this stack:
zenml image-builder register <NAME> ...
zenml stack update 1bfa7357-73fd-44a1-b0c2-a9feb8532ed8 -i <NAME>
Running pipeline training_fashion_mnist on stack local_bentoml_stack (caching disabled)
The stack local_bentoml_stack contains components that require building Docker images. Older versions of ZenML always built these images locally, but since version 0.32.0 this behavior can be configured using the image_builder stack component. This stack will temporarily default to a local image builder that mirrors the previous behavior, but this will be removed in future versions of ZenML. Please add an image builder to this stack:
zenml image-builder register <NAME> ...
zenml stack update 1bfa7357-73fd-44a1-b0c2-a9feb8532ed8 -i <NAME>
Step importer has started.
Step importer has finished in 0.428s.
The stack local_bentoml_stack contains components that require building Docker images. Older versions of ZenML always built these images locally, but since version 0.32.0 this behavior can be configured using the image_builder stack component. This stack will temporarily default to a local image builder that mirrors the previous behavior, but this will be removed in future versions of ZenML. Please add an image builder to this stack:
zenml image-builder register <NAME> ...
zenml stack update 1bfa7357-73fd-44a1-b0c2-a9feb8532ed8 -i <NAME>
Step trainer has started.
No materializer is registered for type <class 'transformers.pipelines.text_generation.TextGenerationPipeline'>, so the default Pickle materializer was used. Pickle is not production ready and should only be used for prototyping as the artifacts cannot be loaded when running with a different Python version. Please consider implementing a custom materializer for type <class 'transformers.pipelines.text_generation.TextGenerationPipeline'> according to the instructions at https://docs.zenml.io/user-guide/advanced-guide/handle-custom-data-types.
Step trainer has finished in 2.763s.
The stack local_bentoml_stack contains components that require building Docker images. Older versions of ZenML always built these images locally, but since version 0.32.0 this behavior can be configured using the image_builder stack component. This stack will temporarily default to a local image builder that mirrors the previous behavior, but this will be removed in future versions of ZenML. Please add an image builder to this stack:
zenml image-builder register <NAME> ...
zenml stack update 1bfa7357-73fd-44a1-b0c2-a9feb8532ed8 -i <NAME>
Step bento_builder has started.
The stack local_bentoml_stack contains components that require building Docker images. Older versions of ZenML always built these images locally, but since version 0.32.0 this behavior can be configured using the image_builder stack component. This stack will temporarily default to a local image builder that mirrors the previous behavior, but this will be removed in future versions of ZenML. Please add an image builder to this stack:
zenml image-builder register <NAME> ...
zenml stack update 1bfa7357-73fd-44a1-b0c2-a9feb8532ed8 -i <NAME>
INFO:bentoml._internal.frameworks.transformers:Using the default model signature for Transformers ({'__call__': ModelSignature(batchable=False, batch_dim=(0, 0), input_spec=None, output_spec=None)}) for model "gpt_pipeline".
Step bento_builder has finished in 2m43s.
The stack local_bentoml_stack contains components that require building Docker images. Older versions of ZenML always built these images locally, but since version 0.32.0 this behavior can be configured using the image_builder stack component. This stack will temporarily default to a local image builder that mirrors the previous behavior, but this will be removed in future versions of ZenML. Please add an image builder to this stack:
zenml image-builder register <NAME> ...
zenml stack update 1bfa7357-73fd-44a1-b0c2-a9feb8532ed8 -i <NAME>
Step evaluator has started.
Step evaluator has finished in 0.403s.
The stack local_bentoml_stack contains components that require building Docker images. Older versions of ZenML always built these images locally, but since version 0.32.0 this behavior can be configured using the image_builder stack component. This stack will temporarily default to a local image builder that mirrors the previous behavior, but this will be removed in future versions of ZenML. Please add an image builder to this stack:
zenml image-builder register <NAME> ...
zenml stack update 1bfa7357-73fd-44a1-b0c2-a9feb8532ed8 -i <NAME>
Step deployment_trigger has started.
Step deployment_trigger has finished in 0.034s.
The stack local_bentoml_stack contains components that require building Docker images. Older versions of ZenML always built these images locally, but since version 0.32.0 this behavior can be configured using the image_builder stack component. This stack will temporarily default to a local image builder that mirrors the previous behavior, but this will be removed in future versions of ZenML. Please add an image builder to this stack:
zenml image-builder register <NAME> ...
zenml stack update 1bfa7357-73fd-44a1-b0c2-a9feb8532ed8 -i <NAME>
Step deployer has started.
The stack local_bentoml_stack contains components that require building Docker images. Older versions of ZenML always built these images locally, but since version 0.32.0 this behavior can be configured using the image_builder stack component. This stack will temporarily default to a local image builder that mirrors the previous behavior, but this will be removed in future versions of ZenML. Please add an image builder to this stack:
zenml image-builder register <NAME> ...
zenml stack update 1bfa7357-73fd-44a1-b0c2-a9feb8532ed8 -i <NAME>
Updating an existing BentoML deployment service: BentoMLDeploymentService[a7c73553-7e7d-439f-9c57-ded90067834a] (type: model-serving, flavor: bentoml)
⠼ Starting service 'BentoMLDeploymentService (type: model-serving, flavor: bentoml)'.
Timed out waiting for service BentoMLDeploymentService[a7c73553-7e7d-439f-9c57-ded90067834a] (type: model-serving, flavor: bentoml) to become active:
  Administrative state: active
  Operational state: inactive
  Last status message: 'service daemon is not running'
For more information on the service status, please see the following log file: /home/user/.config/zenml/local_stores/806c2477-e505-480c-a4fc-e64ed626d194/a7c73553-7e7d-439f-9c57-ded90067834a/service.log
Failed to run step deployer.
Pipeline run training_fashion_mnist-2023_06_05-08_11_57_162806 failed.
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /home/user/praca/zenml_playground/zenml_examples/bentoml_deployment/run.py:85 in <module>        │
│                                                                                                  │
│   82                                                                                             │
│   83                                                                                             │
│   84 if __name__ == "__main__":                                                                  │
│ ❱ 85 │   main()                                                                                  │
│   86                                                                                             │
│                                                                                                  │
│ /home/user/praca/zenml_playground/.venv/lib/python3.8/site-packages/click/core.py:1130 in        │
│ __call__                                                                                         │
│                                                                                                  │
│   1127 │                                                                                         │
│   1128 │   def __call__(self, *args: t.Any, **kwargs: t.Any) -> t.Any:                           │
│   1129 │   │   """Alias for :meth:`main`."""                                                     │
│ ❱ 1130 │   │   return self.main(*args, **kwargs)                                                 │
│   1131                                                                                           │
│   1132                                                                                           │
│   1133 class Command(BaseCommand):                                                               │
│                                                                                                  │
│ /home/user/praca/zenml_playground/.venv/lib/python3.8/site-packages/click/core.py:1055 in main   │
│                                                                                                  │
│   1052 │   │   try:                                                                              │
│   1053 │   │   │   try:                                                                          │
│   1054 │   │   │   │   with self.make_context(prog_name, args, **extra) as ctx:                  │
│ ❱ 1055 │   │   │   │   │   rv = self.invoke(ctx)                                                 │
│   1056 │   │   │   │   │   if not standalone_mode:                                               │
│   1057 │   │   │   │   │   │   return rv                                                         │
│   1058 │   │   │   │   │   # it's not safe to `ctx.exit(rv)` here!                               │
│                                                                                                  │
│ /home/user/praca/zenml_playground/.venv/lib/python3.8/site-packages/click/core.py:1404 in invoke │
│                                                                                                  │
│   1401 │   │   │   echo(style(message, fg="red"), err=True)                                      │
│   1402 │   │                                                                                     │
│   1403 │   │   if self.callback is not None:                                                     │
│ ❱ 1404 │   │   │   return ctx.invoke(self.callback, **ctx.params)                                │
│   1405 │                                                                                         │
│   1406 │   def shell_complete(self, ctx: Context, incomplete: str) -> t.List["CompletionItem"]:  │
│   1407 │   │   """Return a list of completions for the incomplete value. Looks                   │
│                                                                                                  │
│ /home/user/praca/zenml_playground/.venv/lib/python3.8/site-packages/click/core.py:760 in invoke  │
│                                                                                                  │
│    757 │   │                                                                                     │
│    758 │   │   with augment_usage_errors(__self):                                                │
│    759 │   │   │   with ctx:                                                                     │
│ ❱  760 │   │   │   │   return __callback(*args, **kwargs)                                        │
│    761 │                                                                                         │
│    762 │   def forward(                                                                          │
│    763 │   │   __self, __cmd: "Command", *args: t.Any, **kwargs: t.Any  # noqa: B902             │
│                                                                                                  │
│ /home/user/praca/zenml_playground/zenml_examples/bentoml_deployment/run.py:58 in main            │
│                                                                                                  │
│   55 │   predict = config == PREDICT or config == DEPLOY_AND_PREDICT                             │
│   56 │                                                                                           │
│   57 │   if deploy:                                                                              │
│ ❱ 58 │   │   training_fashion_mnist(                                                             │
│   59 │   │   │   importer=importer_mnist(),                                                      │
│   60 │   │   │   trainer=trainer(),                                                              │
│   61 │   │   │   evaluator=evaluator(),                                                          │
│                                                                                                  │
│ /home/user/praca/zenml_playground/.venv/lib/python3.8/site-packages/zenml/pipelines/base_pipelin │
│ e.py:184 in run                                                                                  │
│                                                                                                  │
│   181 │   │   )                                                                                  │
│   182 │   │   pipeline_copy._run_args.update(new_run_args)                                       │
│   183 │   │                                                                                      │
│ ❱ 184 │   │   pipeline_copy()                                                                    │
│   185 │                                                                                          │
│   186 │   def _compute_invocation_id(                                                            │
│   187 │   │   self,                                                                              │
│                                                                                                  │
│ /home/user/praca/zenml_playground/.venv/lib/python3.8/site-packages/zenml/new/pipelines/pipeline │
│ .py:1167 in __call__                                                                             │
│                                                                                                  │
│   1164 │   │   │   return self.entrypoint(*args, **kwargs)                                       │
│   1165 │   │                                                                                     │
│   1166 │   │   self.prepare(*args, **kwargs)                                                     │
│ ❱ 1167 │   │   return self._run(**self._run_args)                                                │
│   1168 │                                                                                         │
│   1169 │   def _call_entrypoint(self, *args: Any, **kwargs: Any) -> None:                        │
│   1170 │   │   """Calls the pipeline entrypoint function with the given arguments.               │
│                                                                                                  │
│ /home/user/praca/zenml_playground/.venv/lib/python3.8/site-packages/zenml/new/pipelines/pipeline │
│ .py:659 in _run                                                                                  │
│                                                                                                  │
│    656 │   │   │   # unexpected behavior                                                         │
│    657 │   │   │   constants.SHOULD_PREVENT_PIPELINE_EXECUTION = True                            │
│    658 │   │   │   try:                                                                          │
│ ❱  659 │   │   │   │   stack.deploy_pipeline(deployment=deployment_model)                        │
│    660 │   │   │   finally:                                                                      │
│    661 │   │   │   │   constants.SHOULD_PREVENT_PIPELINE_EXECUTION = False                       │
│    662                                                                                           │
│                                                                                                  │
│ /home/user/praca/zenml_playground/.venv/lib/python3.8/site-packages/zenml/stack/stack.py:864 in  │
│ deploy_pipeline                                                                                  │
│                                                                                                  │
│    861 │   │   Returns:                                                                          │
│    862 │   │   │   The return value of the call to `orchestrator.run_pipeline(...)`.             │
│    863 │   │   """                                                                               │
│ ❱  864 │   │   return self.orchestrator.run(deployment=deployment, stack=self)                   │
│    865 │                                                                                         │
│    866 │   def _get_active_components_for_step(                                                  │
│    867 │   │   self, step_config: "StepConfiguration"                                            │
│                                                                                                  │
│ /home/user/praca/zenml_playground/.venv/lib/python3.8/site-packages/zenml/orchestrators/base_orc │
│ hestrator.py:166 in run                                                                          │
│                                                                                                  │
│   163 │   │                                                                                      │
│   164 │   │   environment = get_config_environment_vars()                                        │
│   165 │   │   try:                                                                               │
│ ❱ 166 │   │   │   result = self.prepare_or_run_pipeline(                                         │
│   167 │   │   │   │   deployment=deployment, stack=stack, environment=environment                │
│   168 │   │   │   )                                                                              │
│   169 │   │   finally:                                                                           │
│                                                                                                  │
│ /home/user/praca/zenml_playground/.venv/lib/python3.8/site-packages/zenml/orchestrators/local/lo │
│ cal_orchestrator.py:82 in prepare_or_run_pipeline                                                │
│                                                                                                  │
│    79 │   │   │   │   │   step_name,                                                             │
│    80 │   │   │   │   )                                                                          │
│    81 │   │   │                                                                                  │
│ ❱  82 │   │   │   self.run_step(                                                                 │
│    83 │   │   │   │   step=step,                                                                 │
│    84 │   │   │   )                                                                              │
│    85                                                                                            │
│                                                                                                  │
│ /home/user/praca/zenml_playground/.venv/lib/python3.8/site-packages/zenml/orchestrators/base_orc │
│ hestrator.py:186 in run_step                                                                     │
│                                                                                                  │
│   183 │   │   │   step=step,                                                                     │
│   184 │   │   │   orchestrator_run_id=self.get_orchestrator_run_id(),                            │
│   185 │   │   )                                                                                  │
│ ❱ 186 │   │   launcher.launch()                                                                  │
│   187 │                                                                                          │
│   188 │   @staticmethod                                                                          │
│   189 │   def requires_resources_in_orchestration_environment(                                   │
│                                                                                                  │
│ /home/user/praca/zenml_playground/.venv/lib/python3.8/site-packages/zenml/orchestrators/step_lau │
│ ncher.py:176 in launch                                                                           │
│                                                                                                  │
│   173 │   │   │                                                                                  │
│   174 │   │   │   if execution_needed:                                                           │
│   175 │   │   │   │   try:                                                                       │
│ ❱ 176 │   │   │   │   │   self._run_step(                                                        │
│   177 │   │   │   │   │   │   pipeline_run=pipeline_run,                                         │
│   178 │   │   │   │   │   │   step_run=step_run_response,                                        │
│   179 │   │   │   │   │   )                                                                      │
│                                                                                                  │
│ /home/user/praca/zenml_playground/.venv/lib/python3.8/site-packages/zenml/orchestrators/step_lau │
│ ncher.py:354 in _run_step                                                                        │
│                                                                                                  │
│   351 │   │   │   │   │   step_run_info=step_run_info,                                           │
│   352 │   │   │   │   )                                                                          │
│   353 │   │   │   else:                                                                          │
│ ❱ 354 │   │   │   │   self._run_step_without_step_operator(                                      │
│   355 │   │   │   │   │   step_run_info=step_run_info,                                           │
│   356 │   │   │   │   │   input_artifacts=step_run.input_artifacts,                              │
│   357 │   │   │   │   │   output_artifact_uris=output_artifact_uris,                             │
│                                                                                                  │
│ /home/user/praca/zenml_playground/.venv/lib/python3.8/site-packages/zenml/orchestrators/step_lau │
│ ncher.py:424 in _run_step_without_step_operator                                                  │
│                                                                                                  │
│   421 │   │   │   output_artifact_uris: The output artifact URIs of the current step.            │
│   422 │   │   """                                                                                │
│   423 │   │   runner = StepRunner(step=self._step, stack=self._stack)                            │
│ ❱ 424 │   │   runner.run(                                                                        │
│   425 │   │   │   input_artifacts=input_artifacts,                                               │
│   426 │   │   │   output_artifact_uris=output_artifact_uris,                                     │
│   427 │   │   │   step_run_info=step_run_info,                                                   │
│                                                                                                  │
│ /home/user/praca/zenml_playground/.venv/lib/python3.8/site-packages/zenml/orchestrators/step_run │
│ ner.py:132 in run                                                                                │
│                                                                                                  │
│   129 │   │   │   self._stack.prepare_step_run(info=step_run_info)                               │
│   130 │   │   │   step_failed = False                                                            │
│   131 │   │   │   try:                                                                           │
│ ❱ 132 │   │   │   │   return_values = step_instance.call_entrypoint(                             │
│   133 │   │   │   │   │   **function_params                                                      │
│   134 │   │   │   │   )                                                                          │
│   135 │   │   │   except BaseException as step_exception:  # noqa: E722                          │
│                                                                                                  │
│ /home/user/praca/zenml_playground/.venv/lib/python3.8/site-packages/zenml/steps/base_step.py:582 │
│ in call_entrypoint                                                                               │
│                                                                                                  │
│    579 │   │   except ValidationError as e:                                                      │
│    580 │   │   │   raise StepInterfaceError("Invalid entrypoint arguments.") from e              │
│    581 │   │                                                                                     │
│ ❱  582 │   │   return self.entrypoint(**validated_args)                                          │
│    583 │                                                                                         │
│    584 │   @property                                                                             │
│    585 │   def name(self) -> str:                                                                │
│                                                                                                  │
│ /home/user/praca/zenml_playground/.venv/lib/python3.8/site-packages/zenml/integrations/bentoml/s │
│ teps/bentoml_deployer.py:164 in bentoml_model_deployer_step                                      │
│                                                                                                  │
│   161 │   # create a new model deployment and replace an old one if it exists                    │
│   162 │   new_service = cast(                                                                    │
│   163 │   │   BentoMLDeploymentService,                                                          │
│ ❱ 164 │   │   model_deployer.deploy_model(                                                       │
│   165 │   │   │   replace=True,                                                                  │
│   166 │   │   │   config=predictor_cfg,                                                          │
│   167 │   │   │   timeout=params.timeout,                                                        │
│                                                                                                  │
│ /home/user/praca/zenml_playground/.venv/lib/python3.8/site-packages/zenml/integrations/bentoml/m │
│ odel_deployers/bentoml_model_deployer.py:223 in deploy_model                                     │
│                                                                                                  │
│   220 │   │   │   config.root_runtime_path = self.local_path                                     │
│   221 │   │   │   service.stop(timeout=timeout, force=True)                                      │
│   222 │   │   │   service.update(config)                                                         │
│ ❱ 223 │   │   │   service.start(timeout=timeout)                                                 │
│   224 │   │   else:                                                                              │
│   225 │   │   │   # create a new BentoMLDeploymentService instance                               │
│   226 │   │   │   service = self._create_new_service(timeout, config)                            │
│                                                                                                  │
│ /home/user/praca/zenml_playground/.venv/lib/python3.8/site-packages/zenml/services/local/local_s │
│ ervice.py:444 in start                                                                           │
│                                                                                                  │
│   441 │   │   │   │   the service status.                                                        │
│   442 │   │   """                                                                                │
│   443 │   │   if not self.config.blocking:                                                       │
│ ❱ 444 │   │   │   super().start(timeout)                                                         │
│   445 │   │   else:                                                                              │
│   446 │   │   │   self.run()                                                                     │
│   447                                                                                            │
│                                                                                                  │
│ /home/user/praca/zenml_playground/.venv/lib/python3.8/site-packages/zenml/services/service.py:44 │
│ 0 in start                                                                                       │
│                                                                                                  │
│   437 │   │   │   self.provision()                                                               │
│   438 │   │   │   if timeout > 0:                                                                │
│   439 │   │   │   │   if not self.poll_service_status(timeout):                                  │
│ ❱ 440 │   │   │   │   │   raise RuntimeError(                                                    │
│   441 │   │   │   │   │   │   f"Failed to start service {self}\n"                                │
│   442 │   │   │   │   │   │   + self.get_service_status_message()                                │
│   443 │   │   │   │   │   )                                                                      │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
RuntimeError: Failed to start service BentoMLDeploymentService[a7c73553-7e7d-439f-9c57-ded90067834a] (type: model-serving, flavor: bentoml)
  Administrative state: `active`
  Operational state: `inactive`
  Last status message: 'service daemon is not running'
For more information on the service status, please see the following log file: 
/home/user/.config/zenml/local_stores/806c2477-e505-480c-a4fc-e64ed626d194/a7c73553-7e7d-439f-9c57-ded90067834a/service.log

Code of Conduct

strickvl commented 1 year ago

Thanks for the bug report. I've investigated this and we've figured out what's going wrong. I'll update once this has been fixed.

strickvl commented 1 year ago

This was fixed in #1647. Sorry for not updating here till now @adamwawrzynski !