search

zenml-io / zenml

ZenML 🙏: The bridge between ML and Ops. https://zenml.io.
https://zenml.io
Apache License 2.0
3.95k stars 430 forks source link

[BUG]: Orchestrator - connection error #1588

Closed adamwawrzynski closed 1 year ago

adamwawrzynski commented 1 year ago

Contact Details [Optional]

wawrzynski.adam@protonmail.com

System Information

ZENML_LOCAL_VERSION: 0.40.1 ZENML_SERVER_VERSION: 0.40.1 ZENML_SERVER_DATABASE: sqlite ZENML_SERVER_DEPLOYMENT_TYPE: local ZENML_CONFIG_DIR: /home/user/.config/zenml ZENML_LOCAL_STORE_DIR: /home/user/.config/zenml/local_stores ZENML_SERVER_URL: http://127.0.0.1:8237 ZENML_ACTIVE_REPOSITORY_ROOT: /home/user/praca/zenml_playground/zenml_examples/airflow_orchestration PYTHON_VERSION: 3.8.16 ENVIRONMENT: native SYSTEM_INFO: {'os': 'linux', 'linux_distro': 'ubuntu', 'linux_distro_like': 'debian', 'linux_distro_version': '22.04'} ACTIVE_WORKSPACE: default ACTIVE_STACK: local_airflow ACTIVE_USER: default TELEMETRY_STATUS: enabled ANALYTICS_CLIENT_ID: d5906bff-2e58-4a09-9b7b-20885e68b848 ANALYTICS_USER_ID: 8b7c567b-1293-4d86-8907-ffabc4acfa46 ANALYTICS_SERVER_ID: d5906bff-2e58-4a09-9b7b-20885e68b848 INTEGRATIONS: ['airflow', 'kaniko', 'mlflow', 'pillow', 'pytorch', 'scipy', 'sklearn', 'tensorboard', 'tensorflow'] PACKAGES: {'certifi': '2023.5.7', 'tzdata': '2023.3', 'pytz': '2022.7.1', 'pytzdata': '2020.1', 'setuptools': '65.6.3', 'cryptography': '41.0.1', 'pyzmq': '25.1.0', 'flatbuffers': '23.5.26', 'cattrs': '23.1.2', 'pip': '23.1.2', 'packaging': '23.1', 'aiofiles': '23.1.0', 'attrs': '23.1.0', 'gevent': '22.10.2', 'argon2-cffi': '21.3.0', 'argon2-cffi-bindings': '21.2.0', 'isoduration': '20.11.0', 'clickclick': '20.10.2', 'gunicorn': '20.1.0', 'lit': '16.0.5', 'libclang': '16.0.0', 'rich': '12.6.0', 'nvidia-cublas-cu11': '11.10.3.66', 'nvidia-cuda-cupti-cu11': '11.7.101', 'nvidia-cuda-runtime-cu11': '11.7.99', 'nvidia-cuda-nvrtc-cu11': '11.7.99', 'nvidia-nvtx-cu11': '11.7.91', 'nvidia-cusparse-cu11': '11.7.4.91', 'nvidia-cusolver-cu11': '11.4.0.1', 'websockets': '11.0.3', 'pyarrow': '11.0.0', 'nvidia-cufft-cu11': '10.9.0.58', 'nvidia-curand-cu11': '10.2.10.91', 'pillow': '9.5.0', 'ipython': '8.12.2', 'nvidia-cudnn-cu11': '8.5.0.96', 'tenacity': '8.2.2', 'jupyter-client': '8.2.0', 'click': '8.1.3', 'python-slugify': '8.0.1', 'ipywidgets': '7.7.5', 'nbconvert': '7.4.0', 'overrides': '7.3.1', 'ipykernel': '6.23.1', 'importlib-metadata': '6.6.0', 'notebook': '6.5.4', 'tornado': '6.3.2', 'docker': '6.1.3', 'multidict': '6.0.4', 'zope.interface': '6.0', 'bleach': '6.0.0', 'importlib-resources': '5.12.0', 'psutil': '5.9.5', 'nbformat': '5.9.0', 'traitlets': '5.9.0', 'pyyaml': '5.4.1', 'cachetools': '5.3.1', 'jupyter-core': '5.3.0', 'decorator': '5.1.1', 'smmap': '5.0.0', 'tqdm': '4.65.0', 'fonttools': '4.39.4', 'jsonschema': '4.17.3', 'beautifulsoup4': '4.12.2', 'rsa': '4.9', 'pexpect': '4.8.0', 'colorlog': '4.8.0', 'typing-extensions': '4.6.3', 'zope.event': '4.6', 'flask-jwt-extended': '4.5.2', 'apache-airflow-providers-http': '4.4.1', 'flask-appbuilder': '4.1.4', 'gitdb': '4.0.10', 'async-timeout': '4.0.2', 'bcrypt': '4.0.1', 'cmake': '3.26.3', 'protobuf': '3.20.3', 'marshmallow': '3.19.0', 'zipp': '3.15.0', 'filelock': '3.12.0', 'orjson': '3.8.14', 'aiohttp': '3.8.4', 'h5py': '3.8.0', 'asgiref': '3.7.2', 'matplotlib': '3.7.1', 'apache-airflow-providers-docker': '3.7.0', 'anyio': '3.7.0', 'widgetsnbextension': '3.6.4', 'platformdirs': '3.5.1', 'markdown': '3.4.3', 'apache-airflow-providers-ftp': '3.4.1', 'apache-airflow-providers-sqlite': '3.4.1', 'idna': '3.4', 'apispec': '3.3.2', 'python-jose': '3.3.0', 'opt-einsum': '3.3.0', 'oauthlib': '3.2.2', 'apache-airflow-providers-imap': '3.2.1', 'gitpython': '3.1.31', 'jinja2': '3.1.2', 'configupdater': '3.1.1', 'charset-normalizer': '3.1.0', 'threadpoolctl': '3.1.0', 'networkx': '3.1', 'prompt-toolkit': '3.0.38', 'argcomplete': '3.0.8', 'python-daemon': '3.0.1', 'wtforms': '3.0.1', 'marshmallow-oneofschema': '3.0.1', 'tritonclient': '2.34.0', 'requests': '2.31.0', 'pycparser': '2.21', 'google-auth': '2.19.1', 'fastjsonschema': '2.17.1', 'pygments': '2.15.1', 'nvidia-nccl-cu11': '2.14.3', 'connexion': '2.14.2', 'babel': '2.12.1', 'python-dateutil': '2.8.2', 'tensorflow': '2.8.0', 'keras': '2.8.0', 'tensorboard': '2.8.0', 'tf-estimator-nightly': '2.8.0.dev2021122109', 'pyjwt': '2.7.0', 'jupyter-server': '2.6.0', 'flask-sqlalchemy': '2.5.1', 'pyparsing': '2.4.7', 'apache-airflow': '2.4.3', 'soupsieve': '2.4.1', 'jsonpointer': '2.3', 'dnspython': '2.3.0', 'termcolor': '2.3.0', 'flask': '2.2.5', 'werkzeug': '2.2.3', 'mlflow': '2.2.2', 'asttokens': '2.2.1', 'cloudpickle': '2.2.1', 'markdown-it-py': '2.2.0', 'markupsafe': '2.1.2', 'pendulum': '2.1.2', 'itsdangerous': '2.1.2', 'python-json-logger': '2.0.7', 'mistune': '2.0.5', 'linkify-it-py': '2.0.2', 'greenlet': '2.0.2', 'geventhttpclient': '2.0.2', 'flask-caching': '2.0.2', 'kafka-python': '2.0.2', 'pandas': '2.0.2', 'torch': '2.0.1', 'triton': '2.0.0', 'flask-babel': '2.0.0', 'grpcio': '1.54.2', 'urllib3': '1.26.16', 'numpy': '1.24.3', 'six': '1.16.0', 'cffi': '1.15.1', 'wrapt': '1.15.0', 'webcolors': '1.13', 'sympy': '1.12', 'pydantic': '1.10.8', 'scipy': '1.10.1', 'backoff': '1.10.0', 'python-rapidjson': '1.10', 'yarl': '1.9.2', 'lazy-object-proxy': '1.9.0', 'send2trash': '1.8.2', 'tensorboard-plugin-wit': '1.8.1', 'alembic': '1.8.1', 'distro': '1.8.0', 'passlib': '1.7.4', 'debugpy': '1.6.7', 'astunparse': '1.6.3', 'blinker': '1.6.2', 'monotonic': '1.6', 'nest-asyncio': '1.5.6', 'websocket-client': '1.5.2', 'marshmallow-enum': '1.5.1', 'fqdn': '1.5.1', 'apache-airflow-providers-common-sql': '1.5.1', 'pandocfilters': '1.5.0', 'sqlalchemy': '1.4.41', 'kiwisolver': '1.4.4', 'analytics-python': '1.4.post1', 'absl-py': '1.4.0', 'cron-descriptor': '1.4.0', 'croniter': '1.3.15', 'pkgutil-resolve-name': '1.3.10', 'frozenlist': '1.3.3', 'mlserver': '1.3.3', 'mlserver-mlflow': '1.3.3', 'setproctitle': '1.3.2', 'email-validator': '1.3.1', 'requests-oauthlib': '1.3.1', 'aiosignal': '1.3.1', 'sniffio': '1.3.0', 'text-unidecode': '1.3', 'mpmath': '1.3.0', 'deprecated': '1.2.14', 'querystring-parser': '1.2.4', 'mako': '1.2.4', 'arrow': '1.2.3', 'scikit-learn': '1.2.2', 'tinycss2': '1.2.1', 'executing': '1.2.0', 'uri-template': '1.2.0', 'joblib': '1.2.0', 'jupyterlab-widgets': '1.1.4', 'keras-preprocessing': '1.1.2', 'flask-wtf': '1.1.1', 'exceptiongroup': '1.1.1', 'brotli': '1.0.9', 'contourpy': '1.0.7', 'pymysql': '1.0.3', 'uc-micro-py': '1.0.2', 'sqlalchemy-jsonfield': '1.0.1.post0', 'pluggy': '1.0.0', 'requests-toolbelt': '1.0.0', 'python-dotenv': '1.0.0', 'nbclassic': '1.0.0', 'fastapi': '0.89.1', 'numba': '0.57.0', 'shap': '0.41.0', 'zenml': '0.40.1', 'llvmlite': '0.40.0', 'wheel': '0.38.4', 'sqlalchemy-utils': '0.38.3', 'marshmallow-sqlalchemy': '0.26.1', 'httpx': '0.24.1', 'tensorflow-io-gcs-filesystem': '0.24.0', 'tensorflow-io': '0.24.0', 'starlette': '0.22.0', 'graphviz': '0.20.1', 'docutils': '0.20.1', 'pyrsistent': '0.19.3', 'httplib2': '0.19.1', 'validators': '0.18.2', 'jedi': '0.18.2', 'ecdsa': '0.18.0', 'databricks-cli': '0.17.7', 'uvicorn': '0.17.6', 'httpcore': '0.17.2', 'terminado': '0.17.1', 'prometheus-client': '0.17.0', 'uvloop': '0.17.0', 'starlette-exporter': '0.16.0', 'torchvision': '0.15.2', 'python-nvd3': '0.15.0', 'unicodecsv': '0.14.1', 'h11': '0.14.0', 'lockfile': '0.12.2', 'cycler': '0.11.0', 'python-terraform': '0.10.1', 'commonmark': '0.9.1', 'tabulate': '0.9.0', 'pathspec': '0.9.0', 'cachelib': '0.9.0', 'parso': '0.8.3', 'watchgod': '0.8.2', 'aiokafka': '0.8.1', 'nbclient': '0.8.0', 'pickleshare': '0.7.5', 'defusedxml': '0.7.1', 'ptyprocess': '0.7.0', 'py-grpc-prometheus': '0.7.0', 'jupyter-events': '0.6.3', 'stack-data': '0.6.2', 'flask-login': '0.6.2', 'tensorboard-data-server': '0.6.1', 'gast': '0.5.4', 'webencodings': '0.5.1', 'inflection': '0.5.1', 'flask-session': '0.5.0', 'httptools': '0.5.0', 'pyasn1': '0.5.0', 'google-auth-oauthlib': '0.4.6', 'colorama': '0.4.6', 'sqlparse': '0.4.4', 'jupyter-server-terminals': '0.4.4', 'entrypoints': '0.4', 'dill': '0.3.6', 'mdit-py-plugins': '0.3.5', 'pyasn1-modules': '0.3.0', 'click-params': '0.3.0', 'wcwidth': '0.2.6', 'notebook-shim': '0.2.3', 'jupyterlab-pygments': '0.2.2', 'pure-eval': '0.2.2', 'fastapi-utils': '0.2.1', 'prison': '0.2.1', 'google-pasta': '0.2.0', 'ipython-genutils': '0.2.0', 'backcall': '0.2.0', 'matplotlib-inline': '0.1.6', 'rfc3339-validator': '0.1.4', 'comm': '0.1.3', 'mdurl': '0.1.2', 'rfc3986-validator': '0.1.1', 'swagger-ui-bundle': '0.0.9', 'sqlmodel': '0.0.8', 'slicer': '0.0.7', 'python-multipart': '0.0.6', 'sqlalchemy2-stubs': '0.0.2a34'} The stack local_airflow contains components that require building Docker images. Older versions of ZenML always built these images locally, but since version 0.32.0 this behavior can be configured using the image_builder stack component. This stack will temporarily default to a local image builder that mirrors the previous behavior, but this will be removed in future versions of ZenML. Please add an image builder to this stack: zenml image-builder register ... zenml stack update fb4ba9a8-0e94-4f53-bc7c-797091f95543 -i

CURRENT STACK

Name: local_airflow ID: fb4ba9a8-0e94-4f53-bc7c-797091f95543 Shared: No User: default / 8b7c567b-1293-4d86-8907-ffabc4acfa46 Workspace: default / 7c8b8bb1-f3de-455b-8f22-f97e7db496a6

ORCHESTRATOR: local_airflow

Name: local_airflow ID: 85a80b05-fd05-494a-92eb-74acc1e2656d Type: orchestrator Flavor: airflow Configuration: {'dag_output_dir': None, 'dag_id': None, 'dag_tags': [], 'dag_args': {}, 'operator': 'airflow.providers.docker.operators.docker.DockerOperator', 'operator_args': {}, 'custom_dag_generator': None, 'local': True} Shared: No User: default / 8b7c567b-1293-4d86-8907-ffabc4acfa46 Workspace: default / 7c8b8bb1-f3de-455b-8f22-f97e7db496a6

ARTIFACT_STORE: default

Name: default ID: 5cbecc33-6a1b-425d-b104-b92c26131886 Type: artifact_store Flavor: local Configuration: {'path': ''} Shared: No User: default / 8b7c567b-1293-4d86-8907-ffabc4acfa46 Workspace: default / 7c8b8bb1-f3de-455b-8f22-f97e7db496a6

What happened?

I've tried to run airflow_orchestration example but in Airflow DAG logs I've got a connection error. The same happened when I tried to use LocalDockerOrchestrator with vanilla mlflow_deployment example. This might be problem with my Docker configuration.

Reproduction steps

  1. Install zenml==0.40.1
  2. Run zenml example run airflow_orchestration

Relevant log output

*** Reading local file: /home/user/.config/zenml/airflow/85a80b05-fd05-494a-92eb-74acc1e2656d/logs/dag_id=fashion_mnist_pipeline_default_2ea3e949/run_id=scheduled__2023-05-26T16:19:50.223877+00:00/task_id=importer_mnist/attempt=1.log
[2023-06-02, 16:19:52 UTC] {taskinstance.py:1165} INFO - Dependencies all met for <TaskInstance: fashion_mnist_pipeline_default_2ea3e949.importer_mnist scheduled__2023-05-26T16:19:50.223877+00:00 [queued]>
[2023-06-02, 16:19:52 UTC] {taskinstance.py:1165} INFO - Dependencies all met for <TaskInstance: fashion_mnist_pipeline_default_2ea3e949.importer_mnist scheduled__2023-05-26T16:19:50.223877+00:00 [queued]>
[2023-06-02, 16:19:52 UTC] {taskinstance.py:1362} INFO - 
--------------------------------------------------------------------------------
[2023-06-02, 16:19:52 UTC] {taskinstance.py:1363} INFO - Starting attempt 1 of 1
[2023-06-02, 16:19:52 UTC] {taskinstance.py:1364} INFO - 
--------------------------------------------------------------------------------
[2023-06-02, 16:19:52 UTC] {taskinstance.py:1383} INFO - Executing <Task(DockerOperator): importer_mnist> on 2023-05-26 16:19:50.223877+00:00
[2023-06-02, 16:19:52 UTC] {standard_task_runner.py:55} INFO - Started process 29753 to run task
[2023-06-02, 16:19:52 UTC] {standard_task_runner.py:82} INFO - Running: ['airflow', 'tasks', 'run', 'fashion_mnist_pipeline_default_2ea3e949', 'importer_mnist', 'scheduled__2023-05-26T16:19:50.223877+00:00', '--job-id', '3', '--raw', '--subdir', 'DAGS_FOLDER/fashion_mnist_pipeline_default_2ea3e949.zip/dag.py', '--cfg-path', '/tmp/tmp4apm_m9g']
[2023-06-02, 16:19:52 UTC] {standard_task_runner.py:83} INFO - Job 3: Subtask importer_mnist
[2023-06-02, 16:19:52 UTC] {task_command.py:376} INFO - Running <TaskInstance: fashion_mnist_pipeline_default_2ea3e949.importer_mnist scheduled__2023-05-26T16:19:50.223877+00:00 [running]> on host user-desktop
[2023-06-02, 16:19:52 UTC] {taskinstance.py:1590} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=fashion_mnist_pipeline_default_2ea3e949
AIRFLOW_CTX_TASK_ID=importer_mnist
AIRFLOW_CTX_EXECUTION_DATE=2023-05-26T16:19:50.223877+00:00
AIRFLOW_CTX_TRY_NUMBER=1
AIRFLOW_CTX_DAG_RUN_ID=scheduled__2023-05-26T16:19:50.223877+00:00
[2023-06-02, 16:19:52 UTC] {docker.py:334} INFO - Starting docker container from image zenml:fashion_mnist_pipeline-orchestrator
[2023-06-02, 16:19:55 UTC] {docker.py:403} INFO - ╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /usr/local/lib/python3.8/site-packages/urllib3/connection.py:200 in          │
│ _new_conn                                                                    │
│                                                                              │
│   197 │   │   :return: New socket connection.                                │
│   198 │   │   """                                                            │
│   199 │   │   try:                                                           │
│ ❱ 200 │   │   │   sock = connection.create_connection(                       │
│   201 │   │   │   │   (self._dns_host, self.port),                           │
│   202 │   │   │   │   self.timeout,                                          │
│   203 │   │   │   │   source_address=self.source_address,                    │
│                                                                              │
│ /usr/local/lib/python3.8/site-packages/urllib3/util/connection.py:85 in      │
│ create_connection                                                            │
│                                                                              │
│    82 │                                                                      │
│    83 │   if err is not None:                                                │
│    84 │   │   try:                                                           │
│ ❱  85 │   │   │   raise err                                                  │
│    86 │   │   finally:                                                       │
│    87 │   │   │   # Break explicitly a reference cycle                       │
│    88 │   │   │   err = None                                                 │
│                                                                              │
│ /usr/local/lib/python3.8/site-packages/urllib3/util/connection.py:73 in      │
│ create_connection                                                            │
│                                                                              │
│    70 │   │   │   │   sock.settimeout(timeout)                               │
│    71 │   │   │   if source_address:                                         │
│    72 │   │   │   │   sock.bind(source_address)                              │
│ ❱  73 │   │   │   sock.connect(sa)                                           │
│    74 │   │   │   # Break explicitly a reference cycle                       │
│    75 │   │   │   err = None                                                 │
│    76 │   │   │   return sock                                                │
╰──────────────────────────────────────────────────────────────────────────────╯
ConnectionRefusedError: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py:790 in      │
│ urlopen                                                                      │
│                                                                              │
│    787 │   │   │   response_conn = conn if not release_conn else None        │
│    788 │   │   │                                                             │
│    789 │   │   │   # Make the request on the HTTPConnection object           │
│ ❱  790 │   │   │   response = self._make_request(                            │
│    791 │   │   │   │   conn,                                                 │
│    792 │   │   │   │   method,                                               │
│    793 │   │   │   │   url,                                                  │
│                                                                              │
│ /usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py:496 in      │
│ _make_request                                                                │
│                                                                              │
│    493 │   │   # conn.request() calls http.client.*.request, not the method  │
│    494 │   │   # urllib3.request. It also calls makefile (recv) on the socke │
│    495 │   │   try:                                                          │
│ ❱  496 │   │   │   conn.request(                                             │
│    497 │   │   │   │   method,                                               │
│    498 │   │   │   │   url,                                                  │
│    499 │   │   │   │   body=body,                                            │
│                                                                              │
│ /usr/local/lib/python3.8/site-packages/urllib3/connection.py:388 in request  │
│                                                                              │
│   385 │   │   │   self.putheader("User-Agent", _get_default_user_agent())    │
│   386 │   │   for header, value in headers.items():                          │
│   387 │   │   │   self.putheader(header, value)                              │
│ ❱ 388 │   │   self.endheaders()                                              │
│   389 │   │                                                                  │
│   390 │   │   # If we're given a body we start sending that in chunks.       │
│   391 │   │   if chunks is not None:                                         │
│                                                                              │
│ /usr/local/lib/python3.8/http/client.py:1251 in endheaders                   │
│                                                                              │
│   1248 │   │   │   self.__state = _CS_REQ_SENT                               │
│   1249 │   │   else:                                                         │
│   1250 │   │   │   raise CannotSendHeader()                                  │
│ ❱ 1251 │   │   self._send_output(message_body, encode_chunked=encode_chunked │
│   1252 │                                                                     │
│   1253 │   def request(self, method, url, body=None, headers={}, *,          │
│   1254 │   │   │   │   encode_chunked=False):                                │
│                                                                              │
│ /usr/local/lib/python3.8/http/client.py:1011 in _send_output                 │
│                                                                              │
│   1008 │   │   self._buffer.extend((b"", b""))                               │
│   1009 │   │   msg = b"\r\n".join(self._buffer)                              │
│   1010 │   │   del self._buffer[:]                                           │
│ ❱ 1011 │   │   self.send(msg)                                                │
│   1012 │   │                                                                 │
│   1013 │   │   if message_body is not None:                                  │
│   1014                                                                       │
│                                                                              │
│ /usr/local/lib/python3.8/http/client.py:951 in send                          │
│                                                                              │
│    948 │   │                                                                 │
│    949 │   │   if self.sock is None:                                         │
│    950 │   │   │   if self.auto_open:                                        │
│ ❱  951 │   │   │   │   self.connect()                                        │
│    952 │   │   │   else:                                                     │
│    953 │   │   │   │   raise NotConnected()                                  │
│    954                                                                       │
│                                                                              │
│ /usr/local/lib/python3.8/site-packages/urllib3/connection.py:236 in connect  │
│                                                                              │
│   233 │   │   self._tunnel_scheme = scheme                                   │
│   234 │                                                                      │
│   235 │   def connect(self) -> None:                                         │
│ ❱ 236 │   │   self.sock = self._new_conn()                                   │
│   237 │   │   if self._tunnel_host:                                          │
│   238 │   │   │   # If we're tunneling it means we're connected to our proxy │
│   239 │   │   │   self._has_connected_to_proxy = True                        │
│                                                                              │
│ /usr/local/lib/python3.8/site-packages/urllib3/connection.py:215 in          │
│ _new_conn                                                                    │
│                                                                              │
│   212 │   │   │   ) from e                                                   │
│   213 │   │                                                                  │
│   214 │   │   except OSError as e:                                           │
│ ❱ 215 │   │   │   raise NewConnectionError(                                  │
│   216 │   │   │   │   self, f"Failed to establish a new connection: {e}"     │
│   217 │   │   │   ) from e                                                   │
│   218                                                                        │
╰──────────────────────────────────────────────────────────────────────────────╯
NewConnectionError: <urllib3.connection.HTTPConnection object at 
0x7fe94612fca0>: Failed to establish a new connection: [Errno 111] Connection 
refused
...

Code of Conduct

AlexejPenner commented 1 year ago

Hey @adamwawrzynski, Sorry your Issue somehow got lost in our Inbox. Is this still an Issue that you are running into?

adamwawrzynski commented 1 year ago

@AlexejPenner Actually no, it works. I will have to check this issue and dig in. Maybe I've got cluttered virtualenv or mismatch of example and zenml versions.

adamwawrzynski commented 1 year ago

Couldn't reproduce. Closing.