search

zenml-io / zenml

ZenML 🙏: The bridge between ML and Ops. https://zenml.io.
https://zenml.io
Apache License 2.0
4.08k stars 442 forks source link

[BUG]: S3ArtifactStore.open() got an unexpected keyword argument 'name' #3219

Open clenico opened 2 hours ago

clenico commented 2 hours ago

Contact Details [Optional]

clement.nicolas.1996.pro@gmail.com

System Information

ZENML_LOCAL_VERSION: 0.70.0 ZENML_SERVER_VERSION: 0.70.0 ZENML_SERVER_DATABASE: mysql ZENML_SERVER_DEPLOYMENT_TYPE: other ZENML_CONFIG_DIR: /home/user/.config/zenml ZENML_LOCAL_STORE_DIR: /home/user/.config/zenml/local_stores ZENML_SERVER_URL: http://deepthought1.multitel.be:10539 ZENML_ACTIVE_REPOSITORY_ROOT: None PYTHON_VERSION: 3.10.13 ENVIRONMENT: native SYSTEM_INFO: {'os': 'linux', 'linux_distro': 'ubuntu', 'linux_distro_like': 'debian', 'linux_distro_version': '22.04'} ACTIVE_WORKSPACE: default ACTIVE_STACK: remote-k8s-3dplts ACTIVE_USER: nicolas TELEMETRY_STATUS: enabled ANALYTICS_CLIENT_ID: 1147c8ab-ff98-4c33-a8b9-0ca0393eeb67 ANALYTICS_USER_ID: 89c3cd0b-11d0-4715-ab32-532edaaecf69 ANALYTICS_SERVER_ID: 78da78ba-593a-482c-b798-668c5a3a5f0d INTEGRATIONS: ['airflow', 'bitbucket', 'numpy', 'pandas', 'kaniko', 'mlflow', 'pigeon', 'pillow', 'pytorch', 'pytorch_lightning', 'scipy', 'sklearn', 'tensorboard'] PACKAGES: {'argparse': '1.4.0', 'black': '24.10.0', 'boto3': '1.35.68', 'botocore': '1.35.68', 'durationpy': '0.9', 'fastapi': '0.109.2', 'importlib-resources': '5.13.0', 'kubernetes': '31.0.0', 'mlserver': '1.4.0', 'mlserver-mlflow': '1.6.1', 'mypy-extensions': '1.0.0', 'pathspec': '0.12.1', 'pydantic': '2.8.2', 'pydantic-core': '2.20.1', 'ranger-fm': '1.9.4', 's3fs': '0.4.2', 'tritonclient': '2.42.0', 'utils-3dplts': '2.0.0', 'zenml': '0.70.0', 'brotli': '1.0.9', 'deprecated': '1.2.14', 'gitpython': '3.1.43', 'jinja2': '3.1.2', 'mako': '1.3.6', 'markdown': '3.7', 'markupsafe': '2.1.3', 'pillow': '9.5.0', 'pymysql': '1.1.1', 'pysocks': '1.7.1', 'pyyaml': '6.0.1', 'pygments': '2.15.1', 'qtpy': '2.4.1', 'sqlalchemy': '2.0.36', 'sqlalchemy-utils': '0.41.2', 'send2trash': '1.8.3', 'absl-py': '2.1.0', 'aiobotocore': '2.15.2', 'aiofiles': '24.1.0', 'aiohappyeyeballs': '2.4.0', 'aiohttp': '3.10.5', 'aioitertools': '0.12.0', 'aiokafka': '0.11.0', 'aiosignal': '1.3.1', 'alabaster': '0.7.16', 'alembic': '1.8.1', 'alibi-detect': '0.12.0', 'annotated-types': '0.7.0', 'anyio': '4.4.0', 'archspec': '0.2.1', 'argon2-cffi': '23.1.0', 'argon2-cffi-bindings': '21.2.0', 'arrow': '1.3.0', 'asgiref': '3.8.1', 'astroid': '2.15.8', 'asttokens': '2.0.5', 'astunparse': '1.6.3', 'async-lru': '2.0.4', 'async-timeout': '4.0.3', 'attrs': '23.1.0', 'aws-profile-manager': '0.7.3', 'babel': '2.16.0', 'bcrypt': '4.0.1', 'beautifulsoup4': '4.12.2', 'bleach': '6.1.0', 'blinker': '1.8.2', 'boltons': '23.0.0', 'cachetools': '5.5.0', 'catalogue': '2.0.10', 'certifi': '2023.11.17', 'cffi': '1.16.0', 'chardet': '4.0.0', 'charset-normalizer': '2.0.4', 'click': '8.1.3', 'click-params': '0.3.0', 'cloudpickle': '2.2.1', 'comm': '0.2.2', 'compress-pickle': '2.1.0', 'conda': '23.9.0', 'conda-build': '3.28.4', 'conda-content-trust': '0.2.0', 'conda-index': '0.4.0', 'conda-libmamba-solver': '23.7.0', 'conda-package-handling': '2.2.0', 'conda-package-streaming': '0.9.0', 'configparser': '7.1.0', 'contourpy': '1.2.1', 'cryptography': '41.0.7', 'cycler': '0.12.1', 'data-quality': '1.0.0', 'databricks-sdk': '0.36.0', 'datasets': '2.14.4', 'debugpy': '1.8.5', 'decorator': '5.1.1', 'defusedxml': '0.7.1', 'dill': '0.3.7', 'distro': '1.8.0', 'dnspython': '2.5.0', 'docker': '7.1.0', 'docutils': '0.20.1', 'entrypoints': '0.4', 'exceptiongroup': '1.2.0', 'executing': '0.8.3', 'expecttest': '0.2.1', 'fastjsonschema': '2.20.0', 'filelock': '3.13.1', 'flask': '3.0.3', 'fonttools': '4.53.1', 'fqdn': '1.5.1', 'frozenlist': '1.4.1', 'fsspec': '2024.6.1', 'gevent': '24.10.3', 'geventhttpclient': '2.0.2', 'gitdb': '4.0.11', 'gmpy2': '2.1.2', 'google-auth': '2.34.0', 'google-auth-oauthlib': '1.0.0', 'googleapis-common-protos': '1.65.0', 'grad-cam': '1.4.8', 'graphene': '3.4', 'graphql-core': '3.2.5', 'graphql-relay': '3.2.0', 'greenlet': '3.1.1', 'grpcio': '1.65.5', 'gunicorn': '22.0.0', 'h11': '0.14.0', 'httpcore': '1.0.5', 'httplib2': '0.19.1', 'httpx': '0.27.0', 'huggingface-hub': '0.24.6', 'hypothesis': '6.97.3', 'idna': '3.4', 'imagecorruptions': '1.1.2', 'imageio': '2.35.1', 'imagesize': '1.4.1', 'importlib-metadata': '7.2.1', 'iniconfig': '2.0.0', 'ipykernel': '6.29.5', 'ipython': '8.20.0', 'ipywidgets': '8.1.3', 'isoduration': '20.11.0', 'isort': '5.13.2', 'itsdangerous': '2.2.0', 'jedi': '0.18.1', 'jmespath': '1.0.1', 'joblib': '1.4.2', 'json5': '0.9.25', 'jsonpatch': '1.32', 'jsonpointer': '2.1', 'jsonschema': '4.19.2', 'jsonschema-specifications': '2023.7.1', 'jupyter': '1.0.0', 'jupyter-client': '8.6.2', 'jupyter-console': '6.6.3', 'jupyter-core': '5.7.2', 'jupyter-events': '0.10.0', 'jupyter-lsp': '2.2.5', 'jupyter-server': '2.14.2', 'jupyter-server-terminals': '0.5.3', 'jupyterlab': '4.2.4', 'jupyterlab-pygments': '0.3.0', 'jupyterlab-server': '2.27.3', 'jupyterlab-widgets': '3.0.11', 'kiwisolver': '1.4.5', 'latexcodec': '3.0.0', 'lazy-loader': '0.4', 'lazy-object-proxy': '1.10.0', 'libarchive-c': '2.9', 'libmambapy': '1.5.3', 'lightning-utilities': '0.11.6', 'llvmlite': '0.39.1', 'lmdb': '1.4.1', 'lz4': '4.3.3', 'markdown-it-py': '3.0.0', 'matplotlib': '3.8.4', 'matplotlib-inline': '0.1.6', 'mccabe': '0.7.0', 'mdit-py-plugins': '0.4.1', 'mdurl': '0.1.2', 'menuinst': '2.0.1', 'mistune': '3.0.2', 'mkl-fft': '1.3.8', 'mkl-random': '1.2.4', 'mkl-service': '2.4.0', 'mlflow': '2.15.1', 'mlflow-skinny': '2.15.1', 'more-itertools': '10.1.0', 'mpmath': '1.3.0', 'multidict': '6.0.5', 'multiprocess': '0.70.15', 'myst-parser': '2.0.0', 'nbclient': '0.10.0', 'nbconvert': '7.16.4', 'nbformat': '5.10.4', 'nbsphinx': '0.9.3', 'nest-asyncio': '1.6.0', 'networkx': '3.1', 'nose2': '0.13.0', 'notebook': '7.2.1', 'notebook-shim': '0.2.4', 'numba': '0.56.4', 'numpy': '1.23.5', 'nvidia-cublas-cu12': '12.1.3.1', 'nvidia-cuda-cupti-cu12': '12.1.105', 'nvidia-cuda-nvrtc-cu12': '12.1.105', 'nvidia-cuda-runtime-cu12': '12.1.105', 'nvidia-cudnn-cu12': '8.9.2.26', 'nvidia-cufft-cu12': '11.0.2.54', 'nvidia-curand-cu12': '10.3.2.106', 'nvidia-cusolver-cu12': '11.4.5.107', 'nvidia-cusparse-cu12': '12.1.0.106', 'nvidia-nccl-cu12': '2.18.1', 'nvidia-nvjitlink-cu12': '12.6.20', 'nvidia-nvtx-cu12': '12.1.105', 'oauthlib': '3.2.2', 'opencv-python': '4.10.0.84', 'opentelemetry-api': '1.27.0', 'opentelemetry-exporter-otlp-proto-common': '1.27.0', 'opentelemetry-exporter-otlp-proto-grpc': '1.27.0', 'opentelemetry-instrumentation': '0.48b0', 'opentelemetry-instrumentation-asgi': '0.48b0', 'opentelemetry-instrumentation-fastapi': '0.48b0', 'opentelemetry-instrumentation-grpc': '0.48b0', 'opentelemetry-proto': '1.27.0', 'opentelemetry-sdk': '1.27.0', 'opentelemetry-semantic-conventions': '0.48b0', 'opentelemetry-util-http': '0.48b0', 'optree': '0.10.0', 'orjson': '3.10.10', 'overrides': '7.7.0', 'packaging': '24.1', 'pandas': '2.1.4', 'pandocfilters': '1.5.1', 'parso': '0.8.3', 'passlib': '1.7.4', 'patsy': '0.5.6', 'pbr': '6.0.0', 'pexpect': '4.8.0', 'phx-class-registry': '4.0.6', 'pip': '23.3.1', 'pkginfo': '1.9.6', 'platformdirs': '3.10.0', 'plotly': '5.14.1', 'pluggy': '1.0.0', 'prometheus-client': '0.20.0', 'prompt-toolkit': '3.0.43', 'protobuf': '4.25.5', 'psutil': '5.9.0', 'ptyprocess': '0.7.0', 'pure-eval': '0.2.2', 'pyopenssl': '23.2.0', 'py-grpc-prometheus': '0.8.0', 'pyarrow': '15.0.2', 'pyasn1': '0.6.0', 'pyasn1-modules': '0.4.0', 'pybtex': '0.24.0', 'pybtex-docutils': '1.0.3', 'pycosat': '0.6.6', 'pycparser': '2.21', 'pydantic-numpy': '2.2.1', 'pydantic-settings': '2.2.1', 'pydantic-yaml': '1.3.0', 'pylint': '2.17.5', 'pyparsing': '2.4.7', 'pytest': '7.4.0', 'python-dateutil': '2.9.0.post0', 'python-dotenv': '1.0.1', 'python-etcd': '0.4.5', 'python-json-logger': '2.0.7', 'python-multipart': '0.0.12', 'python-rapidjson': '1.14', 'pytorch-lightning': '2.0.7', 'pytz': '2023.3.post1', 'pywavelets': '1.7.0', 'pyzmq': '26.1.1', 'qtconsole': '5.5.2', 'querystring-parser': '1.2.4', 'referencing': '0.30.2', 'regex': '2024.7.24', 'requests': '2.31.0', 'requests-oauthlib': '2.0.0', 'rfc3339-validator': '0.1.4', 'rfc3986-validator': '0.1.1', 'rich': '13.9.3', 'rpds-py': '0.10.6', 'rsa': '4.9', 'ruamel.yaml': '0.17.21', 'ruamel.yaml.clib': '0.2.6', 's3transfer': '0.10.3', 'safetensors': '0.4.4', 'scikit-image': '0.20.0', 'scikit-learn': '1.2.2', 'scipy': '1.11.4', 'seaborn': '0.12.2', 'secure': '0.3.0', 'setuptools': '68.0.0', 'six': '1.16.0', 'smmap': '5.0.1', 'sniffio': '1.3.1', 'snowballstemmer': '2.2.0', 'sortedcontainers': '2.4.0', 'soupsieve': '2.5', 'sphinx': '7.2.5', 'sphinx-autodoc-typehints': '1.24.0', 'sphinx-design': '0.5.0', 'sphinx-rtd-theme': '3.0.1', 'sphinxcontrib-apidoc': '0.4.0', 'sphinxcontrib-applehelp': '2.0.0', 'sphinxcontrib-bibtex': '2.6.1', 'sphinxcontrib-devhelp': '2.0.0', 'sphinxcontrib-htmlhelp': '2.1.0', 'sphinxcontrib-jquery': '4.1', 'sphinxcontrib-jsmath': '1.0.1', 'sphinxcontrib-qthelp': '2.0.0', 'sphinxcontrib-serializinghtml': '2.0.0', 'sqlmodel': '0.0.18', 'sqlparse': '0.5.1', 'stack-data': '0.2.0', 'starlette': '0.36.3', 'starlette-exporter': '0.23.0', 'statsmodels': '0.14.0', 'sympy': '1.12', 'tenacity': '9.0.0', 'tensorboard': '2.14.0', 'tensorboard-data-server': '0.7.2', 'terminado': '0.18.1', 'threadpoolctl': '3.5.0', 'tifffile': '2024.8.10', 'tinycss2': '1.3.0', 'tokenizers': '0.13.3', 'toml': '0.10.2', 'tomli': '2.0.1', 'tomlkit': '0.13.2', 'toolz': '0.12.0', 'torch': '2.1.2', 'torchaudio': '2.2.0', 'torchelastic': '0.2.2', 'torchmetrics': '1.4.1', 'torchvision': '0.16.2', 'tornado': '6.4.1', 'tqdm': '4.66.5', 'traitlets': '5.7.1', 'transformers': '4.32.1', 'triton': '2.1.0', 'truststore': '0.8.0', 'ttach': '0.0.3', 'types-dataclasses': '0.6.6', 'types-python-dateutil': '2.9.0.20240316', 'typing-extensions': '4.12.2', 'tzdata': '2024.1', 'uri-template': '1.3.0', 'urllib3': '2.2.3', 'uvicorn': '0.32.0', 'uvloop': '0.21.0', 'validators': '0.18.2', 'wcwidth': '0.2.5', 'webcolors': '24.8.0', 'webencodings': '0.5.1', 'websocket-client': '1.8.0', 'werkzeug': '3.0.3', 'wheel': '0.41.2', 'widgetsnbextension': '4.0.11', 'wrapt': '1.16.0', 'xxhash': '3.5.0', 'yarl': '1.9.4', 'zipp': '3.20.0', 'zope.event': '5.0', 'zope.interface': '7.1.1', 'zstandard': '0.19.0'}

CURRENT STACK

Name: remote-k8s-3dplts ID: adedaf0a-6bca-4759-89f0-1258cdab6c9a User: 3dplts / 475f3296-18f1-40f5-9838-64da00323fd3 Workspace: default / 6ded3963-f5f5-4735-8f2b-d7b8da2cb807

ORCHESTRATOR: 3dplts-k8s-orchestrator

Name: 3dplts-k8s-orchestrator ID: 3076ab13-0169-4311-9cb5-38ea593d797b Type: orchestrator Flavor: kubernetes Configuration: {'synchronous': False, 'timeout': 0, 'service_account_name': 'zenml-service-account', 'step_pod_service_account_name': None, 'privileged': False, 'pod_settings': None, 'orchestrator_pod_settings': None, 'incluster': False, 'kubernetes_context': None, 'kubernetes_namespace': '3dplts', 'local': False, 'skip_local_validations': False, 'parallel_step_startup_waiting_period': None} User: 3dplts / 475f3296-18f1-40f5-9838-64da00323fd3 Workspace: default / 6ded3963-f5f5-4735-8f2b-d7b8da2cb807

ARTIFACT_STORE: artifact-store-ansible

Name: artifact-store-ansible ID: bc4c1095-9c59-42f9-a7c8-f6a4057e6fa4 Type: artifact_store Flavor: s3 Configuration: {'authentication_secret': None, 'path': 's3://mlops-3dplts-zenml-bucket', 'key': '****', 'secret': '****', 'token': '****', 'client_kwargs': {'endpoint_url': 'http://deepthought1.multitel.be:10502'}, 'config_kwargs': None, 's3_additional_kwargs': None} User: 3dplts / 475f3296-18f1-40f5-9838-64da00323fd3 Workspace: default / 6ded3963-f5f5-4735-8f2b-d7b8da2cb807

CONTAINER_REGISTRY: registry-ansible

Name: registry-ansible ID: b3b51373-1dc2-49ef-a76f-7c1cd947b1e2 Type: container_registry Flavor: default Configuration: {'authentication_secret': None, 'uri': 'registry.multitel.be/signal/tef/mlops-registry/3dplts', 'default_repository': None} User: 3dplts / 475f3296-18f1-40f5-9838-64da00323fd3 Workspace: default / 6ded3963-f5f5-4735-8f2b-d7b8da2cb807

EXPERIMENT_TRACKER: mlflow-3dplts-ansible

Name: mlflow-3dplts-ansible ID: 45d25a5e-ffd5-4517-99dd-6d28a555c653 Type: experiment_tracker Flavor: mlflow Configuration: {'experiment_name': None, 'nested': False, 'tags': {}, 'tracking_uri': 'http://deepthought1.multitel.be:10533', 'tracking_username': '****', 'tracking_password': '****', 'tracking_token': '****', 'tracking_insecure_tls': False, 'databricks_host': None, 'enable_unity_catalog': False} User: 3dplts / 475f3296-18f1-40f5-9838-64da00323fd3 Workspace: default / 6ded3963-f5f5-4735-8f2b-d7b8da2cb807

IMAGE_BUILDER: image-builder-ansible

Name: image-builder-ansible ID: c0a11672-ed99-45ac-b988-ec829a372c82 Type: image_builder Flavor: local Configuration: {} User: 3dplts / 475f3296-18f1-40f5-9838-64da00323fd3 Workspace: default / 6ded3963-f5f5-4735-8f2b-d7b8da2cb807

MODEL_REGISTRY: model-registry-ansible

Name: model-registry-ansible ID: 93b7d6e6-80a6-438d-8dc9-26231be4d9e0 Type: model_registry Flavor: mlflow Configuration: {} User: 3dplts / 475f3296-18f1-40f5-9838-64da00323fd3 Workspace: default / 6ded3963-f5f5-4735-8f2b-d7b8da2cb807

What happened?

Attempting to download the artifact from s3 bucket lead to the following error:

│                                                                                                  │
│    577 │   │   │   │   │   │   file.decode() if isinstance(file, bytes) else file                │
│    578 │   │   │   │   │   )                                                                     │
│    579 │   │   │   │   │   file_path = str(Path(artifact.uri) / file_str)                        │
│ ❱  580 │   │   │   │   │   with artifact_store.open(                                             │
│    581 │   │   │   │   │   │   name=file_path, mode="rb"                                         │
│    582 │   │   │   │   │   ) as store_file:                                                      │
│    583 │   │   │   │   │   │   # Use a loop to read and write chunks of the file                 │
│                                                                                                  │
│ /home/user/.local/lib/python3.10/site-packages/zenml/artifact_stores/base_artifact_store.py:168  │
│ in __call__                                                                                      │
│                                                                                                  │
│   165 │   │   │   for key, value in kwargs.items()                                               │
│   166 │   │   }                                                                                  │
│   167 │   │                                                                                      │
│ ❱ 168 │   │   return self.func(*args, **kwargs)                                                  │
│   169                                                                                            │
│   170                                                                                            │
│   171 class BaseArtifactStoreConfig(StackComponentConfig):                                       │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: S3ArtifactStore.open() got an unexpected keyword argument 'name'

Reproduction steps

Executing the following code: from zenml.client import Client from zenml.artifacts.utils import download_artifact_files_from_response client = Client() artifact_response = client.get_artifact_version("next_id_patch", hydrate=True) artifact_response.download_files("./next_id_patch.zip")

Relevant log output

No response

Code of Conduct

clenico commented 2 hours ago

Relevant lines: https://github.com/zenml-io/zenml/blob/c3429d74a6fd26ce1c3d4e4f6a1940f979eed871/src/zenml/integrations/s3/artifact_stores/s3_artifact_store.py#L243

It seems that the s3 artifact flavor does not follow the interface with name variable and uses a path instead

schustmi commented 2 hours ago

@clenico Thanks, this was actually wrong for multiple artifact stores and fixed in #3220