search

zenml-io / zenml

ZenML 🙏: The bridge between ML and Ops. https://zenml.io.
https://zenml.io
Apache License 2.0
3.95k stars 430 forks source link

[BUG]: S3 artifact store configuration -- region #1591

Closed Yuanlim0919 closed 1 year ago

Yuanlim0919 commented 1 year ago

Contact Details [Optional]

yuanlim0919@gmail.com

System Information

ZENML_LOCAL_VERSION: 0.38.0 ZENML_SERVER_VERSION: 0.38.0 ZENML_SERVER_DATABASE: mysql ZENML_SERVER_DEPLOYMENT_TYPE: other ZENML_CONFIG_DIR: C:\Users\acer\AppData\Roaming\zenml ZENML_LOCAL_STORE_DIR: C:\Users\acer\AppData\Roaming\zenml\local_stores ZENML_SERVER_URL: http://localhost:8080 ZENML_ACTIVE_REPOSITORY_ROOT: C:\Users\acer PYTHON_VERSION: 3.10.11 ENVIRONMENT: native SYSTEM_INFO: {'os': 'windows', 'windows_version_release': '10', 'windows_version': '10.0.19045', 'windows_version_service_pack': 'SP0', 'windows_version_os_type': 'Multiprocessor Free'} ACTIVE_WORKSPACE: default ACTIVE_STACK: default ACTIVE_USER: admin TELEMETRY_STATUS: enabled ANALYTICS_CLIENT_ID: 20d4e795-cb65-407f-b728-45875558b544 ANALYTICS_USER_ID: ff84ca8e-83c4-4cd4-bcd4-f2992d4b3147 ANALYTICS_SERVER_ID: a679abec-f1d4-4296-9aed-9203abbe644f INTEGRATIONS: ['airflow', 'aws', 'bentoml', 'graphviz', 'kaniko', 'pillow', 'pytorch', 's3', 'scipy', 'sklearn'] PACKAGES: {'pip': '22.2', 'babel': '2.12.1', 'configupdater': '3.1.1', 'deprecated': '1.2.13', 'flask': '2.2.5', 'flask-appbuilder': '4.1.4', 'flask-babel': '2.0.0', 'flask-caching': '2.0.2', 'flask-jwt-extended': '4.5.2', 'flask-login': '0.6.2', 'flask-sqlalchemy': '2.5.1', 'flask-wtf': '1.1.1', 'gitpython': '3.1.31', 'jinja2': '3.1.2', 'mako': '1.2.4', 'markdown': '3.4.3', 'markupsafe': '2.1.2', 'pillow': '9.5.0', 'pyjwt': '2.7.0', 'pymysql': '1.0.3', 'pyyaml': '5.4.1', 'pygments': '2.15.1', 'sqlalchemy': '1.4.41', 'sqlalchemy-jsonfield': '1.0.1.post0', 'sqlalchemy-utils': '0.38.3', 'send2trash': '1.8.2', 'wtforms': '3.0.1', 'werkzeug': '2.2.3', 'aiobotocore': '2.4.2', 'aiohttp': '3.8.4', 'aioitertools': '0.11.0', 'aiosignal': '1.3.1', 'alembic': '1.8.1', 'analytics-python': '1.4.post1', 'anyio': '3.6.2', 'apache-airflow': '2.4.3', 'apache-airflow-providers-common-sql': '1.5.1', 'apache-airflow-providers-ftp': '3.4.1', 'apache-airflow-providers-http': '4.4.1', 'apache-airflow-providers-imap': '3.2.1', 'apache-airflow-providers-sqlite': '3.4.1', 'apispec': '3.3.2', 'appdirs': '1.4.4', 'argcomplete': '3.0.8', 'argon2-cffi': '21.3.0', 'argon2-cffi-bindings': '21.2.0', 'arrow': '1.2.3', 'asgiref': '3.7.1', 'asttokens': '2.2.1', 'async-timeout': '4.0.2', 'attrs': '22.2.0', 'backcall': '0.2.0', 'backoff': '1.10.0', 'bcrypt': '4.0.1', 'beautifulsoup4': '4.12.2', 'bentoml': '1.0.20', 'bleach': '6.0.0', 'blinker': '1.6.2', 'boto3': '1.26.144', 'botocore': '1.27.59', 'build': '0.10.0', 'cachelib': '0.9.0', 'cattrs': '22.2.0', 'certifi': '2023.5.7', 'cffi': '1.15.1', 'charset-normalizer': '3.1.0', 'circus': '0.18.0', 'click': '8.1.3', 'click-option-group': '0.5.5', 'click-params': '0.3.0', 'clickclick': '20.10.2', 'cloudpickle': '2.2.1', 'colorama': '0.4.6', 'colorlog': '4.8.0', 'comm': '0.1.3', 'commonmark': '0.9.1', 'connexion': '2.14.2', 'contextlib2': '21.6.0', 'contourpy': '1.0.7', 'cron-descriptor': '1.4.0', 'croniter': '1.3.15', 'cryptography': '41.0.1', 'cycler': '0.11.0', 'databricks-cli': '0.17.7', 'debugpy': '1.6.7', 'decorator': '5.1.1', 'deepmerge': '1.1.0', 'defusedxml': '0.7.1', 'dill': '0.3.6', 'distro': '1.8.0', 'dnspython': '2.3.0', 'docker': '6.0.1', 'docutils': '0.20.1', 'email-validator': '1.3.1', 'entrypoints': '0.4', 'exceptiongroup': '1.1.1', 'executing': '1.2.0', 'fastjsonschema': '2.16.3', 'filelock': '3.12.0', 'flask-session': '0.5.0', 'fonttools': '4.39.4', 'fqdn': '1.5.1', 'frozenlist': '1.3.3', 'fs': '2.4.16', 'fsspec': '2022.11.0', 'gitdb': '4.0.10', 'google-pasta': '0.2.0', 'graphviz': '0.20.1', 'greenlet': '2.0.2', 'grpcio': '1.54.2', 'grpcio-health-checking': '1.54.2', 'gunicorn': '20.1.0', 'h11': '0.14.0', 'httpcore': '0.17.2', 'httplib2': '0.19.1', 'httpx': '0.24.1', 'idna': '3.4', 'importlib-metadata': '4.13.0', 'inflection': '0.5.1', 'ipykernel': '6.23.0', 'ipython': '8.13.2', 'ipython-genutils': '0.2.0', 'ipywidgets': '7.7.5', 'isoduration': '20.11.0', 'itsdangerous': '2.1.2', 'jedi': '0.18.2', 'jmespath': '1.0.1', 'joblib': '1.2.0', 'jsonpointer': '2.3', 'jsonschema': '4.17.3', 'jupyter-client': '8.2.0', 'jupyter-core': '5.3.0', 'jupyter-events': '0.6.3', 'jupyter-server': '2.5.0', 'jupyter-server-terminals': '0.4.4', 'jupyterlab-pygments': '0.2.2', 'jupyterlab-widgets': '1.1.4', 'kiwisolver': '1.4.4', 'lazy-object-proxy': '1.9.0', 'linkify-it-py': '2.0.2', 'lockfile': '0.12.2', 'markdown-it-py': '2.2.0', 'marshmallow': '3.19.0', 'marshmallow-enum': '1.5.1', 'marshmallow-oneofschema': '3.0.1', 'marshmallow-sqlalchemy': '0.26.1', 'matplotlib': '3.7.1', 'matplotlib-inline': '0.1.6', 'mdit-py-plugins': '0.3.5', 'mdurl': '0.1.2', 'mistune': '2.0.5', 'mlflow': '2.3.1', 'monotonic': '1.6', 'mpmath': '1.3.0', 'multidict': '6.0.4', 'multiprocess': '0.70.14', 'nbclassic': '1.0.0', 'nbclient': '0.7.4', 'nbconvert': '7.4.0', 'nbformat': '5.8.0', 'nest-asyncio': '1.5.6', 'networkx': '3.1', 'notebook': '6.5.4', 'notebook-shim': '0.2.3', 'numpy': '1.24.3', 'oauthlib': '3.2.2', 'opentelemetry-api': '1.14.0', 'opentelemetry-instrumentation': '0.35b0', 'opentelemetry-instrumentation-aiohttp-client': '0.35b0', 'opentelemetry-instrumentation-asgi': '0.35b0', 'opentelemetry-instrumentation-grpc': '0.35b0', 'opentelemetry-sdk': '1.14.0', 'opentelemetry-semantic-conventions': '0.35b0', 'opentelemetry-util-http': '0.35b0', 'packaging': '23.1', 'pandas': '2.0.1', 'pandocfilters': '1.5.0', 'parso': '0.8.3', 'passlib': '1.7.4', 'pathos': '0.3.0', 'pathspec': '0.9.0', 'pendulum': '2.1.2', 'pickleshare': '0.7.5', 'pip-requirements-parser': '32.0.1', 'pip-tools': '6.13.0', 'platformdirs': '3.5.0', 'pluggy': '1.0.0', 'pox': '0.3.2', 'ppft': '1.7.6.6', 'prison': '0.2.1', 'prometheus-client': '0.16.0', 'prompt-toolkit': '3.0.38', 'protobuf': '3.20.3', 'protobuf3-to-dict': '0.1.5', 'psutil': '5.9.5', 'pure-eval': '0.2.2', 'pyarrow': '11.0.0', 'pycparser': '2.21', 'pydantic': '1.10.7', 'pyngrok': '6.0.0', 'pynvml': '11.5.0', 'pyparsing': '2.4.7', 'pyproject-hooks': '1.0.0', 'pyrsistent': '0.19.3', 'python-daemon': '3.0.1', 'python-dateutil': '2.8.2', 'python-json-logger': '2.0.7', 'python-multipart': '0.0.6', 'python-nvd3': '0.15.0', 'python-slugify': '8.0.1', 'python-terraform': '0.10.1', 'pytz': '2023.3', 'pytzdata': '2020.1', 'pywin32': '306', 'pywinpty': '2.0.10', 'pyzmq': '25.0.2', 'querystring-parser': '1.2.4', 'requests': '2.30.0', 'requests-toolbelt': '1.0.0', 'rfc3339-validator': '0.1.4', 'rfc3986-validator': '0.1.1', 'rich': '12.6.0', 's3fs': '2022.11.0', 's3transfer': '0.6.1', 'sagemaker': '2.117.0', 'schema': '0.7.5', 'scikit-learn': '1.2.2', 'scipy': '1.10.1', 'setproctitle': '1.3.2', 'setuptools': '66.0.0', 'simple-di': '0.1.5', 'six': '1.16.0', 'smdebug-rulesconfig': '1.0.1', 'smmap': '5.0.0', 'sniffio': '1.3.0', 'soupsieve': '2.4.1', 'sqlalchemy2-stubs': '0.0.2a34', 'sqlmodel': '0.0.8', 'sqlparse': '0.4.4', 'stack-data': '0.6.2', 'starlette': '0.25.0', 'swagger-ui-bundle': '0.0.9', 'sympy': '1.12', 'tabulate': '0.9.0', 'tenacity': '8.2.2', 'termcolor': '2.3.0', 'terminado': '0.17.1', 'text-unidecode': '1.3', 'threadpoolctl': '3.1.0', 'tinycss2': '1.2.1', 'tomli': '2.0.1', 'torch': '2.0.1', 'torchvision': '0.15.2', 'tornado': '6.3.1', 'traitlets': '5.9.0', 'typing-extensions': '4.5.0', 'tzdata': '2023.3', 'uc-micro-py': '1.0.2', 'unicodecsv': '0.14.1', 'uri-template': '1.2.0', 'urllib3': '1.26.15', 'uvicorn': '0.22.0', 'validators': '0.18.2', 'waitress': '2.1.2', 'watchfiles': '0.19.0', 'wcwidth': '0.2.6', 'webcolors': '1.13', 'webencodings': '0.5.1', 'websocket-client': '1.5.1', 'wheel': '0.38.4', 'widgetsnbextension': '3.6.4', 'wrapt': '1.15.0', 'yarl': '1.9.2', 'zenml': '0.38.0', 'zipp': '3.15.0'}

CURRENT STACK

Name: default ID: 14b9765f-8b13-4ba8-b83b-fd61f4ada526 Shared: No User: admin / ff84ca8e-83c4-4cd4-bcd4-f2992d4b3147 Workspace: default / e6d3ad08-426e-4cbb-a367-fd816184cb7f

ORCHESTRATOR: default

Name: default ID: 7a5cbef0-12ee-4532-be7f-3471b65383fe Type: orchestrator Flavor: local Configuration: {} Shared: No User: admin / ff84ca8e-83c4-4cd4-bcd4-f2992d4b3147 Workspace: default / e6d3ad08-426e-4cbb-a367-fd816184cb7f

ARTIFACT_STORE: default

Name: default ID: dd539621-c548-4563-9f86-23e333d7affa Type: artifact_store Flavor: local Configuration: {'path': ''} Shared: No User: admin / ff84ca8e-83c4-4cd4-bcd4-f2992d4b3147 Workspace: default / e6d3ad08-426e-4cbb-a367-fd816184cb7f

What happened?

Initially, I've created an artifact store of s3 by using following command: zenml artifact-store register s3_bucket --flavor=s3 --path='s3://<my_own_path>/' --authentication_secret=s3_secret_key

When I execute the ML pipeline with the artifact store, it replies: PermissionError: The unspecified location constraint is incompatible for the region specific endpoint this request was sent to.

I wish to update the artifact store with --client_kwargs parameter, so I enter the following command: zenml artifact-store update s3_store_new --client_kwargs='{"region_name": "ap-southeast-2"}'

It responds: Connected to the ZenML server: 'http://localhost:8080' Running with active workspace: 'default' (repository) Running with active stack: 'default' (repository) Error: Invalid argument: 's3_bucket'. Please provide args with a proper identifier as the key and the following structure: --custom_argument="value"

Reproduction steps

1.I tried to reregister a new artifact store with the same configuration: zenml artifact-store register s3_bucket_01 --flavor=s3 --path='s3://<my_own_path>/' --authentication_secret=s3_secret_key --custom_argument='{"region_name": "ap-southeast-2"}'

But the error response still the same:

Connected to the ZenML server: 'http://localhost:8080' Running with active workspace: 'default' (repository) Running with active stack: 'default' (repository) Error: Invalid argument: 's3_bucket_01'. Please provide args with a proper identifier as the key and the following structure: --custom_argument="value"

I am just following the syntax provided in official docs: s3 advanced configuration ...

Relevant log output

No response

Code of Conduct

stefannica commented 1 year ago

@Yuanlim0919 I think there is some inconsistency in your bug report. This is what I tried on my machine and worked perfectly:

$ zenml artifact-store register s3_bucket --flavor=s3 --path='s3://<my_own_path>/' --authentication_secret=s3_secret_key
Successfully registered artifact_store `s3_bucket`.

$ zenml artifact-store update s3_bucket --client_kwargs='{"region_name": "ap-southeast-2"}'
Successfully updated artifact store `s3_bucket`.

$ zenml artifact-store describe s3_bucket
Artifact_Store 's3_bucket' of flavor 's3' with id '1daa61a6-dac5-42bb-af24-8df0b8a4e001' is owned by user 'default' and is 'private'.
     's3_bucket' ARTIFACT_STORE Component Configuration      
┏━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ COMPONENT_PROPERTY    │ VALUE                             ┃
┠───────────────────────┼───────────────────────────────────┨
┃ PATH                  │ s3://<my_own_path>/               ┃
┠───────────────────────┼───────────────────────────────────┨
┃ AUTHENTICATION_SECRET │ s3_secret_key                     ┃
┠───────────────────────┼───────────────────────────────────┨
┃ CLIENT_KWARGS         │ {"region_name": "ap-southeast-2"} ┃
┗━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛

On the other hand, some of the CLI commands you included in the bug descriptions failed because they had incorrect names or arguments:

$ zenml artifact-store update s3_store_new --client_kwargs='{"region_name": "ap-southeast-2"}'
Error: "No stack_components have been found that have an ID prefix that matches the provided string 's3_store_new'."

$ zenml artifact-store register s3_bucket_01 --flavor=s3 --path='s3://<my_own_path>/' --authentication_secret=s3_secret_key --custom_argument='{"region_name": "ap-southeast-2"}'
...
│ /home/stefan/aspyre/src/zenml/src/zenml/stack/stack_component.py:106 in __init__                 │
│                                                                                                  │
│   103 │   │   │   │   │   "is required for this attribute."                                      │
│   104 │   │   │   │   )                                                                          │
│   105 │   │                                                                                      │
│ ❱ 106 │   │   super().__init__(**kwargs)                                                         │
│   107 │                                                                                          │
│   108 │   @property                                                                              │
│   109 │   def required_secrets(self) -> Set[secret_utils.SecretReference]:                       │
│                                                                                                  │
│ /home/stefan/aspyre/src/zenml/examples/_as_simple_as_possible/pydantic/main.py:341 in            │
│ pydantic.main.BaseModel.__init__                                                                 │
│                                                                                                  │
│ [Errno 2] No such file or directory:                                                             │
│ '/home/stefan/aspyre/src/zenml/examples/_as_simple_as_possible/pydantic/main.py'                 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValidationError: 1 validation error for S3ArtifactStoreConfig
custom_argument
  extra fields not permitted (type=value_error.extra)

As a side-note, please be aware that there is a more convenient way to manage authentication connections to external services like AWS: Service Connectors. In your case, the AWS Service Connector can give you more control over how you link stack components to AWS, e.g.:

$ pip install zenml[connectors-aws]

$ zenml service-connector list-types --type aws
┏━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━┯━━━━━━━┯━━━━━━━━┓
┃         NAME          │ TYPE   │ RESOURCE TYPES        │ AUTH METHODS     │ LOCAL │ REMOTE ┃
┠───────────────────────┼────────┼───────────────────────┼──────────────────┼───────┼────────┨
┃ AWS Service Connector │ 🔶 aws │ 🔶 aws-generic        │ implicit         │ ✅    │ ✅     ┃
┃                       │        │ 📦 s3-bucket          │ secret-key       │       │        ┃
┃                       │        │ 🌀 kubernetes-cluster │ sts-token        │       │        ┃
┃                       │        │ 🐳 docker-registry    │ iam-role         │       │        ┃
┃                       │        │                       │ session-token    │       │        ┃
┃                       │        │                       │ federation-token │       │        ┃
┗━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━┷━━━━━━━┷━━━━━━━━┛

$ AWS_DEFAULT_REGION=eu-central-1 zenml service-connector register --type aws --auto-configure aws-multi
⠇ Registering service connector 'aws-multi'...
Successfully registered service connector `aws-multi` with access to the following resources:
┏━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃     RESOURCE TYPE     │ RESOURCE NAMES                                  ┃
┠───────────────────────┼─────────────────────────────────────────────────┨
┃    🔶 aws-generic     │ eu-central-1                                    ┃
┠───────────────────────┼─────────────────────────────────────────────────┨
┃     📦 s3-bucket      │ s3://aws-ia-mwaa-715803424590                   ┃
┃                       │ s3://zenbytes-bucket                            ┃
┃                       │ s3://zenfiles                                   ┃
┠───────────────────────┼─────────────────────────────────────────────────┨
┃ 🌀 kubernetes-cluster │ kubeflowmultitenant                             ┃
┃                       │ zenbox                                          ┃
┠───────────────────────┼─────────────────────────────────────────────────┨
┃  🐳 docker-registry   │ 715803424590.dkr.ecr.eu-central-1.amazonaws.com ┃
┗━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛

$ zenml artifact-store register s3_bucket --flavor=s3 --path=s3://zenfiles
Successfully registered artifact_store `s3_bucket`.

$ zenml artifact-store connect s3_bucket --connector aws-multi
Successfully connected artifact store `s3_bucket` to the following resources:
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┓
┃             CONNECTOR ID             │ CONNECTOR NAME │ CONNECTOR TYPE │ RESOURCE TYPE │ RESOURCE NAMES ┃
┠──────────────────────────────────────┼────────────────┼────────────────┼───────────────┼────────────────┨
┃ 73c8ecbb-4a8a-461e-a9b8-7e1fe36711ec │ aws-multi      │ 🔶 aws         │ 📦 s3-bucket  │ s3://zenfiles  ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┛
Yuanlim0919 commented 1 year ago

Thank you for your response, I tried it once again but it was still the same. Is it matters if I am using Anaconda Prompt in Windows?

zenml artifact-store register s3_bucket_01 --flavor=s3 --path='s3://mlopstestforzenml/' --authentication_secret=s3_secret_key Connected to the ZenML server: 'http://localhost:8080' Running with active workspace: 'default' (repository) Running with active stack: 'stack_test001' (repository) Successfully registered artifact_store s3_bucket_01.

zenml artifact-store update s3_bucket_01 --client_kwargs='{"region_name": "ap-southeast-2"}' Connected to the ZenML server: 'http://localhost:8080' Running with active workspace: 'default' (repository) Running with active stack: 'stack_test001' (repository) Error: Invalid argument: 's3_bucket_01'. Please provide args with a proper identifier as the key and the following structure: --custom_argument="value"

This is the result of artifact-store describe

zenml artifact-store describe s3_bucket_01 Connected to the ZenML server: 'http://localhost:8080' Running with active workspace: 'default' (repository) Running with active stack: 'stack_test001' (repository) Artifact_Store 's3_bucket_01' of flavor 's3' with id 'd8b345d1-5ed7-45cf-8ef5-02abe9b68939' is owned by user 'admin' and is 'private'. 's3_bucket_01' ARTIFACT_STORE Component Configuration ┌───────────────────────┬───────────────────────────┐ │ COMPONENT_PROPERTY │ VALUE │ ├───────────────────────┼───────────────────────────┤ │ PATH │ 's3://mlopstestforzenml/' │ ├───────────────────────┼───────────────────────────┤ │ AUTHENTICATION_SECRET │ s3_secret_key │ └───────────────────────┴───────────────────────────┘ No labels are set for this component.

stefannica commented 1 year ago

Thank you for your response, I tried it once again but it was still the same. Is it matters if I am using Anaconda Prompt in Windows?

Yes, there's a high chance that this could be related. Would you mind setting a breakpoint() in the zenml.cli.utils.parse_name_and_extra_arguments function and printing out the args contents ? That would help me tremendously.

Yuanlim0919 commented 1 year ago

The print out results of args: ["--client_kwargs='{region_name:", "ap-southeast-2}'", 's3_bucket_01'] I think that the problem may comes from here (line 615~line 621 in zenml.cil.utils.parse_name_and_extra_arguments):

for a in args:
    if not a.startswith("--") or "=" not in a:
        error(f"Invalid argument: '{a}'. {message}")
    key, value = a[2:].split("=", maxsplit=1)
    if not key.isidentifier():
        error(f"Invalid argument: '{a}'. {message}")
    args_dict[key] = value

Also, zenml version I am using now is 0.38

stefannica commented 1 year ago

The print out results of args: ["--client_kwargs='{region_name:", "ap-southeast-2}'", 's3_bucket_01']

This is exactly what I get on my system (minus a quote or two):

['--client_kwargs={"region_name": "ap-southeast-2"}', 's3_bucket_01']

So the Anaconda prompt doesn't seem to be the cause here.

I think that the problem may comes from here (line 615~line 621 in zenml.cil.utils.parse_name_and_extra_arguments):

for a in args:
    if not a.startswith("--") or "=" not in a:
        error(f"Invalid argument: '{a}'. {message}")
    key, value = a[2:].split("=", maxsplit=1)
    if not key.isidentifier():
        error(f"Invalid argument: '{a}'. {message}")
    args_dict[key] = value

I don't see how, because the code block above it should have removed the name from the list:

    for i, arg in enumerate(args):
        if arg.startswith("--"):
            continue
        name = args.pop(i)
        break
    else:
        if name_mandatory:
            error(
                "A name must be supplied. Please see the command help for more "
                "information."
            )

It almost looks like args.pop(i) doesn't work on your machine or Python environment.

The other thing that makes absolutely no sense to me is that this same parse_name_and_extra_arguments function is also called when you run zenml artifact-store register, but it works in that case.

I'm sorry, but I cannot reproduce this on my end, even with the 0.38 ZenML version. You may try to debug that CLI utils function and perhaps get to the bottom of it, if you have time.

stefannica commented 1 year ago

@Yuanlim0919 is there any more information you can share about this issue ?

stefannica commented 1 year ago

I'm closing this issue on account of not having enough information to reproduce it. Please feel free to re-open if still applicable.