search

fuzzylabs / matcha

A tool to provision MLOps environments in Azure
http://mymatcha.ai/
Apache License 2.0
30 stars 2 forks source link

[BUG]: ModuleNotFoundError when running `matcha provision` #218

Open fschlz opened 10 months ago

fschlz commented 10 months ago

Contact details [optional]

No response

Tell us about the bug in plenty of detail

Just installed the tool via poetry add, got nothing else installed. Running Python 3.12.0

Steps to reproduce the bug

  1. poetry init
  2. add matcha-ml
  3. poetry install
  4. activate venv
  5. matcha provision

Any relevant log output

❯ matcha provision
Traceback (most recent call last):
  File "/Users/fschulz/dev/learn-matcha/.venv/bin/matcha", line 5, in <module>
    from matcha_ml.cli.cli import app
  File "/Users/fschulz/dev/learn-matcha/.venv/lib/python3.12/site-packages/matcha_ml/cli/cli.py", line 6, in <module>
    from matcha_ml import __version__, core
  File "/Users/fschulz/dev/learn-matcha/.venv/lib/python3.12/site-packages/matcha_ml/core/__init__.py", line 2, in <module>
    from .core import (
  File "/Users/fschulz/dev/learn-matcha/.venv/lib/python3.12/site-packages/matcha_ml/core/core.py", line 6, in <module>
    from matcha_ml.cli._validation import get_command_validation
  File "/Users/fschulz/dev/learn-matcha/.venv/lib/python3.12/site-packages/matcha_ml/cli/_validation.py", line 9, in <module>
    from matcha_ml.core._validation import is_valid_prefix, is_valid_region
  File "/Users/fschulz/dev/learn-matcha/.venv/lib/python3.12/site-packages/matcha_ml/core/_validation.py", line 4, in <module>
    from matcha_ml.services import AzureClient
  File "/Users/fschulz/dev/learn-matcha/.venv/lib/python3.12/site-packages/matcha_ml/services/__init__.py", line 2, in <module>
    from .azure_service import AzureClient
  File "/Users/fschulz/dev/learn-matcha/.venv/lib/python3.12/site-packages/matcha_ml/services/azure_service.py", line 8, in <module>
    from azure.identity import AzureCliCredential, CredentialUnavailableError
  File "/Users/fschulz/dev/learn-matcha/.venv/lib/python3.12/site-packages/azure/identity/__init__.py", line 10, in <module>
    from ._credentials import (
  File "/Users/fschulz/dev/learn-matcha/.venv/lib/python3.12/site-packages/azure/identity/_credentials/__init__.py", line 11, in <module>
    from .default import DefaultAzureCredential
  File "/Users/fschulz/dev/learn-matcha/.venv/lib/python3.12/site-packages/azure/identity/_credentials/default.py", line 20, in <module>
    from .vscode import VisualStudioCodeCredential
  File "/Users/fschulz/dev/learn-matcha/.venv/lib/python3.12/site-packages/azure/identity/_credentials/vscode.py", line 22, in <module>
    from .._internal.macos_vscode_adapter import get_refresh_token, get_user_settings
  File "/Users/fschulz/dev/learn-matcha/.venv/lib/python3.12/site-packages/azure/identity/_internal/macos_vscode_adapter.py", line 8, in <module>
    from msal_extensions.osx import Keychain, KeychainError
  File "/Users/fschulz/dev/learn-matcha/.venv/lib/python3.12/site-packages/msal_extensions/__init__.py", line 11, in <module>
    from .cache_lock import CrossPlatLock
  File "/Users/fschulz/dev/learn-matcha/.venv/lib/python3.12/site-packages/msal_extensions/cache_lock.py", line 7, in <module>
    from distutils.version import LooseVersion
ModuleNotFoundError: No module named 'distutils'

❯ poetry show
azure-common             1.1.28     Microsoft Azure Client Library for Python (Common)
azure-core               1.29.5     Microsoft Azure Core Library for Python
azure-identity           1.15.0     Microsoft Azure Identity Library for Python
azure-mgmt-authorization 3.0.0      Microsoft Azure Authorization Management Client Library for Python
azure-mgmt-confluent     1.0.0      Microsoft Azure Confluent Management Client Library for Python
azure-mgmt-core          1.4.0      Microsoft Azure Management Core Library for Python
azure-mgmt-resource      23.0.1     Microsoft Azure Resource Management Client Library for Python
azure-mgmt-storage       21.1.0     Microsoft Azure Storage Management Client Library for Python
azure-mgmt-subscription  3.1.1      Microsoft Azure Subscription Management Client Library for Python
azure-storage-blob       12.18.3    Microsoft Azure Blob Storage Client Library for Python
backoff                  2.2.1      Function decoration for backoff and retry
certifi                  2023.7.22  Python package for providing Mozilla's CA Bundle.
cffi                     1.16.0     Foreign Function Interface for Python calling C code.
charset-normalizer       3.3.1      The Real First Universal Charset Detector. Open, modern and actively maintained alternative to Chardet.
click                    8.1.7      Composable command line interface toolkit
colorama                 0.4.6      Cross-platform colored terminal text.
commonmark               0.9.1      Python parser for the CommonMark Markdown spec
cryptography             41.0.5     cryptography is a package which provides cryptographic recipes and primitives to Python developers.
dataclasses-json         0.5.9      Easily serialize dataclasses to and from JSON
idna                     3.4        Internationalized Domain Names in Applications (IDNA)
isodate                  0.6.1      An ISO 8601 date/time/duration parser and formatter
marshmallow              3.20.1     A lightweight library for converting complex datatypes to and from native Python datatypes.
marshmallow-enum         1.5.1      Enum field for Marshmallow
matcha-ml                0.2.9      Matcha: An open source tool for provisioning MLOps environments to the cloud.
monotonic                1.6        An implementation of time.monotonic() for Python 2 & < 3.3
msal                     1.24.1     The Microsoft Authentication Library (MSAL) for Python library
msal-extensions          1.0.0      Microsoft Authentication Library extensions (MSAL EX) provides a persistence API that can save your data on disk, encrypted on Windows, macOS and Linux. Concurrent data access will ...
msrest                   0.7.1      AutoRest swagger generator Python client runtime.
mypy-extensions          1.0.0      Type system extensions for programs checked with the mypy type checker.
oauthlib                 3.2.2      A generic, spec-compliant, thorough implementation of the OAuth request-signing logic
packaging                23.2       Core utilities for Python packages
portalocker              2.8.2      Wraps the portalocker recipe for easy usage
pycparser                2.21       C parser in Python
pygments                 2.16.1     Pygments is a syntax highlighting package written in Python.
pyjwt                    2.8.0      JSON Web Token implementation in Python
python-dateutil          2.8.2      Extensions to the standard Python datetime module
python-terraform         0.10.1     This is a python module provide a wrapper of terraform command line tool
pyyaml                   6.0.1      YAML parser and emitter for Python
requests                 2.31.0     Python HTTP for Humans.
requests-oauthlib        1.3.1      OAuthlib authentication support for Requests.
rich                     12.6.0     Render rich text, tables, progress bars, syntax highlighting, markdown and more to the terminal
segment-analytics-python 2.2.3      The hassle-free way to integrate analytics into any python application.
shellingham              1.5.4      Tool to Detect Surrounding Shell
six                      1.16.0     Python 2 and 3 compatibility utilities
typer                    0.7.0      Typer, build great CLIs. Easy to code. Based on Python type hints.
types-pyyaml             6.0.12.12  Typing stubs for PyYAML
types-urllib3            1.26.25.14 Typing stubs for urllib3
typing-extensions        4.8.0      Backported and Experimental Type Hints for Python 3.8+
typing-inspect           0.9.0      Runtime inspection utilities for typing module.
urllib3                  1.26.6     HTTP library with thread-safe connection pooling, file post, and more.
fschlz commented 10 months ago

downgrading to Python 3.8 lets me run the command matcha provision, but finally results in another error:

⠦ 0:10:28 ApplyingWARNING:python_terraform:error: b'\nError: Provider configuration: cannot load Kubernetes client config\n\n  with provider["registry.terraform.io/hashicorp/kubernetes"],\n  on kubernetes.tf line 15, in provider "kubernetes":\n  15: provider "kubernetes" {\n\ninvalid configuration: [client-cert-data and client-cert are both specified\nfor custom. client-cert-data will override., client-key-data and client-key\nare both specified for custom; client-key-data will override]\n'
⠦ 0:10:28 Applying
Terraform failed because of the following error: '
Error: Provider configuration: cannot load Kubernetes client config

  with provider["registry.terraform.io/hashicorp/kubernetes"],
  on kubernetes.tf line 15, in provider "kubernetes":
  15: provider "kubernetes" {

invalid configuration: 
'.
fschlz commented 10 months ago

running matcha destroy to unprovision all created resources throws another error:

❯ matcha destroy

The following resources will be destroyed:

1. Azure Kubernetes Service (AKS): A kubernetes cluster
2. Two Storage Containers: A storage container for experiment tracking artifacts and a second for model training artifacts
3. Seldon Core: A framework for model deployment on top of a kubernetes cluster
4. Azure Container Registry: A container registry for storing docker images
5. ZenServer: A zenml server required for remote orchestration
6. Azure Resource Group: The resource group containing the provisioned resources
7. Matcha State Container: A storage container for tracking matcha state

Destroying the resources may take approximately 20 minutes. May we suggest you grab a cup of 🍵?

Are you happy for 'destroy' to run? [y/N]: y

⏳ Destroying Matcha resources...

⠼ 0:00:01 Destroying
Terraform failed because of the following error: '
Error: Failed to read variables file

Given variables file
/Users/fschulz/dev/learn-matcha/.matcha/infrastructure/resources/terraform.tfvars.json
does not exist.
'.