search

zenml-io / zenml

ZenML 🙏: The bridge between ML and Ops. https://zenml.io.
https://zenml.io
Apache License 2.0
4.07k stars 439 forks source link

[BUG]: integration install fails when using a uv-provisioned venv #2929

Closed mennoliefstingh closed 2 months ago

mennoliefstingh commented 3 months ago

Contact Details [Optional]

[removed] because I get too much spam already. You know where to find me on slack ;)

System Information

ZENML_LOCAL_VERSION: 0.64.0 ZENML_SERVER_VERSION: 0.64.0 ZENML_SERVER_DATABASE: mysql ZENML_SERVER_DEPLOYMENT_TYPE: cloud ZENML_CONFIG_DIR: /Users/menno/Library/Application Support/zenml ZENML_LOCAL_STORE_DIR: /Users/menno/Library/Application Support/zenml/local_stores ZENML_SERVER_URL: https://564de462-zenml.cloudinfra.zenml.io ZENML_ACTIVE_REPOSITORY_ROOT: /Users/menno/alliander/aix-use-case-facilities/zenml_starter PYTHON_VERSION: 3.10.13 ENVIRONMENT: native SYSTEM_INFO: {'os': 'mac', 'mac_version': '14.5'} ACTIVE_WORKSPACE: default ACTIVE_STACK: default ACTIVE_USER: menno.liefstingh@alliander.com TELEMETRY_STATUS: enabled ANALYTICS_CLIENT_ID: 0c575a10-bad7-485a-95eb-4df4655ed703 ANALYTICS_USER_ID: fb49ba82-f5df-4ffd-8394-c1e5f1b1d030 ANALYTICS_SERVER_ID: 8d822bf3-e508-4376-81d5-b9de45f25eba INTEGRATIONS: ['airflow', 'bitbucket', 'kaniko', 'pigeon', 'scipy', 'sklearn', 'xgboost'] PACKAGES: {'babel': '2.14.0', 'gitpython': '3.1.43', 'mako': '1.3.5', 'markupsafe': '2.1.5', 'pyjwt': '2.7.0', 'pymysql': '1.1.1', 'pyyaml': '6.0.1', 'qtpy': '2.4.1', 'sqlalchemy': '2.0.32', 'sqlalchemy-utils': '0.41.2', 'send2trash': '1.8.3', 'absl-py': '1.4.0', 'aiobotocore': '2.12.2', 'aiohttp': '3.9.3', 'aioitertools': '0.11.0', 'aiosignal': '1.3.1', 'alembic': '1.8.1', 'annotated-types': '0.6.0', 'anyio': '3.7.1', 'appnope': '0.1.4', 'argon2-cffi': '23.1.0', 'argon2-cffi-bindings': '21.2.0', 'arrow': '1.3.0', 'asn1crypto': '1.5.1', 'asttokens': '2.4.1', 'async-lru': '2.0.4', 'async-timeout': '4.0.3', 'atpublic': '4.1.0', 'attrs': '23.2.0', 'bcrypt': '4.0.1', 'beautifulsoup4': '4.12.3', 'bleach': '6.1.0', 'botocore': '1.34.51', 'cachetools': '5.3.3', 'certifi': '2024.2.2', 'cffi': '1.16.0', 'charset-normalizer': '3.3.2', 'click': '8.1.3', 'click-params': '0.3.0', 'cloudpickle': '2.2.1', 'colorama': '0.4.6', 'comm': '0.2.2', 'copier': '9.3.1', 'cryptography': '42.0.5', 'debugpy': '1.8.1', 'decorator': '5.1.1', 'defusedxml': '0.7.1', 'distro': '1.9.0', 'docker': '7.1.0', 'dunamai': '1.22.0', 'exceptiongroup': '1.2.0', 'executing': '2.0.1', 'fastapi': '0.110.0', 'fastjsonschema': '2.19.1', 'filelock': '3.13.4', 'fqdn': '1.5.1', 'frozenlist': '1.4.1', 'fsspec': '2023.12.2', 'funcy': '2.0', 'gitdb': '4.0.11', 'h11': '0.14.0', 'httpcore': '1.0.5', 'httplib2': '0.19.1', 'httptools': '0.6.1', 'httpx': '0.27.0', 'idna': '3.7', 'importlib-resources': '6.4.0', 'ipinfo': '5.0.1', 'ipykernel': '6.29.4', 'ipython': '8.18.0', 'ipywidgets': '8.1.2', 'isoduration': '20.11.0', 'jedi': '0.19.1', 'jinja2': '3.1.4', 'jinja2-ansible-filters': '1.3.2', 'jinja2-time': '0.2.0', 'jmespath': '1.0.1', 'joblib': '1.4.0', 'json5': '0.9.24', 'jsonpointer': '2.4', 'jsonschema': '4.21.1', 'jsonschema-specifications': '2023.12.1', 'jupyter': '1.0.0', 'jupyter-client': '8.6.1', 'jupyter-console': '6.6.3', 'jupyter-core': '5.7.2', 'jupyter-events': '0.10.0', 'jupyter-lsp': '2.2.5', 'jupyter-server': '2.13.0', 'jupyter-server-terminals': '0.5.3', 'jupyterlab': '4.1.6', 'jupyterlab-pygments': '0.3.0', 'jupyterlab-server': '2.26.0', 'jupyterlab-widgets': '3.0.10', 'markdown-it-py': '3.0.0', 'matplotlib-inline': '0.1.6', 'mdurl': '0.1.2', 'mistune': '3.0.2', 'multidict': '6.0.5', 'nbclient': '0.10.0', 'nbconvert': '7.16.3', 'nbformat': '5.10.4', 'nest-asyncio': '1.6.0', 'notebook': '7.1.2', 'notebook-shim': '0.2.4', 'numpy': '1.26.4', 'orjson': '3.10.7', 'overrides': '7.7.0', 'packaging': '24.1', 'pandas': '1.5.3', 'pandocfilters': '1.5.1', 'parso': '0.8.4', 'passlib': '1.7.4', 'pathspec': '0.12.1', 'pexpect': '4.9.0', 'pip': '24.2', 'platformdirs': '4.2.0', 'plumbum': '1.8.3', 'prometheus-client': '0.20.0', 'prompt-toolkit': '3.0.36', 'psutil': '5.9.8', 'ptyprocess': '0.7.0', 'pure-eval': '0.2.2', 'pyopenssl': '24.1.0', 'pyarrow': '15.0.2', 'pycparser': '2.22', 'pydantic': '2.7.4', 'pydantic-core': '2.18.4', 'pydantic-settings': '2.2.1', 'pygments': '2.17.2', 'pyparsing': '2.4.7', 'python-dateutil': '2.9.0.post0', 'python-dotenv': '1.0.1', 'python-json-logger': '2.0.7', 'python-multipart': '0.0.9', 'pytimeparse': '1.1.8', 'pytz': '2024.1', 'pyyaml-include': '1.4.1', 'pyzmq': '25.1.2', 'qtconsole': '5.5.1', 'questionary': '2.0.1', 'referencing': '0.34.0', 'requests': '2.31.0', 'retrying': '1.3.4', 'rfc3339-validator': '0.1.4', 'rfc3986-validator': '0.1.1', 'rich': '13.7.1', 'rpds-py': '0.18.0', 'ruff': '0.5.7', 's3fs': '2023.12.2', 'scikit-learn': '1.3.2', 'scipy': '1.13.0', 'secure': '0.3.0', 'setuptools': '69.2.0', 'six': '1.16.0', 'smmap': '5.0.1', 'sniffio': '1.3.1', 'snowflake': '0.7.0', 'snowflake-connector-python': '3.8.1', 'snowflake-core': '0.7.0', 'snowflake-legacy': '0.7.0', 'snowflake-ml-python': '1.4.0', 'snowflake-snowpark-python': '1.14.0', 'sortedcontainers': '2.4.0', 'soupsieve': '2.5', 'sqlmodel': '0.0.18', 'sqlparse': '0.4.4', 'stack-data': '0.6.3', 'starlette': '0.36.3', 'terminado': '0.18.1', 'threadpoolctl': '3.4.0', 'tinycss2': '1.2.1', 'tomli': '2.0.1', 'tomlkit': '0.12.4', 'tornado': '6.4', 'traitlets': '5.14.2', 'types-python-dateutil': '2.9.0.20240316', 'typing-extensions': '4.11.0', 'tzdata': '2024.1', 'uri-template': '1.3.0', 'urllib3': '2.0.7', 'uvicorn': '0.30.6', 'uvloop': '0.19.0', 'validators': '0.18.2', 'watchfiles': '0.23.0', 'wcwidth': '0.2.13', 'webcolors': '1.13', 'webencodings': '0.5.1', 'websocket-client': '1.7.0', 'websockets': '12.0', 'wheel': '0.43.0', 'widgetsnbextension': '4.0.10', 'wrapt': '1.16.0', 'xgboost': '1.7.6', 'yarl': '1.9.4', 'zenml': '0.64.0'}

CURRENT STACK

Name: default ID: d12077da-c36e-4c8f-8dee-9cbed28174de Workspace: default / 4ef3a615-ed4b-4dbb-a2c9-dbbcbd58aefa

ORCHESTRATOR: default

Name: default ID: 1acad1e9-82e0-41ac-9e46-cdb0451e5db9 Type: orchestrator Flavor: local Configuration: {} Workspace: default / 4ef3a615-ed4b-4dbb-a2c9-dbbcbd58aefa

ARTIFACT_STORE: default

Name: default ID: f9766ddc-03ff-45b2-8149-d4d8e8e37d40 Type: artifact_store Flavor: local Configuration: {'path': ''} Workspace: default / 4ef3a615-ed4b-4dbb-a2c9-dbbcbd58aefa

What happened?

if [ ! -d ".venv" ]; then echo "Creating new virtual environment and activating it..." uv venv else echo "Activating existing virtual environment..." fi

- Tried to run `zenml integration install sklearn -y`
- Got the following error:

zenml integration install sklearn -y /Users/menno/alliander/aix-use-case-facilities/.venv/bin/python3: No module named pip ╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮ │ /Users/menno/alliander/aix-use-case-facilities/.venv/bin/zenml:8 in │ │ │ │ 5 from zenml.cli.cli import cli │ │ 6 if name == "main": │ │ 7 │ sys.argv[0] = re.sub(r"(-script.pyw|.exe)?$", "", sys.argv[0]) │ │ ❱ 8 │ sys.exit(cli()) │ │ 9 │ │ │ │ /Users/menno/alliander/aix-use-case-facilities/.venv/lib/python3.10/site-packages/click/core.py: │ │ 1130 in call │ │ │ │ 1127 │ │ │ 1128 │ def call(self, *args: t.Any, kwargs: t.Any) -> t.Any: │ │ 1129 │ │ """Alias for :meth:main.""" │ │ ❱ 1130 │ │ return self.main(*args, kwargs) │ │ 1131 │ │ 1132 │ │ 1133 class Command(BaseCommand): │ │ │ │ /Users/menno/alliander/aix-use-case-facilities/.venv/lib/python3.10/site-packages/click/core.py: │ │ 1055 in main │ │ │ │ 1052 │ │ try: │ │ 1053 │ │ │ try: │ │ 1054 │ │ │ │ with self.make_context(prog_name, args, extra) as ctx: │ │ ❱ 1055 │ │ │ │ │ rv = self.invoke(ctx) │ │ 1056 │ │ │ │ │ if not standalone_mode: │ │ 1057 │ │ │ │ │ │ return rv │ │ 1058 │ │ │ │ │ # it's not safe to ctx.exit(rv) here! │ │ │ │ /Users/menno/alliander/aix-use-case-facilities/.venv/lib/python3.10/site-packages/click/core.py: │ │ 1657 in invoke │ │ │ │ 1654 │ │ │ │ super().invoke(ctx) │ │ 1655 │ │ │ │ sub_ctx = cmd.make_context(cmd_name, args, parent=ctx) │ │ 1656 │ │ │ │ with sub_ctx: │ │ ❱ 1657 │ │ │ │ │ return _process_result(sub_ctx.command.invoke(sub_ctx)) │ │ 1658 │ │ │ │ 1659 │ │ # In chain mode we create the contexts step by step, but after the │ │ 1660 │ │ # base command has been invoked. Because at that point we do not │ │ │ │ /Users/menno/alliander/aix-use-case-facilities/.venv/lib/python3.10/site-packages/click/core.py: │ │ 1657 in invoke │ │ │ │ 1654 │ │ │ │ super().invoke(ctx) │ │ 1655 │ │ │ │ sub_ctx = cmd.make_context(cmd_name, args, parent=ctx) │ │ 1656 │ │ │ │ with sub_ctx: │ │ ❱ 1657 │ │ │ │ │ return _process_result(sub_ctx.command.invoke(sub_ctx)) │ │ 1658 │ │ │ │ 1659 │ │ # In chain mode we create the contexts step by step, but after the │ │ 1660 │ │ # base command has been invoked. Because at that point we do not │ │ │ │ /Users/menno/alliander/aix-use-case-facilities/.venv/lib/python3.10/site-packages/click/core.py: │ │ 1404 in invoke │ │ │ │ 1401 │ │ │ echo(style(message, fg="red"), err=True) │ │ 1402 │ │ │ │ 1403 │ │ if self.callback is not None: │ │ ❱ 1404 │ │ │ return ctx.invoke(self.callback, *ctx.params) │ │ 1405 │ │ │ 1406 │ def shell_complete(self, ctx: Context, incomplete: str) -> t.List["CompletionItem"]: │ │ 1407 │ │ """Return a list of completions for the incomplete value. Looks │ │ │ │ /Users/menno/alliander/aix-use-case-facilities/.venv/lib/python3.10/site-packages/click/core.py: │ │ 760 in invoke │ │ │ │ 757 │ │ │ │ 758 │ │ with augment_usage_errors(self): │ │ 759 │ │ │ with ctx: │ │ ❱ 760 │ │ │ │ return callback(args, kwargs) │ │ 761 │ │ │ 762 │ def forward( │ │ 763 │ │ self, cmd: "Command", *args: t.Any, **kwargs: t.Any # noqa: B902 │ │ │ │ /Users/menno/alliander/aix-use-case-facilities/.venv/lib/python3.10/site-packages/zenml/cli/inte │ │ gration.py:313 in install │ │ │ │ 310 │ │ ) │ │ 311 │ ): │ │ 312 │ │ with console.status("Installing integrations..."): │ │ ❱ 313 │ │ │ install_packages(requirements, use_uv=uv) │ │ 314 │ │ 315 │ │ 316 @integration.command( │ │ │ │ /Users/menno/alliander/aix-use-case-facilities/.venv/lib/python3.10/site-packages/zenml/cli/util │ │ s.py:1117 in install_packages │ │ │ │ 1114 │ │ │ │ "Run uv venv to create a virtualenv and retry." │ │ 1115 │ │ │ ) │ │ 1116 │ │ else: │ │ ❱ 1117 │ │ │ raise e │ │ 1118 │ │ 1119 │ │ 1120 def uninstall_package(package: str, use_uv: bool = False) -> None: │ │ │ │ /Users/menno/alliander/aix-use-case-facilities/.venv/lib/python3.10/site-packages/zenml/cli/util │ │ s.py:1105 in install_packages │ │ │ │ 1102 │ │ │ command.append("--no-warn-conflicts") │ │ 1103 │ │ │ 1104 │ try: │ │ ❱ 1105 │ │ subprocess.check_call(command) │ │ 1106 │ except subprocess.CalledProcessError as e: │ │ 1107 │ │ if ( │ │ 1108 │ │ │ use_uv │ │ │ │ /opt/homebrew/Cellar/python@3.10/3.10.13_2/Frameworks/Python.framework/Versions/3.10/lib/python3 │ │ .10/subprocess.py:369 in check_call │ │ │ │ 366 │ │ cmd = kwargs.get("args") │ │ 367 │ │ if cmd is None: │ │ 368 │ │ │ cmd = popenargs[0] │ │ ❱ 369 │ │ raise CalledProcessError(retcode, cmd) │ │ 370 │ return 0 │ │ 371 │ │ 372 │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯ CalledProcessError: Command '['/Users/menno/alliander/aix-use-case-facilities/.venv/bin/python3', '-m', 'pip', 'install', 'scikit-learn', '-qqq', '--no-warn-conflicts']' returned non-zero exit status 1.


This happens because `pip` isn't installed by default in venvs created using `uv`. 

I mitigated this by installing pip in my venv (`uv pip install pip`). 
To be fair I usually blame myself for using uv when I get these kinds of bugs, but as `zenml` supports `uv` (LOVE IT!) in so many places I think this warrants a fix of at least a warning of some sort. 

### Reproduction steps

See above (sorry)

### Relevant log output

```shell
also see above

Code of Conduct

mennoliefstingh commented 3 months ago

Reproduced the error in a 'clean' environment: 

uv venv && source .venv/bin/activate
Using Python 3.11.4 interpreter at: /Library/Frameworks/Python.framework/Versions/3.11/bin/python3
Creating virtualenv at: .venv
Activate with: source .venv/bin/activate
python3
Python 3.11.4 (v3.11.4:d2340ef257, Jun  6 2023, 19:15:51) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> subprocess.run(["python3", "-m", "pip", "install", "requests"])
/Users/menno/dev/python/test/.venv/bin/python3: No module named pip
CompletedProcess(args=['python3', '-m', 'pip', 'install', 'requests'], returncode=1)
>>> exit()
uv pip install pip
Resolved 1 package in 154ms
Installed 1 package in 17ms
 + pip==24.2
python3
Python 3.11.4 (v3.11.4:d2340ef257, Jun  6 2023, 19:15:51) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> subprocess.run(["python3", "-m", "pip", "install", "requests"])
Looking in indexes: https://pypi.org/simple, https://al*****:****@alliander.jfrog.io/artifactory/api/pypi/pypi-all/simple
Collecting requests, etc etc it worked after this
htahir1 commented 2 months ago

Fixed with #2930