[Bug] make venv failing in noop/python

daw3rd commented 5 days ago

Search before asking

[X] I searched the issues and found no similar issues.

Component

Makefiles

What happened + What you expected to happen

Creating the venv in the noop/python project dir is failing. There is a secondary problem in that the make did not terminate when it got the error from pip install.

Reproduction script

cd transforms/universal/noop/python
make venv
python -m venv venv
Requirement already satisfied: pip in ./venv/lib/python3.11/site-packages (24.0)
Collecting pip
  Using cached pip-24.3.1-py3-none-any.whl.metadata (3.7 kB)
Using cached pip-24.3.1-py3-none-any.whl (1.8 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 24.0
    Uninstalling pip-24.0:
      Successfully uninstalled pip-24.0
Successfully installed pip-24.3.1
Collecting wheel
  Using cached wheel-0.45.0-py3-none-any.whl.metadata (2.3 kB)
Collecting pytest
  Using cached pytest-8.3.3-py3-none-any.whl.metadata (7.5 kB)
Collecting pytest-cov
  Using cached pytest_cov-6.0.0-py3-none-any.whl.metadata (27 kB)
Collecting iniconfig (from pytest)
  Using cached iniconfig-2.0.0-py3-none-any.whl.metadata (2.6 kB)
Collecting packaging (from pytest)
  Using cached packaging-24.2-py3-none-any.whl.metadata (3.2 kB)
Collecting pluggy<2,>=1.5 (from pytest)
  Using cached pluggy-1.5.0-py3-none-any.whl.metadata (4.8 kB)
Collecting coverage>=7.5 (from coverage[toml]>=7.5->pytest-cov)
  Using cached coverage-7.6.7-cp311-cp311-macosx_11_0_arm64.whl.metadata (8.2 kB)
Using cached wheel-0.45.0-py3-none-any.whl (72 kB)
Using cached pytest-8.3.3-py3-none-any.whl (342 kB)
Using cached pytest_cov-6.0.0-py3-none-any.whl (22 kB)
Using cached coverage-7.6.7-cp311-cp311-macosx_11_0_arm64.whl (207 kB)
Using cached pluggy-1.5.0-py3-none-any.whl (20 kB)
Using cached iniconfig-2.0.0-py3-none-any.whl (5.9 kB)
Using cached packaging-24.2-py3-none-any.whl (65 kB)
Installing collected packages: wheel, pluggy, packaging, iniconfig, coverage, pytest, pytest-cov
Successfully installed coverage-7.6.7 iniconfig-2.0.0 packaging-24.2 pluggy-1.5.0 pytest-8.3.3 pytest-cov-6.0.0 wheel-0.45.0
Installing Python data processing library source to existing venv
pip uninstall -y data-prep-toolkit 
WARNING: Skipping data-prep-toolkit as it is not installed.
Begin installing source from ../../../../data-processing-lib into venv
Obtaining file:///Users/dawood/git/data-prep-kit/data-processing-lib
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... error
  error: subprocess-exited-with-error

  × Getting requirements to build editable did not run successfully.
  │ exit code: 1
  ╰─> [6 lines of output]
      running egg_info
      writing data_prep_toolkit.egg-info/PKG-INFO
      writing dependency_links to data_prep_toolkit.egg-info/dependency_links.txt
      writing requirements to data_prep_toolkit.egg-info/requires.txt
      writing top-level names to data_prep_toolkit.egg-info/top_level.txt
      error: package directory 'ray/src/data_processing/runtime/pure_python' does not exist
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build editable did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
make[1]: *** [.defaults.install-src-venv] Error 1
Installed source from Python processing library for /Users/dawood/git/data-prep-kit/transforms/universal/noop/python/venv/bin/python
Installing from pyproject.toml
Obtaining file:///Users/dawood/git/data-prep-kit/transforms/universal/noop/python
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Collecting data-prep-toolkit==0.2.2.dev2 (from dpk_noop_transform_python==0.2.2.dev2)
  Using cached data_prep_toolkit-0.2.2.dev2-py3-none-any.whl.metadata (2.2 kB)
Collecting numpy<1.29.0 (from data-prep-toolkit==0.2.2.dev2->dpk_noop_transform_python==0.2.2.dev2)
  Using cached numpy-1.26.4-cp311-cp311-macosx_11_0_arm64.whl.metadata (114 kB)
Collecting pyarrow==16.1.0 (from data-prep-toolkit==0.2.2.dev2->dpk_noop_transform_python==0.2.2.dev2)
  Using cached pyarrow-16.1.0-cp311-cp311-macosx_11_0_arm64.whl.metadata (3.0 kB)
Collecting boto3==1.34.69 (from data-prep-toolkit==0.2.2.dev2->dpk_noop_transform_python==0.2.2.dev2)
  Using cached boto3-1.34.69-py3-none-any.whl.metadata (6.6 kB)
Collecting argparse (from data-prep-toolkit==0.2.2.dev2->dpk_noop_transform_python==0.2.2.dev2)
  Using cached argparse-1.4.0-py2.py3-none-any.whl.metadata (2.8 kB)
Collecting mmh3 (from data-prep-toolkit==0.2.2.dev2->dpk_noop_transform_python==0.2.2.dev2)
  Using cached mmh3-5.0.1-cp311-cp311-macosx_11_0_arm64.whl.metadata (14 kB)
Collecting psutil (from data-prep-toolkit==0.2.2.dev2->dpk_noop_transform_python==0.2.2.dev2)
  Using cached psutil-6.1.0-cp36-abi3-macosx_11_0_arm64.whl.metadata (22 kB)
Collecting botocore<1.35.0,>=1.34.69 (from boto3==1.34.69->data-prep-toolkit==0.2.2.dev2->dpk_noop_transform_python==0.2.2.dev2)
  Using cached botocore-1.34.162-py3-none-any.whl.metadata (5.7 kB)
Collecting jmespath<2.0.0,>=0.7.1 (from boto3==1.34.69->data-prep-toolkit==0.2.2.dev2->dpk_noop_transform_python==0.2.2.dev2)
  Using cached jmespath-1.0.1-py3-none-any.whl.metadata (7.6 kB)
Collecting s3transfer<0.11.0,>=0.10.0 (from boto3==1.34.69->data-prep-toolkit==0.2.2.dev2->dpk_noop_transform_python==0.2.2.dev2)
  Using cached s3transfer-0.10.3-py3-none-any.whl.metadata (1.7 kB)
Collecting python-dateutil<3.0.0,>=2.1 (from botocore<1.35.0,>=1.34.69->boto3==1.34.69->data-prep-toolkit==0.2.2.dev2->dpk_noop_transform_python==0.2.2.dev2)
  Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata (8.4 kB)
Collecting urllib3!=2.2.0,<3,>=1.25.4 (from botocore<1.35.0,>=1.34.69->boto3==1.34.69->data-prep-toolkit==0.2.2.dev2->dpk_noop_transform_python==0.2.2.dev2)
  Using cached urllib3-2.2.3-py3-none-any.whl.metadata (6.5 kB)
Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1->botocore<1.35.0,>=1.34.69->boto3==1.34.69->data-prep-toolkit==0.2.2.dev2->dpk_noop_transform_python==0.2.2.dev2)
  Using cached six-1.16.0-py2.py3-none-any.whl.metadata (1.8 kB)
Using cached data_prep_toolkit-0.2.2.dev2-py3-none-any.whl (112 kB)
Using cached boto3-1.34.69-py3-none-any.whl (139 kB)
Using cached pyarrow-16.1.0-cp311-cp311-macosx_11_0_arm64.whl (26.0 MB)
Using cached numpy-1.26.4-cp311-cp311-macosx_11_0_arm64.whl (14.0 MB)
Using cached argparse-1.4.0-py2.py3-none-any.whl (23 kB)
Using cached mmh3-5.0.1-cp311-cp311-macosx_11_0_arm64.whl (38 kB)
Using cached psutil-6.1.0-cp36-abi3-macosx_11_0_arm64.whl (248 kB)
Using cached botocore-1.34.162-py3-none-any.whl (12.5 MB)
Using cached jmespath-1.0.1-py3-none-any.whl (20 kB)
Using cached s3transfer-0.10.3-py3-none-any.whl (82 kB)
Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)
Using cached urllib3-2.2.3-py3-none-any.whl (126 kB)
Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Building wheels for collected packages: dpk_noop_transform_python
  Building editable for dpk_noop_transform_python (pyproject.toml) ... done
  Created wheel for dpk_noop_transform_python: filename=dpk_noop_transform_python-0.2.2.dev2-0.editable-py3-none-any.whl size=2869 sha256=ec81b8ce21ee7f5552da9d85f251ea7ef3ad3ab30a31bdc43bb6df7e446b25b9
  Stored in directory: /private/var/folders/tc/l4tdn6zn1q57q5cw4vqzlgxr0000gn/T/pip-ephem-wheel-cache-mnu8vz8_/wheels/e4/f0/0f/56cf21232c3479d893ef41592bbce7eaa0b45e7b756d2f7522
Successfully built dpk_noop_transform_python
Installing collected packages: argparse, urllib3, six, psutil, numpy, mmh3, jmespath, python-dateutil, pyarrow, botocore, s3transfer, boto3, data-prep-toolkit, dpk_noop_transform_python
Successfully installed argparse-1.4.0 boto3-1.34.69 botocore-1.34.162 data-prep-toolkit-0.2.2.dev2 dpk_noop_transform_python-0.2.2.dev2 jmespath-1.0.1 mmh3-5.0.1 numpy-1.26.4 psutil-6.1.0 pyarrow-16.1.0 python-dateutil-2.9.0.post0 s3transfer-0.10.3 six-1.16.0 urllib3-2.2.3

Anything else

No response

OS

MacOS (limited support)

Python

3.11.x

Are you willing to submit a PR?

[ ] Yes I am willing to submit a PR!

daw3rd commented 5 days ago

In addition, make test-src and ci/cd is passing since 0.2.2.dev2 is the current repo dev branch version, but is also in pypi, It seems that after the above failure to install the data-prep-toolkit==0.2.2.dev from repo source in data-processing-lib dirs into the venv, it is falling back to pypi for the data-prep-toolkit-0.2.2.dev2 requirement.

daw3rd commented 5 days ago

This did not seem to happen on a newly cloned repo, so will close this for now.

IBM / data-prep-kit