Open ns-rse opened 3 weeks ago
Acribbs
commented
3 weeks ago oh I see what the potential problem is... im not sure why our testing didnt pick this up. In the latest release connected_components.h isnt in the setup.py. In the meantime can you use the 0.7.2 version?
Acribbs
commented
3 weeks ago I will fix this in next few days and make a new release
Acribbs
commented
3 weeks ago Iv pulled the release for the moment
Acribbs
commented
3 weeks ago you able to test this pull, it should fix the issues? https://github.com/cgat-developers/cgat-apps/pull/135
ns-rse
commented
1 week ago Sorry for the delay, only work on the project that uses this part time and have been on annual leave.
In a clean environment this now installs...
(tmp-3c6c4b33689cb6a) ❱ git clone https://github.com/cgat-developers/cgat-apps.git
Cloning into 'cgat-apps'...
remote: Enumerating objects: 39732, done.
remote: Counting objects: 100% (1615/1615), done.
remote: Compressing objects: 100% (775/775), done.
pReceiving objects: 13% (5474/39732), 25.81 MiB | 5.33 MiB/s/s/s
remote: Total 39732 (delta 784), reused 1355 (delta 725), pack-reused 38117 (from 1)
Receiving objects: 100% (39732/39732), 390.89 MiB | 4.12 MiB/s, done.
Resolving deltas: 100% (24717/24717), done.
(tmp-3c6c4b33689cb6a) ❱ cd cgat-apps
(tmp-3c6c4b33689cb6a) ❱ git switch AC-fixsetup
branch 'AC-fixsetup' set up to track 'origin/AC-fixsetup'.
Switched to a new branch 'AC-fixsetup'
(tmp-3c6c4b33689cb6a) ❱ pip install .
Processing /home/neil/tmp/cgat/cgat-apps
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: cgat
Building wheel for cgat (pyproject.toml) ... \
done
Created wheel for cgat: filename=cgat-0.7.4-cp312-cp312-linux_x86_64.whl size=10638983 sha256=a0474c183080bdfadba0946cd20ce9c4539f6b88eb15f40f7586668822453751
Stored in directory: /home/neil/.cache/pip/wheels/e2/2f/5d/1497f4f335bec91ea7fb6335d7da2df5e56a38796409b114e2
Successfully built cgat
Installing collected packages: cgat
Successfully installed cgat-0.7.4I'm getting bitten by this in CI now as cgat-0.7.2 is looking for numpy and failing to find it, despite it explicitly being installed first.
Log file can be found here but since they disappear after a while the output from pip install is shown below.
I've highlighted where numpy is installed prior the attempt to install cgat.
Run pip install --upgrade virtualenv
pip install --upgrade virtualenv
pip install --upgrade pip setuptools
virtualenv --upgrade-embed-wheels
pip install numpy
pip install -e .[tests]
shell: /usr/bin/bash -e {0}
env:
pythonLocation: /opt/hostedtoolcache/Python/3.1[2](https://github.com/sudlab/IsoSLAM/actions/runs/11953428873/job/33321276774?pr=62#step:4:2).7/x64
PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/[3](https://github.com/sudlab/IsoSLAM/actions/runs/11953428873/job/33321276774?pr=62#step:4:3).12.7/x64/lib/pkgconfig
Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.12.7/x6[4](https://github.com/sudlab/IsoSLAM/actions/runs/11953428873/job/33321276774?pr=62#step:4:4)
Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.12.7/x64
Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.12.7/x64
LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.12.7/x64/lib
Collecting virtualenv
Downloading virtualenv-20.27.1-py3-none-any.whl.metadata (4.[5](https://github.com/sudlab/IsoSLAM/actions/runs/11953428873/job/33321276774?pr=62#step:4:5) kB)
Collecting distlib<1,>=0.3.7 (from virtualenv)
Downloading distlib-0.3.9-py2.py3-none-any.whl.metadata (5.2 kB)
Collecting filelock<4,>=3.12.2 (from virtualenv)
Downloading filelock-3.1[6](https://github.com/sudlab/IsoSLAM/actions/runs/11953428873/job/33321276774?pr=62#step:4:6).1-py3-none-any.whl.metadata (2.9 kB)
Collecting platformdirs<5,>=3.9.1 (from virtualenv)
Downloading platformdirs-4.3.6-py3-none-any.whl.metadata (11 kB)
Downloading virtualenv-20.2[7](https://github.com/sudlab/IsoSLAM/actions/runs/11953428873/job/33321276774?pr=62#step:4:7).1-py3-none-any.whl (3.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.1/3.1 MB 60.[8](https://github.com/sudlab/IsoSLAM/actions/runs/11953428873/job/33321276774?pr=62#step:4:8) MB/s eta 0:00:00
Downloading distlib-0.3.[9](https://github.com/sudlab/IsoSLAM/actions/runs/11953428873/job/33321276774?pr=62#step:4:9)-py2.py3-none-any.whl (468 kB)
Downloading filelock-3.16.1-py3-none-any.whl (16 kB)
Downloading platformdirs-4.3.6-py3-none-any.whl (18 kB)
Installing collected packages: distlib, platformdirs, filelock, virtualenv
Successfully installed distlib-0.3.9 filelock-3.16.1 platformdirs-4.3.6 virtualenv-20.27.1
Requirement already satisfied: pip in /opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages (24.3.1)
Collecting setuptools
Downloading setuptools-75.6.0-py3-none-any.whl.metadata (6.7 kB)
Downloading setuptools-75.6.0-py3-none-any.whl (1.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 35.9 MB/s eta 0:00:00
Installing collected packages: setuptools
Successfully installed setuptools-75.6.0
upgrade pip for python 3.8 with current pip-24.3.1-py3-none-any.whl
upgrade setuptools for python 3.8 with current setuptools-75.2.0-py3-none-any.whl
upgrade wheel for python 3.8 with current wheel-0.44.0-py3-none-any.whl
upgrade pip for python 3.9 with current pip-24.3.1-py3-none-any.whl
upgrade setuptools for python 3.9 with current setuptools-75.2.0-py3-none-any.whl
upgrade wheel for python 3.9 with current wheel-0.44.0-py3-none-any.whl
upgrade pip for python 3.[10](https://github.com/sudlab/IsoSLAM/actions/runs/11953428873/job/33321276774?pr=62#step:4:10) with current pip-24.3.1-py3-none-any.whl
upgrade setuptools for python 3.10 with current setuptools-75.2.0-py3-none-any.whl
upgrade wheel for python 3.10 with current wheel-0.44.0-py3-none-any.whl
upgrade pip for python 3.[11](https://github.com/sudlab/IsoSLAM/actions/runs/11953428873/job/33321276774?pr=62#step:4:11) with current pip-24.3.1-py3-none-any.whl
upgrade setuptools for python 3.11 with current setuptools-75.2.0-py3-none-any.whl
upgrade wheel for python 3.11 with current wheel-0.44.0-py3-none-any.whl
upgrade pip for python 3.[12](https://github.com/sudlab/IsoSLAM/actions/runs/11953428873/job/33321276774?pr=62#step:4:12) with current pip-24.3.1-py3-none-any.whl
upgrade setuptools for python 3.12 with current setuptools-75.2.0-py3-none-any.whl
upgrade wheel for python 3.12 with current wheel-0.44.0-py3-none-any.whl
upgrade pip for python 3.[13](https://github.com/sudlab/IsoSLAM/actions/runs/11953428873/job/33321276774?pr=62#step:4:13) with current pip-24.3.1-py3-none-any.whl
upgrade setuptools for python 3.13 with current setuptools-75.2.0-py3-none-any.whl
upgrade wheel for python 3.13 with current wheel-0.44.0-py3-none-any.whl
upgrade pip for python 3.[14](https://github.com/sudlab/IsoSLAM/actions/runs/11953428873/job/33321276774?pr=62#step:4:14) with current pip-24.3.1-py3-none-any.whl
upgrade setuptools for python 3.14 with current setuptools-75.2.0-py3-none-any.whl
upgrade wheel for python 3.14 with current wheel-0.44.0-py3-none-any.whl
upgraded wheel for python 3.8 in 0:00:13.481616 new entries found:
NewVersion(filename=wheel-0.45.0-py3-none-any.whl), found_date=2024-11-21 14:12:37.776518+00:00, release_date=2024-11-08 21:44:59+00:00, source=manual)
upgraded pip for python 3.10 in 0:00:13.508307 new entries found:
NewVersion(filename=pip-24.3.1-py3-none-any.whl), found_date=2024-11-21 14:12:37.763969+00:00, release_date=2024-10-27 18:35:53+00:00, source=manual)
upgraded wheel for python 3.13 in 0:00:13.441020 new entries found:
NewVersion(filename=wheel-0.45.0-py3-none-any.whl), found_date=2024-11-21 14:12:37.895536+00:00, release_date=2024-11-08 21:44:59+00:00, source=manual)
upgraded wheel for python 3.11 in 0:00:13.675127 new entries found:
NewVersion(filename=wheel-0.45.0-py3-none-any.whl), found_date=2024-11-21 14:12:37.819624+00:00, release_date=2024-11-08 21:44:59+00:00, source=manual)
upgraded pip for python 3.8 in 0:00:13.770411 new entries found:
NewVersion(filename=pip-24.3.1-py3-none-any.whl), found_date=2024-11-21 14:12:37.771740+00:00, release_date=2024-10-27 18:35:53+00:00, source=manual)
upgraded pip for python 3.9 in 0:00:13.775614 new entries found:
NewVersion(filename=pip-24.3.1-py3-none-any.whl), found_date=2024-11-21 14:12:37.764774+00:00, release_date=2024-10-27 18:35:53+00:00, source=manual)
upgraded wheel for python 3.10 in 0:00:13.841341 new entries found:
NewVersion(filename=wheel-0.45.0-py3-none-any.whl), found_date=2024-11-21 14:12:37.775167+00:00, release_date=2024-11-08 21:44:59+00:00, source=manual)
upgraded pip for python 3.13 in 0:00:13.76[15](https://github.com/sudlab/IsoSLAM/actions/runs/11953428873/job/33321276774?pr=62#step:4:16)38 new entries found:
NewVersion(filename=pip-24.3.1-py3-none-any.whl), found_date=2024-11-21 14:12:37.868022+00:00, release_date=2024-10-27 18:35:53+00:00, source=manual)
upgraded wheel for python 3.12 in 0:00:13.790123 new entries found:
NewVersion(filename=wheel-0.45.0-py3-none-any.whl), found_date=2024-11-21 14:12:37.848349+00:00, release_date=2024-11-08 21:44:59+00:00, source=manual)
upgraded wheel for python 3.9 in 0:00:13.875404 new entries found:
NewVersion(filename=wheel-0.45.0-py3-none-any.whl), found_date=2024-11-21 14:12:37.769526+00:00, release_date=2024-11-08 21:44:59+00:00, source=manual)
upgraded pip for python 3.12 in 0:00:13.868303 new entries found:
NewVersion(filename=pip-24.3.1-py3-none-any.whl), found_date=2024-11-21 14:12:37.821266+00:00, release_date=2024-10-27 18:35:53+00:00, source=manual)
upgraded wheel for python 3.14 in 0:00:13.809338 new entries found:
NewVersion(filename=wheel-0.45.0-py3-none-any.whl), found_date=2024-11-21 14:12:37.935188+00:00, release_date=2024-11-08 21:44:59+00:00, source=manual)
upgraded pip for python 3.11 in 0:00:13.980762 new entries found:
NewVersion(filename=pip-24.3.1-py3-none-any.whl), found_date=2024-11-21 14:12:37.780362+00:00, release_date=2024-10-27 18:35:53+00:00, source=manual)
upgraded pip for python 3.14 in 0:00:13.901360 new entries found:
NewVersion(filename=pip-24.3.1-py3-none-any.whl), found_date=2024-11-21 14:12:37.915952+00:00, release_date=2024-10-27 18:35:53+00:00, source=manual)
upgraded setuptools for python 3.8 in 0:00:14.19[16](https://github.com/sudlab/IsoSLAM/actions/runs/11953428873/job/33321276774?pr=62#step:4:17)76 new entries found:
NewVersion(filename=setuptools-75.3.0-py3-none-any.whl), found_date=2024-11-21 14:12:37.769901+00:00, release_date=2024-10-29 10:23:24+00:00, source=manual)
upgraded setuptools for python 3.12 in 0:00:14.128076 new entries found:
NewVersion(filename=setuptools-75.6.0-py3-none-any.whl), found_date=2024-11-21 14:12:37.831347+00:00, release_date=2024-11-20 18:16:10+00:00, source=manual)
upgraded setuptools for python 3.13 in 0:00:14.155189 new entries found:
NewVersion(filename=setuptools-75.6.0-py3-none-any.whl), found_date=2024-11-21 14:12:37.887668+00:00, release_date=2024-11-20 18:16:10+00:00, source=manual)
upgraded setuptools for python 3.10 in 0:00:14.313731 new entries found:
NewVersion(filename=setuptools-75.6.0-py3-none-any.whl), found_date=2024-11-21 14:12:37.772635+00:00, release_date=2024-11-20 18:16:10+00:00, source=manual)
upgraded setuptools for python 3.9 in 0:00:14.332700 new entries found:
NewVersion(filename=setuptools-75.6.0-py3-none-any.whl), found_date=2024-11-21 14:12:37.7655[17](https://github.com/sudlab/IsoSLAM/actions/runs/11953428873/job/33321276774?pr=62#step:4:18)+00:00, release_date=2024-11-20 18:16:10+00:00, source=manual)
upgraded setuptools for python 3.11 in 0:00:14.324987 new entries found:
NewVersion(filename=setuptools-75.6.0-py3-none-any.whl), found_date=2024-11-21 14:12:37.788419+00:00, release_date=2024-11-20 [18](https://github.com/sudlab/IsoSLAM/actions/runs/11953428873/job/33321276774?pr=62#step:4:19):16:10+00:00, source=manual)
upgraded setuptools for python 3.14 in 0:00:14.225363 new entries found:
NewVersion(filename=setuptools-75.6.0-py3-none-any.whl), found_date=[20](https://github.com/sudlab/IsoSLAM/actions/runs/11953428873/job/33321276774?pr=62#step:4:21)24-11-21 14:12:37.926997+00:00, release_date=2024-11-20 18:16:10+00:00, source=manual)
SystemExit: None
Collecting numpy
Downloading numpy-2.1.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (62 kB)
Downloading numpy-2.1.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16.0/16.0 MB 161.4 MB/s eta 0:00:00
Installing collected packages: numpy
Successfully installed numpy-2.1.3 ############# Numpy is installed
Obtaining file:///home/runner/work/IsoSLAM/IsoSLAM
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Checking if build backend supports build_editable: started
Checking if build backend supports build_editable: finished with status 'done'
Getting requirements to build editable: started
Getting requirements to build editable: finished with status 'done'
Preparing editable metadata (pyproject.toml): started
Preparing editable metadata (pyproject.toml): finished with status 'done'
Collecting apsw (from isoslam==0.1.dev1+g57124b0)
Downloading apsw-3.47.0.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.4 kB)
Collecting cgat (from isoslam==0.1.dev1+g57124b0)
Downloading cgat-0.7.2.tar.gz (4[21](https://github.com/sudlab/IsoSLAM/actions/runs/11953428873/job/33321276774?pr=62#step:4:22) kB)
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'error'
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [26 lines of output]
Traceback (most recent call last):
File "<string>", line 15, in <module>
ModuleNotFoundError: No module named 'numpy'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
main()
File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
return hook(config_settings)
^^^^^^^^^^^^^^^^^^^^^
File "/tmp/pip-build-env-p_l_nrzv/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 334, in get_requires_for_build_wheel
return self._get_build_requires(config_settings, requirements=[])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/pip-build-env-p_l_nrzv/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 304, in _get_build_requires
self.run_setup()
File "/tmp/pip-build-env-p_l_nrzv/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 5[22](https://github.com/sudlab/IsoSLAM/actions/runs/11953428873/job/33321276774?pr=62#step:4:23), in run_setup
super().run_setup(setup_script=setup_script)
File "/tmp/pip-build-env-p_l_nrzv/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 320, in run_setup
exec(code, locals())
File "<string>", line 17, in <module>
ImportError: the CGAT code collection requires numpy to be installed before running setup.py (pip install numpy)
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
Error: Process completed with exit code 1.Looking at your github actions file, have you tried adding --no-build-isolation to the pip install cgat, looks like there may be an isolation issue occuring where cgat builds some modules that are already there. Could be wrong though.
Further to cgat-core #177 I'm still having trouble installing
cgaton my system.NB I've read the install instructions and have seen the preferred approach is to use Conda environments but would prefer to use plain virtual environments because of the recent furore and potential licensing issues associated with Conda.
In #177 I attempted to abstract out the problem from the
pyproject.tomlI mentioned and manually install packages and dependencies. I continue with that approach here.Downloading the source, extracting and attempting to install...
requires.txtLooking for
requires.txtin the package files.Searching for
requires.txtThe file doesn't exist.
Checking
cgat-0.7.2The file is present in the previous release, but its an empty file...
Workaround
Add an empty file to
cgat-0.7.4and its possible topip install ., but it fails further alongThere are some warnings about the
cgat.VCFTools/cgat.BamTools/cgat.Components/cgat.FastqToolsnot being correctly configured forsetuptoolsbut the install proceeds.Compilation of the C++ code fails because there is a missing header file
connected_components.h.Trying to install
cgat-0.7.2Dropping back a version I encounter again the Numpy issue described in #177 as the very first thing
setup.pydoes is check thatnumpycan be imported. It should be importable because I have explicitly installed it...However this is not detected correctly...
I decided to try using the Setuptools method of Declaring required dependency using
setup.cfg(although the current ecosystem is standardising on thepyproject.tomlformat as per PEP621).setup.cfgsetup.pyDisable the checks for these packages at the start[^1]
Try installing but it now doesn't install because an attempt is made to import from
Cython, which I have explicitly manually installed along withpysam, fails.Possible solutions?
connected_components.hseems like it would be needed to build and install the package (after adding back in an emptyrequires.txt.pyproject.tomlwould perhaps help with some of these problems assetup.pytries to import and use some of the packages it depends on before they have been pulled in and installed as dependencies.[^1]: I note the comments in
setup.pyaboutIs there a way to do this more elegantly?. The traditional method of specifying dependencies insetup.pyis within the call tosetup()seesetup.pytab here but as mentioned the preferred configuration is now viapyproject.tomland I think the reason this inelegant solution exists is because the packages are imported and used before they have been installed as dependencies.