search

ComparativeGenomicsToolkit / cactus

Official home of genome aligner based upon notion of Cactus graphs
Other
499 stars 109 forks source link

Installation from release failed with python3.11 #1135

Closed jowodo closed 10 months ago

jowodo commented 1 year ago

I'm on a HPC system with modules. I've managed to install with python version 3.8, 3.9, 3.10 but not python3.11. This is how I attempted installation: 

PYTHONVERSION=3.11.4
INSTALLVERSION=2.6.6
INSTALLDIR=/app/cactus/$INSTALLVERSION-$PYTHONVERSION
TD=$(mktemp -d)
pushd $TD
URL="https://github.com/ComparativeGenomicsToolkit/cactus/releases/download/v$INSTALLVERSION/cactus-bin-v${INSTALLVERSION}.tar.gz"
wget -q $URL
tar xf cactus-bin-v${INSTALLVERSION}.tar.gz
#cd cactus-bin-v${INSTALLVERSION}
mv cactus-bin-v${INSTALLVERSION} $INSTALLDIR
cd $INSTALLDIR
python3 -m venv venv.cactus
export PATH=$(pwd)/bin:$PATH
export PYTHONPATH=$(pwd)/lib:$PYTHONPATH
source venv.cactus/bin/activate
#python3 -m pip install -U setuptools pip # already part of base python installation 
python3 -m pip install -U .
python3 -m pip install -U -r ./toil-requirement.txt
cd bin
for i in wigToBigWig faToTwoBit bedToBigBed bigBedToBed axtChain pslPosTarget bedSort hgGcPercent mafToBigMaf hgLoadMafSummary
do
        wget -q http://hgdownload.cse.ucsc.edu/admin/exe/linux.x86_64/${i}
        chmod +x ${i}
done
This is the stdout and stderr combined: ``` Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done Collecting decorator (from Cactus==2.6.6) Using cached decorator-5.1.1-py3-none-any.whl (9.1 kB) Collecting networkx<2.8.1,>=2 (from Cactus==2.6.6) Using cached networkx-2.8-py3-none-any.whl (2.0 MB) Collecting pytest (from Cactus==2.6.6) Using cached pytest-7.4.0-py3-none-any.whl (323 kB) Collecting cigar (from Cactus==2.6.6) Using cached cigar-0.1.3.tar.gz (7.0 kB) Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done Collecting biopython (from Cactus==2.6.6) Downloading biopython-1.81-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.1 MB) ________________________________________ 3.1/3.1 MB 77.4 MB/s eta 0:00:00 Collecting numpy (from biopython->Cactus==2.6.6) Downloading numpy-1.25.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.2 MB) ________________________________________ 18.2/18.2 MB 143.9 MB/s eta 0:00:00 Collecting iniconfig (from pytest->Cactus==2.6.6) Using cached iniconfig-2.0.0-py3-none-any.whl (5.9 kB) Collecting packaging (from pytest->Cactus==2.6.6) Using cached packaging-23.1-py3-none-any.whl (48 kB) Collecting pluggy<2.0,>=0.12 (from pytest->Cactus==2.6.6) Using cached pluggy-1.2.0-py3-none-any.whl (17 kB) Building wheels for collected packages: Cactus, cigar Building wheel for Cactus (pyproject.toml) ... error error: subprocess-exited-with-error _ Building wheel for Cactus (pyproject.toml) did not run successfully. _ exit code: 1 __> [140 lines of output] running bdist_wheel running build running build_py creating build creating build/lib creating build/lib/cactus copying src/cactus/__init__.py -> build/lib/cactus creating build/lib/cactus/paf copying src/cactus/paf/pafTest.py -> build/lib/cactus/paf copying src/cactus/paf/__init__.py -> build/lib/cactus/paf copying src/cactus/paf/paf.py -> build/lib/cactus/paf copying src/cactus/paf/local_alignment.py -> build/lib/cactus/paf creating build/lib/cactus/maf copying src/cactus/maf/cactus_hal2chains.py -> build/lib/cactus/maf copying src/cactus/maf/cactus_hal2maf.py -> build/lib/cactus/maf copying src/cactus/maf/__init__.py -> build/lib/cactus/maf copying src/cactus/maf/cactus_maf2bigmaf.py -> build/lib/cactus/maf creating build/lib/cactus/update copying src/cactus/update/cactus_update_prepare.py -> build/lib/cactus/update copying src/cactus/update/__init__.py -> build/lib/cactus/update creating build/lib/cactus/refmap copying src/cactus/refmap/cactus_minigraph.py -> build/lib/cactus/refmap copying src/cactus/refmap/paf_to_lastz.py -> build/lib/cactus/refmap copying src/cactus/refmap/cactus_graphmap.py -> build/lib/cactus/refmap copying src/cactus/refmap/apply_dipcall_bed_filter.py -> build/lib/cactus/refmap copying src/cactus/refmap/cactus_pangenome.py -> build/lib/cactus/refmap copying src/cactus/refmap/cactus_graphmap_split.py -> build/lib/cactus/refmap copying src/cactus/refmap/__init__.py -> build/lib/cactus/refmap copying src/cactus/refmap/fasta_preprocessing.py -> build/lib/cactus/refmap copying src/cactus/refmap/cactus_refmap.py -> build/lib/cactus/refmap copying src/cactus/refmap/cactus_graphmap_join.py -> build/lib/cactus/refmap creating build/lib/cactus/setup copying src/cactus/setup/cactus_align.py -> build/lib/cactus/setup copying src/cactus/setup/__init__.py -> build/lib/cactus/setup creating build/lib/cactus/blast copying src/cactus/blast/__init__.py -> build/lib/cactus/blast copying src/cactus/blast/cactus_blast.py -> build/lib/cactus/blast creating build/lib/cactus/shared copying src/cactus/shared/version.py -> build/lib/cactus/shared copying src/cactus/shared/test.py -> build/lib/cactus/shared copying src/cactus/shared/commonTest.py -> build/lib/cactus/shared copying src/cactus/shared/__init__.py -> build/lib/cactus/shared copying src/cactus/shared/common.py -> build/lib/cactus/shared copying src/cactus/shared/configWrapper.py -> build/lib/cactus/shared creating build/lib/cactus/hal copying src/cactus/hal/cactus_halTest.py -> build/lib/cactus/hal copying src/cactus/hal/__init__.py -> build/lib/cactus/hal creating build/lib/cactus/reference copying src/cactus/reference/__init__.py -> build/lib/cactus/reference copying src/cactus/reference/cactus_referenceTest.py -> build/lib/cactus/reference creating build/lib/cactus/preprocessor copying src/cactus/preprocessor/cutHeaders.py -> build/lib/cactus/preprocessor copying src/cactus/preprocessor/checkUniqueHeaders.py -> build/lib/cactus/preprocessor copying src/cactus/preprocessor/cactus_preprocessorTest.py -> build/lib/cactus/preprocessor copying src/cactus/preprocessor/fileMasking.py -> build/lib/cactus/preprocessor copying src/cactus/preprocessor/dnabrnnMasking.py -> build/lib/cactus/preprocessor copying src/cactus/preprocessor/__init__.py -> build/lib/cactus/preprocessor copying src/cactus/preprocessor/preprocessorTest.py -> build/lib/cactus/preprocessor copying src/cactus/preprocessor/cactus_preprocessor.py -> build/lib/cactus/preprocessor creating build/lib/cactus/progressive copying src/cactus/progressive/cactus_progressive.py -> build/lib/cactus/progressive copying src/cactus/progressive/cactus_constructFromIntermediates.py -> build/lib/cactus/progressive copying src/cactus/progressive/outgroup.py -> build/lib/cactus/progressive copying src/cactus/progressive/__init__.py -> build/lib/cactus/progressive copying src/cactus/progressive/cactus_terra_helper.py -> build/lib/cactus/progressive copying src/cactus/progressive/multiCactusTreeTest.py -> build/lib/cactus/progressive copying src/cactus/progressive/cactus_prepare.py -> build/lib/cactus/progressive copying src/cactus/progressive/multiCactusTree.py -> build/lib/cactus/progressive copying src/cactus/progressive/cactus_progressiveTest.py -> build/lib/cactus/progressive copying src/cactus/progressive/outgroupTest.py -> build/lib/cactus/progressive copying src/cactus/progressive/progressive_decomposition.py -> build/lib/cactus/progressive copying src/cactus/progressive/seqFile.py -> build/lib/cactus/progressive creating build/lib/cactus/pipeline copying src/cactus/pipeline/cactus_evolverTest.py -> build/lib/cactus/pipeline copying src/cactus/pipeline/cactus_workflow.py -> build/lib/cactus/pipeline copying src/cactus/pipeline/cactus_workflowTest.py -> build/lib/cactus/pipeline copying src/cactus/pipeline/__init__.py -> build/lib/cactus/pipeline creating build/lib/cactus/preprocessor/lastzRepeatMasking copying src/cactus/preprocessor/lastzRepeatMasking/cactus_lastzRepeatMask.py -> build/lib/cactus/preprocessor/lastzRepeatMasking copying src/cactus/preprocessor/lastzRepeatMasking/__init__.py -> build/lib/cactus/preprocessor/lastzRepeatMasking copying src/cactus/preprocessor/lastzRepeatMasking/cactus_lastzRepeatMaskTest.py -> build/lib/cactus/preprocessor/lastzRepeatMasking running egg_info writing src/Cactus.egg-info/PKG-INFO writing dependency_links to src/Cactus.egg-info/dependency_links.txt writing entry points to src/Cactus.egg-info/entry_points.txt writing requirements to src/Cactus.egg-info/requires.txt writing top-level names to src/Cactus.egg-info/top_level.txt reading manifest file 'src/Cactus.egg-info/SOURCES.txt' adding license file 'LICENSE.txt' writing manifest file 'src/Cactus.egg-info/SOURCES.txt' copying src/cactus/attcc-alpha.knm -> build/lib/cactus copying src/cactus/cactus_progressive_config.xml -> build/lib/cactus installing to build/bdist.linux-x86_64/wheel running install /app/cactus/2.6.6-3.11.4/venv.cactus/bin/python3: No module named pip Traceback (most recent call last): File "/app/cactus/2.6.6-3.11.4/venv.cactus/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in main() File "/app/cactus/2.6.6-3.11.4/venv.cactus/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main json_out['return_val'] = hook(**hook_input['kwargs']) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/cactus/2.6.6-3.11.4/venv.cactus/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel return _build_backend().build_wheel(wheel_directory, config_settings, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/tmp/pip-build-env-_8tbkyes/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 434, in build_wheel return self._build_with_temp_dir( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/tmp/pip-build-env-_8tbkyes/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 419, in _build_with_temp_dir self.run_setup() File "/tmp/pip-build-env-_8tbkyes/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 507, in run_setup super(_BuildMetaLegacyBackend, self).run_setup(setup_script=setup_script) File "/tmp/pip-build-env-_8tbkyes/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 341, in run_setup exec(code, locals()) File "", line 25, in File "/tmp/pip-build-env-_8tbkyes/overlay/lib/python3.11/site-packages/setuptools/__init__.py", line 107, in setup return distutils.core.setup(**attrs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/tmp/pip-build-env-_8tbkyes/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 185, in setup return run_commands(dist) ^^^^^^^^^^^^^^^^^^ File "/tmp/pip-build-env-_8tbkyes/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 201, in run_commands dist.run_commands() File "/tmp/pip-build-env-_8tbkyes/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands self.run_command(cmd) File "/tmp/pip-build-env-_8tbkyes/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 1233, in run_command super().run_command(command) File "/tmp/pip-build-env-_8tbkyes/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command cmd_obj.run() File "/tmp/pip-build-env-_8tbkyes/overlay/lib/python3.11/site-packages/wheel/bdist_wheel.py", line 384, in run self.run_command("install") File "/tmp/pip-build-env-_8tbkyes/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command self.distribution.run_command(command) File "/tmp/pip-build-env-_8tbkyes/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 1233, in run_command super().run_command(command) File "/tmp/pip-build-env-_8tbkyes/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command cmd_obj.run() File "", line 22, in run File "", line 22, in run File "/app/python3/3.11.4/lib/python3.11/subprocess.py", line 571, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['/app/cactus/2.6.6-3.11.4/venv.cactus/bin/python3', '-m', 'pip', 'install', 'submodules/sonLib']' returned non-zero exit status 1. [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for Cactus Building wheel for cigar (pyproject.toml) ... done Created wheel for cigar: filename=cigar-0.1.3-py3-none-any.whl size=4019 sha256=02c15dfb2c39746858aaa45a5c8fa9bf6ce2ea7cc42f3c8f06b5f73578f62cd0 Stored in directory: /home/user/.cache/pip/wheels/f9/f0/1a/b26850ce8988ee2241b3b83dbb76688daa441d76d58b3ab932 Successfully built cigar Failed to build Cactus ERROR: Could not build wheels for Cactus, which is required to install pyproject.toml-based projects [notice] A new release of pip is available: 23.1.2 -> 23.2.1 [notice] To update, run: pip install --upgrade pip ```

I'm happy to provide more information if needed.

glennhickey commented 1 year ago

It looks like you're commenting out python3 -m pip install pip

#python3 -m pip install -U setuptools pip # already part of base python installation

but then it's failing with 

/app/cactus/2.6.6-3.11.4/venv.cactus/bin/python3: No module named pip

Please follow the installation instructions exactly when installing Cactus.

jowodo commented 1 year ago

Thanks for the swift reply. I get the same error when I update setuptools and pip ...

glennhickey commented 1 year ago

I just tried installing Cactus in Python3.11 and didn't run into any issues. So there aren't any obvious conflicts between Cactus and this Python version that I can see.

I'm noticing your command to create the virtualenv is also different from the instructions, but have no idea if that's related to your issue. Don't think I have anything else to suggest though, apart from sticking with a python version that works for you.

For the record, to test on 3.11 I ran

docker run -it --rm ubuntu:22.04

# setup python3.11 on ubuntu
apt update && apt upgrade -y
apt install -y wget build-essential git software-properties-common
add-apt-repository ppa:deadsnakes/ppa
apt install -y python3.11-dev python3-virtualenv

# get the latest cactus release
wget -q https://github.com/ComparativeGenomicsToolkit/cactus/releases/download/v2.6.7/cactus-bin-v2.6.7.tar.gz
tar -xzf cactus-bin-v2.6.7.tar.gz
cd cactus-bin-v2.6.7

# some weird ownership thing I've never seen before. don't think it's pertinent to this issue
# todo: can the release be fixed to prevent it?
git config --global --add safe.directory /cactus-bin-v2.6.7

# set up Cactus (note I'm using python3.11 on the line below to force it to use this version)
virtualenv -p python3.11 venv-cactus-v2.6.7
printf "export PATH=$(pwd)/bin:\$PATH\nexport PYTHONPATH=$(pwd)/lib:\$PYTHONPATH\n" >> venv-cactus-v2.6.7/bin/activate
source venv-cactus-v2.6.7/bin/activate
python3 -m pip install -U setuptools pip
python3 -m pip install -U .
python3 -m pip install -U -r ./toil-requirement.txt

python3 --version
Python 3.11.4
jowodo commented 10 months ago

I resolved the issue with the following:

python3 -m pip install -U setuptools pip wheel 
python3 -m pip install -U .
python3 -m pip install -U -r ./toil-requirement.txt

It is weird that installation without wheel is depricated, but wheel is not installed per default in an virtual environment ...

jowodo commented 10 months ago

Maybe you can add wheel to README.md like this

python3 -m pip install -U setuptools pip wheel
jowodo commented 10 months ago

awesome! thanks