Dependency make failure at canu

[ibharvey]

I have attempted to make the dependencies using your script on mutliple linux distros, but they all fail with the following error. Do you have a suggestion for how to overcome this?

(base)[harveyi@swift dep]$ make mkdir -p install_flags make -C conda make[1]: Entering directory '/home/harveyi/Documents/Bioinformatics/Programs/smrtsv2/dep/conda' Warning: 'defaults' already in 'channels' list, moving to the top Warning: 'bioconda' already in 'channels' list, moving to the top Warning: 'conda-forge' already in 'channels' list, moving to the top Collecting package metadata (current_repodata.json): done Solving environment: failed with initial frozen solve. Retrying with flexible solve. Collecting package metadata (repodata.json): done Solving environment: failed with initial frozen solve. Retrying with flexible solve. Solving environment: \ Found conflicts! Looking for incompatible packages. This can take several minutes. Press CTRL-C to abort. failed

UnsatisfiableError: The following specifications were found
to be incompatible with the existing python installation in your environment:

Specifications:

- canu=1.9 -> python[version='3.5.*|>=3.5,<3.6.0a0']

Your python: python=3.6.2

If python is on the left-most side of the chain, that's the version you've asked for. When python appears to the right, that indicates that the thing on the left is somehow not available for the python version you are constrained to. Note that conda will not change your python version to a different minor version unless you explicitly specify that.

make[1]: *** [build/install_flags/env_tools_install] Error 1 make[1]: Leaving directory '/home/harveyi/Documents/Bioinformatics/Programs/smrtsv2/dep/conda' make: *** [install_flags/dep_conda_build] Error 2

[paudano]

Run make with: --debug=v

I think it's failing building the tools environment, but I can't tell from the output. I don't know why it's trying to install Canu 1.9 when the tools environment requests 1.8.

Also, you are already in a conda environment (base), which you should not need. It may not be related, but I would exit all conda environments to be sure.

A similar problem was reported in #47, but I don't know what was done to fix it. I have not been able to reproduce the issue on my systems.

[ibharvey]

I deactivated my conda environment and recloned the smrtsv2 directory and here is the complete output from trying to make the dependencies.

20200131_BuildErrorLog.gz

[paudano]

The error makes it look like Canu 1.8 needs Python 3.6.0 or lower, but I can install with the build scripts, which specify Python 3.6.2.

Here's an abriged version of conda list after the install:

# packages in environment at /XXX/smrtsv2/dep/conda/build/envs/tools:
#
# Name                    Version                   Build  Channel
canu                      1.8                  he1b5a44_2    bioconda
python                    3.6.2                         0    conda-forge

You can try changing to python 3.6.0 in dep/conda/makefile (line 28, in Make target "build/install_flags/env_tools_deploy"):

bin/conda create --yes --name tools python=3.6.0

Or you can try changing to canu 1.9 (`dep/conda/install_tools.sh').

I don't know why this is failing in some environments and not others. I build in multiple environments and cannot replicate it.

Which environments are you building in? If you get it working, please let me know what you did.

[ibharvey]

Still failing. I'm primarily trying to get it to work on a Centos OS. Doing the python=3.6.0 swap caused a different set of errors.

date > build/install_flags/env_tools_deploy Warning: 'defaults' already in 'channels' list, moving to the top Warning: 'bioconda' already in 'channels' list, moving to the top Warning: 'conda-forge' already in 'channels' list, moving to the top Collecting package metadata (current_repodata.json): done Solving environment: failed with initial frozen solve. Retrying with flexible solve. Collecting package metadata (repodata.json): done Solving environment: failed with initial frozen solve. Retrying with flexible solve. Solving environment: - Found conflicts! Looking for incompatible packages. This can take several minutes. Press CTRL-C to abort. failed

UnsatisfiableError: The following specifications were found
to be incompatible with the existing python installation in your environment:

Specifications:

• canu=1.8 -> python[version='3.5.*|>=3.5,<3.6.0a0']

Your python: python=3.6.0

If python is on the left-most side of the chain, that's the version you've asked for. When python appears to the right, that indicates that the thing on the left is somehow not available for the python version you are constrained to. Note that conda will not change your python version to a different minor version unless you explicitly specify that.

The following specifications were found to be incompatible with each other:

Package tk conflicts for: boost=1.70.0 -> python[version='>=3.6,<3.7.0a0'] -> tk[version='8.5.|8.6.|>=8.6.10,<8.7.0a0|>=8.6.7,<8.7.0a0|>=8.6.8,<8.7.0a0|>=8.6.9,<8.7.0a0'] bcftools=1.9 -> curl[version='>=7.59.0,<8.0a0'] -> krb5[version='>=1.16.3,<1.17.0a0'] -> tk[version='8.6.|>=8.6.10,<8.7.0a0|>=8.6.7,<8.7.0a0|>=8.6.8,<8.7.0a0|>=8.6.9,<8.7.0a0'] python=3.6.0 -> tk=8.5 samtools=1.9 -> curl[version='>=7.59.0,<8.0a0'] -> krb5[version='>=1.16.3,<1.17.0a0'] -> tk[version='8.6.|>=8.6.10,<8.7.0a0|>=8.6.7,<8.7.0a0|>=8.6.8,<8.7.0a0|>=8.6.9,<8.7.0a0'] canu=1.8 -> gnuplot[version='>=5.0.5'] -> glib=2.55 -> python[version='>=3.8,<3.9.0a0'] -> tk[version='8.6.|>=8.6.10,<8.7.0a0|>=8.6.7,<8.7.0a0|>=8.6.8,<8.7.0a0|>=8.6.9,<8.7.0a0'] bedtools=2.29.0 -> curl[version='>=7.64.1,<8.0a0'] -> krb5[version='>=1.16.1,<1.17.0a0'] -> tk[version='>=8.6.10,<8.7.0a0|>=8.6.7,<8.7.0a0|>=8.6.8,<8.7.0a0|>=8.6.9,<8.7.0a0'] htslib=1.9 -> curl[version='>=7.64.0,<8.0a0'] -> krb5[version='>=1.16.3,<1.17.0a0'] -> tk[version='8.6.|>=8.6.10,<8.7.0a0|>=8.6.7,<8.7.0a0|>=8.6.8,<8.7.0a0|>=8.6.9,<8.7.0a0'] freebayes=1.3.1 -> python[version='>=3.6,<3.7.0a0'] -> tk[version='8.5.|8.6.|>=8.6.10,<8.7.0a0|>=8.6.7,<8.7.0a0|>=8.6.8,<8.7.0a0|>=8.6.9,<8.7.0a0'] Package sqlite conflicts for: boost=1.70.0 -> python[version='>=3.6,<3.7.0a0'] -> sqlite[version='3.13.|3.20.|>=3.20.1,<4.0a0|>=3.22.0,<4.0a0|>=3.23.1,<4.0a0|>=3.24.0,<4.0a0|>=3.25.1,<4.0a0|>=3.25.2,<4.0a0|>=3.25.3,<4.0a0|>=3.26.0,<4.0a0|>=3.27.2,<4.0a0|>=3.28.0,<4.0a0|>=3.29.0,<4.0a0|>=3.30.0,<4.0a0|>=3.30.1,<4.0a0'] freebayes=1.3.1 -> python[version='>=3.6,<3.7.0a0'] -> sqlite[version='3.13.|3.20.|>=3.20.1,<4.0a0|>=3.22.0,<4.0a0|>=3.23.1,<4.0a0|>=3.24.0,<4.0a0|>=3.25.1,<4.0a0|>=3.25.2,<4.0a0|>=3.25.3,<4.0a0|>=3.26.0,<4.0a0|>=3.27.2,<4.0a0|>=3.28.0,<4.0a0|>=3.29.0,<4.0a0|>=3.30.0,<4.0a0|>=3.30.1,<4.0a0'] python=3.6.0 -> sqlite=3.13 canu=1.8 -> gnuplot[version='>=5.0.5'] -> glib=2.55 -> python[version='>=3.8,<3.9.0a0'] -> sqlite[version='3.20.*|>=3.23.1,<4.0a0|>=3.25.1,<4.0a0|>=3.25.2,<4.0a0|>=3.27.2,<4.0a0|>=3.30.0,<4.0a0'] canu=1.8 -> gnuplot[version='>=5.0.5'] -> qt=5.6 -> sqlite[version='>=3.20.1,<4.0a0|>=3.22.0,<4.0a0|>=3.24.0,<4.0a0|>=3.25.3,<4.0a0|>=3.26.0,<4.0a0|>=3.28.0,<4.0a0|>=3.29.0,<4.0a0|>=3.30.1,<4.0a0'] Package atk conflicts for: canu=1.8 -> gnuplot[version='>=5.0.5'] -> gtk2 -> atk[version='>=2.32.0,<3.0a0'] Package gtk2 conflicts for: canu=1.8 -> gnuplot[version='>=5.0.5'] -> gtk2

make[1]: [build/install_flags/env_tools_install] Error 1 make[1]: Leaving directory `/data/fremont/harveyi/Bioinformatics/Programs/smrtsv2/dep/conda' make: [install_flags/dep_conda_build] Error 2

[paudano]

Can you try changing canu=1.8 to canu=1.8=he1b5a44_2 in dep/conda/install_tools.sh and see what that does? Leave python on 3.6.2.

Also, if you see this in an environment I can reproduce it in, like an AWS AMI, please let me know!

My apologies for being so slow to respond. I am balancing too many things, but I'd like to get this solved.

[gkaur]

I had the similar problem. I was able to install everything correctly by reverting miniconda to an older version (Miniconda3-4.7.12 to Miniconda3-4.6.14). I saw at other places on internet that version 4.7 has had problems while resolving environment variables.

In the file dep/conda/Makefile : line 4 I replaced: MINICONDA_INSTALLER = Miniconda3-4.7.12-Linux-x86_64.sh with MINICONDA_INSTALLER = Miniconda3-4.6.14-Linux-x86_64.sh

[osowiecki]

I had the similar problem. I was able to install everything correctly by reverting miniconda to an older version (Miniconda3-4.7.12 to Miniconda3-4.6.14). I saw at other places on internet that version 4.7 has had problems while resolving environment variables.

In the file dep/conda/Makefile : line 4 I replaced: MINICONDA_INSTALLER = Miniconda3-4.7.12-Linux-x86_64.sh with MINICONDA_INSTALLER = Miniconda3-4.6.14-Linux-x86_64.sh

https://stackoverflow.com/questions/59474533/modulenotfounderror-no-module-named-numpy-testing-nosetester

ModuleNotFoundError in line 18 of /media/raid/smrtsv2/rules/genotype.snakefile: No module named 'numpy.testing.nosetester' File "/media/raid/smrtsv2/rules/genotype.snakefile", line 18, in File "/media/raid/smrtsv2/smrtsvlib/ml.py", line 10, in File "/media/raid/smrtsv2/dep/conda/build/envs/python3/lib/python3.6/site-packages/sklearn/model_selection/init.py", line 19, in File "/media/raid/smrtsv2/dep/conda/build/envs/python3/lib/python3.6/site-packages/sklearn/model_selection/_validation.py", line 28, in File "/media/raid/smrtsv2/dep/conda/build/envs/python3/lib/python3.6/site-packages/sklearn/metrics/init.py", line 7, in File "/media/raid/smrtsv2/dep/conda/build/envs/python3/lib/python3.6/site-packages/sklearn/metrics/ranking.py", line 25, in File "/media/raid/smrtsv2/dep/conda/build/envs/python3/lib/python3.6/site-packages/scipy/stats/init.py", line 348, in File "/media/raid/smrtsv2/dep/conda/build/envs/python3/lib/python3.6/site-packages/scipy/stats/stats.py", line 177, in File "/media/raid/smrtsv2/dep/conda/build/envs/python3/lib/python3.6/site-packages/scipy/stats/distributions.py", line 13, in File "/media/raid/smrtsv2/dep/conda/build/envs/python3/lib/python3.6/site-packages/scipy/stats/_continuous_distns.py", line 15, in File "/media/raid/smrtsv2/dep/conda/build/envs/python3/lib/python3.6/site-packages/scipy/_lib/_numpy_compat.py", line 10, in