search

dib-lab / elvers

(formerly eelpond) an automated RNA-Seq workflow system
https://dib-lab.github.io/elvers/
Other
28 stars 3 forks source link

Pinning version numbers in rule environment.yml files #135

Closed charlesreid1 closed 5 years ago

charlesreid1 commented 5 years ago

I am trying to pin version numbers in the environment.yml files. But when I double-check that the environments work by running the command

conda env create --file elvers/rules/<rule-name>/environment.yml -n <env-name>

I run into some problems with the following software:

All due to various issues (python not being found, mismatched python version, specified software version can't be found, etc.)

I will include the output of the above command running with unmodified environment.yml files and check to make sure those are all okay and that this is not an issue with my conda setup.

Output included below.

bowtie works:

+ conda env create --file elvers/rules/bowtie2/environment.yml -n bowtie2_test
Solving environment: ...working... done
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done

==> WARNING: A newer version of conda exists. <==
  current version: 4.5.12
  latest version: 4.6.12

Please update conda by running

    $ conda update -n base conda

#
# To activate this environment, use
#
#     $ conda activate bowtie2_test
#
# To deactivate an active environment, use
#
#     $ conda deactivate

dammit does not work:

+ conda env create --file elvers/rules/dammit/environment.yml -n dammit_test
Solving environment: ...working... done
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... failed

==> WARNING: A newer version of conda exists. <==
  current version: 4.5.12
  latest version: 4.6.12

Please update conda by running

    $ conda update -n base conda

ERROR conda.core.link:_execute(507): An error occurred while installing package 'conda-forge::atomicwrites-1.3.0-py_0'.
FileNotFoundError(2, "No such file or directory: '/Users/charles/.pyenv/versions/miniconda3-4.3.30/envs/dammit_test/bin/python3.6'")
Attempting to roll back.

Rolling back transaction: ...working... done

FileNotFoundError(2, "No such file or directory: '/Users/charles/.pyenv/versions/miniconda3-4.3.30/envs/dammit_test/bin/python3.6'")

deseq2 works:


+ conda env create --file elvers/rules/deseq2/environment.yml -n deseq2_test
Solving environment: ...working... done
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done

==> WARNING: A newer version of conda exists. <==
  current version: 4.5.12
  latest version: 4.6.12

Please update conda by running

    $ conda update -n base conda

#
# To activate this environment, use
#
#     $ conda activate deseq2_test
#
# To deactivate an active environment, use
#
#     $ conda deactivate

edgeR works:

+ conda env create --file elvers/rules/edgeR/environment.yml -n edgeR_test
Solving environment: ...working... done
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done

==> WARNING: A newer version of conda exists. <==
  current version: 4.5.12
  latest version: 4.6.12

Please update conda by running

    $ conda update -n base conda

#
# To activate this environment, use
#
#     $ conda activate edgeR_test
#
# To deactivate an active environment, use
#
#     $ conda deactivate

fastqc works:

+ conda env create --file elvers/rules/fastqc/environment.yml -n fastqc_test
Solving environment: ...working... done
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done

==> WARNING: A newer version of conda exists. <==
  current version: 4.5.12
  latest version: 4.6.12

Please update conda by running

    $ conda update -n base conda

#
# To activate this environment, use
#
#     $ conda activate fastqc_test
#
# To deactivate an active environment, use
#
#     $ conda deactivate

khmer does not work:

+ conda env create --file elvers/rules/khmer/environment.yml -n khmer_test
Solving environment: ...working... done
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... failed

==> WARNING: A newer version of conda exists. <==
  current version: 4.5.12
  latest version: 4.6.12

Please update conda by running

    $ conda update -n base conda

ERROR conda.core.link:_execute(507): An error occurred while installing package 'conda-forge::bz2file-0.98-py_0'.
FileNotFoundError(2, "No such file or directory: '/Users/charles/.pyenv/versions/miniconda3-4.3.30/envs/khmer_test/bin/python3.6'")
Attempting to roll back.

Rolling back transaction: ...working... done

FileNotFoundError(2, "No such file or directory: '/Users/charles/.pyenv/versions/miniconda3-4.3.30/envs/khmer_test/bin/python3.6'")

multiqc does not work:

+ conda env create --file elvers/rules/multiqc/environment.yml -n multiqc_test
Solving environment: ...working... done
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... failed

==> WARNING: A newer version of conda exists. <==
  current version: 4.5.12
  latest version: 4.6.12

Please update conda by running

    $ conda update -n base conda

ERROR conda.core.link:_execute(507): An error occurred while installing package 'conda-forge::click-7.0-py_0'.
FileNotFoundError(2, "No such file or directory: '/Users/charles/.pyenv/versions/miniconda3-4.3.30/envs/multiqc_test/bin/python3.6'")
Attempting to roll back.

Rolling back transaction: ...working... done

FileNotFoundError(2, "No such file or directory: '/Users/charles/.pyenv/versions/miniconda3-4.3.30/envs/multiqc_test/bin/python3.6'")

paladin works:

+ conda env create --file elvers/rules/paladin/environment.yml -n paladin_test
Solving environment: ...working... done

==> WARNING: A newer version of conda exists. <==
  current version: 4.5.12
  latest version: 4.6.12

Please update conda by running

    $ conda update -n base conda

samtools-1.5         | 455 KB    | ########## | 100%
paladin-1.4.4        | 118 KB    | ########## | 100%
Downloading and Extracting Packages
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done

#
# To activate this environment, use
#
#     $ conda activate paladin_test
#
# To deactivate an active environment, use
#
#     $ conda deactivate

pear works:

+ conda env create --file elvers/rules/pear/environment.yml -n pear_test
Solving environment: ...working... done

==> WARNING: A newer version of conda exists. <==
  current version: 4.5.12
  latest version: 4.6.12

Please update conda by running

    $ conda update -n base conda

pear-0.9.6           | 47 KB     | ########## | 100%
Downloading and Extracting Packages
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done

#
# To activate this environment, use
#
#     $ conda activate pear_test
#
# To deactivate an active environment, use
#
#     $ conda deactivate

plass works:

+ conda env create --file elvers/rules/plass/environment.yml -n plass_test
Solving environment: ...working... done

==> WARNING: A newer version of conda exists. <==
  current version: 4.5.12
  latest version: 4.6.12

Please update conda by running

    $ conda update -n base conda

openmp-8.0.0         | 7 KB      | ########## | 100%
gawk-4.2.1           | 1.3 MB    | ########## | 100%
plass-2.c7e35        | 551 KB    | ########## | 100%
Downloading and Extracting Packages
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done

#
# To activate this environment, use
#
#     $ conda activate plass_test
#
# To deactivate an active environment, use
#
#     $ conda deactivate

rcorrector works:

+ conda env create --file elvers/rules/rcorrector/environment.yml -n rcorrector_test
Solving environment: ...working... done

==> WARNING: A newer version of conda exists. <==
  current version: 4.5.12
  latest version: 4.6.12

Please update conda by running

    $ conda update -n base conda

jellyfish-2.2.10     | 543 KB    | ########## | 100%
perl-5.26.2          | 14.2 MB   | ########## | 100%
rcorrector-1.0.3.1   | 29 KB     | ########## | 100%
jemalloc-4.5.0       | 4.1 MB    | ########## | 100%
Downloading and Extracting Packages
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done

#
# To activate this environment, use
#
#     $ conda activate rcorrector_test
#
# To deactivate an active environment, use
#
#     $ conda deactivate

salmon works:

+ conda env create --file elvers/rules/salmon/environment.yml -n salmon_test
Solving environment: ...working... done

==> WARNING: A newer version of conda exists. <==
  current version: 4.5.12
  latest version: 4.6.12

Please update conda by running

    $ conda update -n base conda

jemalloc-5.2.0       | 1.9 MB    | ########## | 100%
Downloading and Extracting Packages
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done

#
# To activate this environment, use
#
#     $ conda activate salmon_test
#
# To deactivate an active environment, use
#
#     $ conda deactivate

sourmash does not work:

+ conda env create --file elvers/rules/sourmash/environment.yml -n sourmash_test
Solving environment: ...working... done
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... failed

==> WARNING: A newer version of conda exists. <==
  current version: 4.5.12
  latest version: 4.6.12

Please update conda by running

    $ conda update -n base conda

ERROR conda.core.link:_execute(507): An error occurred while installing package 'conda-forge::bz2file-0.98-py_0'.
FileNotFoundError(2, "No such file or directory: '/Users/charles/.pyenv/versions/miniconda3-4.3.30/envs/sourmash_test/bin/python3.7'")
Attempting to roll back.

Rolling back transaction: ...working... done

FileNotFoundError(2, "No such file or directory: '/Users/charles/.pyenv/versions/miniconda3-4.3.30/envs/sourmash_test/bin/python3.7'")

trimmomatic works:

+ conda env create --file elvers/rules/trimmomatic/environment.yml -n trimmomatic_test
Solving environment: ...working... done

==> WARNING: A newer version of conda exists. <==
  current version: 4.5.12
  latest version: 4.6.12

Please update conda by running

    $ conda update -n base conda

pigz-2.3.4           | 74 KB     | ########## | 100%
trimmomatic-0.38     | 140 KB    | ########## | 100%
openjdk-11.0.1       | 164.8 MB  | ########## | 100%
Downloading and Extracting Packages
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done

#
# To activate this environment, use
#
#     $ conda activate trimmomatic_test
#
# To deactivate an active environment, use
#
#     $ conda deactivate

trinity does not work:

+ conda env create --file elvers/rules/trinity/environment.yml -n trinity_test
Solving environment: ...working... failed

ResolvePackageNotFound:
  - trinity=2.8.4
charlesreid1 commented 5 years ago

Running the same conda env create command on a stock copy of elvers still fails on the same errors (not finding versions of python 3.6 or python 3.7 in the environment), so these errors are not specifically related to pinning the version numbers in the environment.yml file (except trinity, which works fine if no version number is specified in its environment.yml file).

Should the environment.yml files be specifying a version of python to use?

bluegenes commented 5 years ago

just checking - are you working in the base elvers-env environment?

charlesreid1 commented 5 years ago

Ah, thank you! I was forgetting the initial environment setup using the top-level environment.yml file. Doing a 

conda env create --file environment.yml -n elvers-env

as per the Quickstart fixed the problem.

charlesreid1 commented 5 years ago

Actually, sourmash 2.0 requires python 3.7, and the top level elvers environment.yml specifies python 3.6 or higher, so not quite solved yet.

I will leave it up to you whether to bump the python version in the elvers environment.yml or roll back the sourmash version - just let me know and close this issue and I'll make the change in PR #136.

bluegenes commented 5 years ago

hm, I say bump the version as long as it doesn't screw up dammit, which currently requires python 3.6

charlesreid1 commented 5 years ago

Okay, slight revision, since we're specifying python 3.6 or greater, this means that when we create an environment with sourmash, which requires python 3.7, conda will simply download and install python 3.7, so we don't need to bump the python version in the top-level environment.yml file:

$ conda env create --file elvers/rules/sourmash/environment.yml -n sourmash_test
Solving environment: done

Downloading and Extracting Packages
harfbuzz-2.4.0       | 1.5 MB    | ######################################################################### | 100%
wheel-0.33.1         | 34 KB     | ######################################################################### | 100%
pyparsing-2.4.0      | 55 KB     | ######################################################################### | 100%
sip-4.18.1           | 277 KB    | ######################################################################### | 100%
matplotlib-base-3.0. | 6.6 MB    | ######################################################################### | 100%
sourmash-2.0.0       | 262 KB    | ######################################################################### | 100%
liblapack-3.8.0      | 6 KB      | ######################################################################### | 100%
openblas-0.3.5       | 15.8 MB   | ######################################################################### | 100%
gdk-pixbuf-2.32.2    | 759 KB    | ######################################################################### | 100%
pango-1.40.14        | 524 KB    | ######################################################################### | 100%
glib-2.58.3          | 3.3 MB    | ######################################################################### | 100%
setuptools-41.0.0    | 635 KB    | ######################################################################### | 100%
certifi-2019.3.9     | 149 KB    | ######################################################################### | 100%
ijson-2.3            | 13 KB     | ######################################################################### | 100%
pip-19.0.3           | 1.8 MB    | ######################################################################### | 100%
cairo-1.16.0         | 1.5 MB    | ######################################################################### | 100%
python-dateutil-2.8. | 219 KB    | ######################################################################### | 100%
python-3.7.1         | 36.4 MB   | #########3

Will close this and continue to post info about what's going on in PR #136.