search

sunbeam-labs / sunbeam

A robust, extensible metagenomics pipeline
http://sunbeam.readthedocs.io
166 stars 40 forks source link

Error running run-tests.bash at creating environment for annotation.yml #386

Closed TakacsBertalan closed 1 year ago

TakacsBertalan commented 1 year ago

Hi! After installing sunbeam, I am running the run-tests script, but it results the following error: I use this command to run the script: bash tests/run_tests.bash -e sunbeam3.1.1

The error I get

Running setup... ✔
Running build_test_data... ✔
Running test_all... 
Failed command error output:
Found extension sbx_test in folder sbx_test
Found extension sbx_test_subdir in folder sbx_test_subdir
Found extension sbx_test_smk in folder sbx_test_smk
Collecting host/contaminant genomes... done.
Collecting target genomes... done.
Building DAG of jobs...
Creating conda environment /media/deltagene/Microbiome/sunbeam/rules/annotation/../../envs/annotation.yml...
Downloading and installing remote packages.
CreateCondaEnvironmentException:
Could not create conda environment from /media/deltagene/Microbiome/sunbeam/rules/annotation/../../envs/annotation.yml:
Command:
mamba env create --quiet --file "/media/deltagene/Microbiome/sunbeam/.snakemake/0fde01217a771cb210eeab8977c7c831_.yaml" --prefix "/media/deltagene/Microbiome/sunbeam/.snakemake/0fde01217a771cb210eeab8977c7c831_"
Output:
Retrieving notices: ...working... done
  Package                         Version  Build                            Channel                    Size
─────────────────────────────────────────────────────────────────────────────────────────────────────────────
  Install:
─────────────────────────────────────────────────────────────────────────────────────────────────────────────

  + _libgcc_mutex                     0.1  conda_forge                      conda-forge/linux-64     Cached
  + _openmp_mutex                     4.5  2_gnu                            conda-forge/linux-64     Cached
  + biopython                        1.70  np112py27_1                      bioconda/linux-64        Cached
  + blas                              1.1  openblas                         conda-forge/linux-64     Cached
  + blast                          2.10.1  pl526he19e7b1_3                  bioconda/linux-64        Cached
  + boost-cpp                      1.74.0  h75c5d50_8                       conda-forge/linux-64     Cached
  + bzip2                           1.0.8  h7f98852_4                       conda-forge/linux-64     Cached
  + c-ares                         1.11.0  h470a237_1                       bioconda/linux-64        Cached
  + ca-certificates             2022.12.7  ha878542_0                       conda-forge/linux-64     Cached
  + certifi                    2019.11.28  py27h8c360ce_1                   conda-forge/linux-64     Cached
  + curl                           7.71.1  he644dc0_8                       conda-forge/linux-64     Cached
  + diamond                        2.0.11  hdcc8f71_0                       bioconda/linux-64        Cached
  + entrez-direct                    16.2  he881be0_1                       bioconda/linux-64        Cached
  + freetype                       2.12.1  hca18f0e_1                       conda-forge/linux-64     Cached
  + gettext                        0.21.1  h27087fc_0                       conda-forge/linux-64     Cached
  + icu                              70.1  h27087fc_0                       conda-forge/linux-64     Cached
  + jpeg                               9e  h0b41bf4_3                       conda-forge/linux-64     Cached
  + krb5                           1.17.2  h926e7f8_0                       conda-forge/linux-64     Cached
  + ld_impl_linux-64                 2.40  h41732ed_0                       conda-forge/linux-64     Cached
  + libcurl                        7.71.1  hcdd3856_8                       conda-forge/linux-64     Cached
  + libedit                  3.1.20191231  he28a2e2_2                       conda-forge/linux-64     Cached
  + libev                            4.33  h516909a_1                       conda-forge/linux-64     Cached
  + libffi                          3.2.1  he1b5a44_1007                    conda-forge/linux-64     Cached
  + libgcc-ng                      12.2.0  h65d4601_19                      conda-forge/linux-64     Cached
  + libgfortran-ng                  7.5.0  h14aa051_20                      conda-forge/linux-64     Cached
  + libgfortran4                    7.5.0  h14aa051_20                      conda-forge/linux-64     Cached
  + libgomp                        12.2.0  h65d4601_19                      conda-forge/linux-64     Cached
  + libidn2                         2.3.4  h166bdaf_0                       conda-forge/linux-64     Cached
  + libnghttp2                     1.41.0  hab1572f_1                       conda-forge/linux-64     Cached
  + libpng                         1.6.39  h753d276_0                       conda-forge/linux-64     Cached
  + libsqlite                      3.40.0  h753d276_0                       conda-forge/linux-64     Cached
  + libssh2                        1.10.0  haa6b8db_3                       conda-forge/linux-64     Cached
  + libstdcxx-ng                   12.2.0  h46fd767_19                      conda-forge/linux-64     Cached
  + libtiff                         4.2.0  hf544144_3                       conda-forge/linux-64     Cached
  + libunistring                   0.9.10  h7f98852_0                       conda-forge/linux-64     Cached
  + libwebp-base                    1.2.4  h166bdaf_0                       conda-forge/linux-64     Cached
  + libzlib                        1.2.13  h166bdaf_4                       conda-forge/linux-64     Cached
  + mmtf-python                     1.0.2  py27_0                           bioconda/linux-64        Cached
  + msgpack-python                  1.0.0  py27h9e3301b_1                   conda-forge/linux-64     Cached
  + ncurses                           6.3  h27087fc_1                       conda-forge/linux-64     Cached
  + numpy                          1.12.1  py27_blas_openblash1522bff_1001  conda-forge/linux-64     Cached
  + olefile                          0.46  pyh9f0ad1d_1                     conda-forge/noarch       Cached
  + openblas                        0.3.3  h9ac9557_1001                    conda-forge/linux-64     Cached
  + openssl                        1.1.1t  h0b41bf4_0                       conda-forge/linux-64     Cached
  + pcre                             8.45  h9c3ff4c_0                       conda-forge/linux-64     Cached
  + perl                           5.26.2  h36c2ea0_1008                    conda-forge/linux-64     Cached
  + perl-archive-tar                 2.32  pl526_0                          bioconda/linux-64        Cached
  + perl-carp                        1.38  pl526_3                          bioconda/linux-64        Cached
  + perl-common-sense                3.74  pl526_2                          bioconda/linux-64        Cached
  + perl-compress-raw-bzip2         2.087  pl526he1b5a44_0                  bioconda/linux-64        Cached
  + perl-compress-raw-zlib          2.087  pl526hc9558a2_0                  bioconda/linux-64        Cached
  + perl-exporter                    5.72  pl526_1                          bioconda/linux-64        Cached
  + perl-exporter-tiny           1.002001  pl526_0                          bioconda/linux-64        Cached
  + perl-extutils-makemaker          7.36  pl526_1                          bioconda/linux-64        Cached
  + perl-io-compress                2.087  pl526he1b5a44_0                  bioconda/linux-64        Cached
  + perl-io-zlib                     1.10  pl526_2                          bioconda/linux-64        Cached
  + perl-json                        4.02  pl526_0                          bioconda/linux-64        Cached
  + perl-json-xs                     2.34  pl526h6bb024c_3                  bioconda/linux-64        Cached
  + perl-list-moreutils             0.428  pl526_1                          bioconda/linux-64        Cached
  + perl-list-moreutils-xs          0.428  pl526_0                          bioconda/linux-64        Cached
  + perl-pathtools                   3.75  pl526h14c3975_1                  bioconda/linux-64        Cached
  + perl-scalar-list-utils           1.52  pl526h516909a_0                  bioconda/linux-64        Cached
  + perl-types-serialiser             1.0  pl526_2                          bioconda/linux-64        Cached
  + perl-xsloader                    0.24  pl526_0                          bioconda/linux-64        Cached
  + pillow                          6.2.1  py27hd70f55b_1                   conda-forge/linux-64     Cached
  + pip                            20.1.1  pyh9f0ad1d_0                     conda-forge/noarch       Cached
  + prodigal                        2.6.3  hec16e2b_4                       bioconda/linux-64        Cached
  + python                         2.7.15  h5a48372_1011_cpython            conda-forge/linux-64     Cached
  + python_abi                        2.7  1_cp27mu                         conda-forge/linux-64     Cached
  + readline                        8.1.2  h0f457ee_0                       conda-forge/linux-64     Cached
  + reportlab                      3.5.42  py27heca39a3_0                   conda-forge/linux-64     Cached
  + setuptools                     44.0.0  py27_0                           conda-forge/linux-64     Cached
  + sqlite                         3.40.0  h4ff8645_0                       conda-forge/linux-64     Cached
  + tk                             8.6.12  h27826a3_0                       conda-forge/linux-64     Cached
  + wget                           1.20.3  ha56f1ee_1                       conda-forge/linux-64     Cached
  + wheel                          0.37.1  pyhd8ed1ab_0                     conda-forge/noarch       Cached
  + xz                              5.2.6  h166bdaf_0                       conda-forge/linux-64     Cached
  + zlib                           1.2.13  h166bdaf_4                       conda-forge/linux-64     Cached
  + zstd                            1.5.2  h3eb15da_6                       conda-forge/linux-64     Cached

  Summary:

  Install: 79 packages

  Total download: 0 B

─────────────────────────────────────────────────────────────────────────────────────────────────────────────

Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
ERROR conda.core.link:_execute(740): An error occurred while installing package 'conda-forge::perl-5.26.2-h36c2ea0_1008'.
Rolling back transaction: ...working... done

[Errno 22] Invalid argument: '/media/deltagene/Microbiome/sunbeam/.snakemake/0fde01217a771cb210eeab8977c7c831_/man/man3/App::Cpan.3'
()
x (log: /tmp/tmp.kDOqXsAhMt/test_all.[out/err])
-- TESTS FAILED --

I tried running it multiple times and using real files, but the error seems consistent. It is always at the annotation step and the problem seems to be with a perl package, although not always the same. Sometimes it is perl-5.26.2, other times it is perl-5.5.32.1-2. What could be the problem? Thanks in advance, Bertalan Takács

Ulthran commented 1 year ago

Hi @TakacsBertalan,

Thanks for submitting this issue. Unfortunately it seems like recent releases of mamba (a supposedly better environment solver than conda) have caused lots of issues with it doing its job. My first suggestion here would be to run sunbeam with the conda frontend set to conda instead of mamba. Please try changing line 3 of sunbeam/tests/test_suite.sh from 

sunbeam run -- --configfile=$TEMPDIR/tmp_config.yml -p

to

sunbeam run -- --configfile=$TEMPDIR/tmp_config.yml -p --conda-frontend conda

and then run bash tests/run_tests.bash -e sunbeam3.1.1 -v -t test_all to check if there are other issues.

TakacsBertalan commented 1 year ago

Hi! Thanks for the answer! I modified the file and ran the script again. This is the running_test_all part of the output where the problem occured again:

Running test_all... Found extension sbx_test in folder sbx_test
Found extension sbx_test_subdir in folder sbx_test_subdir
Found extension sbx_test_smk in folder sbx_test_smk
Collecting host/contaminant genomes... done.
Collecting target genomes... done.
Building DAG of jobs...
Creating conda environment /media/deltagene/Microbiome/sunbeam/rules/mapping/../../envs/mapping.yml...
Downloading and installing remote packages.
CreateCondaEnvironmentException:
Could not create conda environment from /media/deltagene/Microbiome/sunbeam/rules/mapping/../../envs/mapping.yml:
Command:
conda env create --quiet --file "/media/deltagene/Microbiome/sunbeam/.snakemake/cd9e002dbe0498010481da5ecc68c23e_.yaml" --prefix "/media/deltagene/Microbiome/sunbeam/.snakemake/cd9e002dbe0498010481da5ecc68c23e_"
Output:
Retrieving notices: ...working... done
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
ERROR conda.core.link:_execute(740): An error occurred while installing package 'conda-forge::perl-5.32.1-2_h7f98852_perl5'.
Rolling back transaction: ...working... done

[Errno 22] Invalid argument: '/media/deltagene/Microbiome/sunbeam/.snakemake/cd9e002dbe0498010481da5ecc68c23e_/man/man3/App::Cpan.3'
()

--configfile flag not found, either it is missing (not ok) or was provided as --configfile=filename (ok)
Running: snakemake --snakefile /media/deltagene/Microbiome/sunbeam/Snakefile -c --use-conda --conda-prefix /media/deltagene/Microbiome/sunbeam/.snakemake --configfile=/tmp/tmp.D9y7PPAwab/tmp_config.yml -p --conda-frontend conda

Failed command error output:
Found extension sbx_test in folder sbx_test
Found extension sbx_test_subdir in folder sbx_test_subdir
Found extension sbx_test_smk in folder sbx_test_smk
Collecting host/contaminant genomes... done.
Collecting target genomes... done.
Building DAG of jobs...
Creating conda environment /media/deltagene/Microbiome/sunbeam/rules/mapping/../../envs/mapping.yml...
Downloading and installing remote packages.
CreateCondaEnvironmentException:
Could not create conda environment from /media/deltagene/Microbiome/sunbeam/rules/mapping/../../envs/mapping.yml:
Command:
conda env create --quiet --file "/media/deltagene/Microbiome/sunbeam/.snakemake/cd9e002dbe0498010481da5ecc68c23e_.yaml" --prefix "/media/deltagene/Microbiome/sunbeam/.snakemake/cd9e002dbe0498010481da5ecc68c23e_"
Output:
Retrieving notices: ...working... done
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
ERROR conda.core.link:_execute(740): An error occurred while installing package 'conda-forge::perl-5.32.1-2_h7f98852_perl5'.
Rolling back transaction: ...working... done

[Errno 22] Invalid argument: '/media/deltagene/Microbiome/sunbeam/.snakemake/cd9e002dbe0498010481da5ecc68c23e_/man/man3/App::Cpan.3'
()
x (log: /tmp/tmp.D9y7PPAwab/test_all.[out/err])
-- TESTS FAILED --
Deleting temporary directory /tmp/tmp.D9y7PPAwab
Restoring extensions folder
Ulthran commented 1 year ago

Ahh that's unfortunate, seems like conda can't solve the environment either. This is the major downside of using underspecified environments, it means that releases of new versions of dependencies can break the solver very unpredictably. We should soon be releasing version 4 of sunbeam that will dramatically reduce the amount of dependencies.

For now, I can make a couple suggestions. For one thing, if you don't need the mapping environment you can just use the portions of the pipeline you need without running the tests. Otherwise, you can try (and this is a real shot in the dark) removing these lines from envs/mapping.yml:

  - conda-forge::gsl         # bcftools
  - conda-forge::libopenblas # bcftools

Sorry for the inconvenience. If none of this works, I'd suggest waiting on v4.

TakacsBertalan commented 1 year ago

Thank you for the suggestions! The mapping portions of the pipeline is not very important for my work, so I will just use the rest.