Problems with installation

Hello Maarten,

I am relatively a novice with bioinformatic tools so maybe this is a stupid question. I am having some issues with installation of epiGBS2v3. Following the tutorial, I have created a conda environment and started the installation with snakemake 7.21.0 since the version 6.1.1 gave me a lot of conflict problems. I installed all dependencies required, cloned the github files in a directory and, from this position, tried the dry run that went OK. I then went on, trying to run the pipeline with example data: the process installed all required software, but ended with error with this message:

(snake) bruno@pleroma:~/miniconda3/epiGBS2$ snakemake -j 14 -p --use-conda Building DAG of jobs... Creating conda environment src/env/stacks.yaml... Downloading and installing remote packages. CreateCondaEnvironmentException: Could not create conda environment from /home/bruno/miniconda3/epiGBS2/src/rules/../env/stacks.yaml: Command: mamba env create --quiet --file "/home/bruno/miniconda3/epiGBS2/.snakemake/conda/32d222dda49752f948109dbbc167083f.yaml" --prefix "/home/bruno/miniconda3/epiGBS2/.snakemake/conda/32d222dda49752f948109dbbc167083f" Output: Could not solve for environment specs Encountered problems while solving:

package c-ares-1.18.1-h7f98852_0 is excluded by strict repo priority
package libdb-6.2.32-he1b5a44_0 is excluded by strict repo priority
package perl-xml-parser-2.44_01-pl5262hc3e0081_1002 is excluded by strict repo priority
package ca-certificates-2021.10.26-h06a4308_2 is excluded by strict repo priority
package gettext-0.21.0-hf68c758_0 is excluded by strict repo priority
package graphite2-1.3.14-h23475e2_0 is excluded by strict repo priority
package libedit-3.1.20210714-h7f8727e_0 is excluded by strict repo priority
package libgd-2.2.5-h8e06009_4 is excluded by strict repo priority
package libxcb-1.14-h7b6447c_0 is excluded by strict repo priority
package perl-net-ssleay-1.74-0 requires perl >=5.22.0,<5.23.0, but none of the providers can be installed
package perl-yaml-1.29-pl526_0 requires perl >=5.26.2,<5.26.3.0a0, but none of the providers can be installed
package python-3.10.0-h543edf9_2_cpython requires openssl >=3.0.0,<4.0a0, but none of the providers can be installed
package perl-soap-lite-1.19-pl526_1 requires perl-io-socket-ssl, but none of the providers can be installed
package perl-bioperl-core-1.007002-pl526_1 requires perl-soap-lite, but none of the providers can be installed
package perl-lwp-protocol-https-6.07-pl526_4 requires perl-io-socket-ssl, but none of the providers can be installed

The environment can't be solved, aborting the operation

I then tried setting conda installation to flexible channel priority and this is the result:

(snake) bruno@pleroma:~/miniconda3/epiGBS2$ snakemake -j 14 -p --use-conda Building DAG of jobs... Your conda installation is not configured to use strict channel priorities. This is however crucial for having robust and correct environments (for details, see https://conda-forge.org/docs/user/tipsandtricks.html). Please consider to configure strict priorities by executing 'conda config --set channelpriority strict'. Creating conda environment src/env/stacks.yaml... Downloading and installing remote packages. CreateCondaEnvironmentException: Could not create conda environment from /home/bruno/miniconda3/epiGBS2/src/rules/../env/stacks.yaml: Command: mamba env create --quiet --file "/home/bruno/miniconda3/epiGBS2/.snakemake/conda/32d222dda49752f948109dbbc167083f.yaml" --prefix "/home/bruno/miniconda3/epiGBS2/.snakemake/conda/32d222dda49752f948109dbbc167083f_" Output: Could not solve for environment specs Encountered problems while solving:

package perl-net-ssleay-1.74-0 requires perl >=5.22.0,<5.23.0, but none of the providers can be installed
package perl-ntlm-1.09-pl526_4 requires perl >=5.26.2,<5.26.3.0a0, but none of the providers can be installed
package python-3.10.0-h543edf9_2_cpython requires openssl >=3.0.0,<4.0a0, but none of the providers can be installed
package perl-lwp-protocol-https-6.07-pl526_4 requires perl-io-socket-ssl, but none of the providers can be installed
package perl-soap-lite-1.19-pl526_1 requires perl-io-socket-ssl, but none of the providers can be installed
package perl-libwww-perl-6.39-pl526_0 requires perl-net-http >=6.18, but none of the providers can be installed
package perl-xml-sax-1.02-pl526_0 requires perl >=5.26.2,<5.26.3.0a0, but none of the providers can be installed
package perl-role-tiny-2.000008-pl526_0 requires perl >=5.26.2,<5.26.3.0a0, but none of the providers can be installed

The environment can't be solved, aborting the operation

It seems that there are some conflicts that I am not able to solve. What did I do wrong? Thank you in advance for your kind help.

Bruno

Hi Bruno, The conda environment is meant to be ran with Snakemake 6.1.1. , the conda environment should be able to be created with this version of the program. Using a newer version might be the reason the conda environment runs into conflict issues. So I would recommend by starting with creating a new conda environment with the snakemake 6.1.1. version and running it from there. Kind regards, Maarten

Hello Maarten, First of all thanks for your answer. I restarted with the installation creating a new Conda environment and installing snakemake 6.1.1 version as you suggested. Unfortunately there is a new error:

Could not solve for environment specs Encountered problems while solving:

nothing provides aioeasywebdav needed by snakemake-6.1.1-hdfd78af_0

The environment can't be solved, aborting the operation

checked what “aioeasywebdav” is and seems to be a dependency of snakemake. In the same site there is a list of other dependencies required for the installation of snakemake. It is possible to install them one by one but I am not sure if that is correct because they are optimized for the latest version of snakemake (actually 7.23.1) but I need to install the version 6.1.1. How can I proceed? Thanks for your patience, best regards,

Bruno

Hi Bruno, There does seems to be an issue with the environment creation. I've created a file with all of the dependencies. (see below) Could you try saving this as snake.yaml and then create the environment using mamba env create -f snake.yaml ?

Kind regards, Maarten

name: snakemake
channels:
  - bioconda
  - conda-forge
  - defaults
  - r
dependencies:
  - _libgcc_mutex=0.1=conda_forge
  - _openmp_mutex=4.5=1_gnu
  - aioeasywebdav=2.4.0=py39hf3d152e_1001
  - aiohttp=3.7.4=py39h3811e60_0
  - amply=0.1.4=py_0
  - appdirs=1.4.4=pyh9f0ad1d_0
  - async-timeout=3.0.1=py_1000
  - atk-1.0=2.36.0=h3371d22_4
  - attmap=0.13.0=pyhd8ed1ab_0
  - attrs=20.3.0=pyhd3deb0d_0
  - azure-common=1.1.27=pyhd8ed1ab_0
  - azure-core=1.12.0=pyhd8ed1ab_0
  - azure-storage-blob=12.8.0=pyhd8ed1ab_0
  - backports=1.0=py_2
  - backports.functools_lru_cache=1.6.3=pyhd8ed1ab_0
  - bcrypt=3.2.0=py39h3811e60_1
  - blinker=1.4=py_1
  - boto=2.49.0=py_0
  - boto3=1.17.49=pyhd8ed1ab_0
  - botocore=1.20.49=pyhd8ed1ab_0
  - brotlipy=0.7.0=py39h3811e60_1001
  - bzip2=1.0.8=h7f98852_4
  - c-ares=1.17.1=h7f98852_1
  - ca-certificates=2020.12.5=ha878542_0
  - cachetools=4.2.1=pyhd8ed1ab_0
  - cairo=1.16.0=h6cf1ce9_1008
  - certifi=2020.12.5=py39hf3d152e_1
  - cffi=1.14.5=py39he32792d_0
  - chardet=4.0.0=py39hf3d152e_1
  - coincbc=2.10.5=hcee13e7_1
  - configargparse=1.4=pyhd8ed1ab_0
  - cryptography=3.4.7=py39hbca0aa6_0
  - datrie=0.8.2=py39h07f9747_1
  - decorator=5.0.6=pyhd8ed1ab_0
  - docutils=0.17=py39hf3d152e_0
  - dropbox=11.6.0=pyhd8ed1ab_0
  - expat=2.3.0=h9c3ff4c_0
  - fftw=3.3.9=nompi_h74d3f13_101
  - filechunkio=1.8=py_2
  - filelock=3.0.12=pyh9f0ad1d_0
  - font-ttf-dejavu-sans-mono=2.37=hab24e00_0
  - font-ttf-inconsolata=2.001=hab24e00_0
  - font-ttf-source-code-pro=2.030=hab24e00_0
  - font-ttf-ubuntu=0.83=hab24e00_0
  - fontconfig=2.13.1=hba837de_1005
  - fonts-conda-ecosystem=1=0
  - fonts-conda-forge=1=0
  - freetype=2.10.4=h0708190_1
  - fribidi=1.0.10=h36c2ea0_0
  - ftputil=5.0.1=pyhd8ed1ab_0
  - gdk-pixbuf=2.42.6=h04a7f16_0
  - gettext=0.19.8.1=h0b5b191_1005
  - ghostscript=9.18=1
  - giflib=5.2.1=h36c2ea0_2
  - gitdb=4.0.7=pyhd8ed1ab_0
  - gitpython=3.1.14=pyhd8ed1ab_0
  - google-api-core=1.26.2=pyhd8ed1ab_0
  - google-api-python-client=2.1.0=pyhd8ed1ab_0
  - google-auth=1.28.0=pyh44b312d_0
  - google-auth-httplib2=0.1.0=pyhd8ed1ab_0
  - google-cloud-core=1.5.0=pyhd3deb0d_0
  - google-cloud-storage=1.19.0=py_0
  - google-crc32c=1.1.2=py39hb81f231_0
  - google-resumable-media=1.2.0=pyhd3deb0d_0
  - googleapis-common-protos=1.53.0=py39hf3d152e_0
  - graphite2=1.3.13=h58526e2_1001
  - graphviz=2.47.0=he056042_1
  - grpcio=1.37.0=py39hff7568b_0
  - gtk2=2.24.33=hab0c2f8_0
  - gts=0.7.6=h64030ff_2
  - harfbuzz=2.8.0=h83ec7ef_1
  - httplib2=0.19.1=pyhd8ed1ab_0
  - icu=68.1=h58526e2_0
  - idna=2.10=pyh9f0ad1d_0
  - imagemagick=7.0.11_6=pl5320hb503478_0
  - importlib-metadata=3.10.0=py39hf3d152e_0
  - iniconfig=1.1.1=pyh9f0ad1d_0
  - ipython_genutils=0.2.0=py_1
  - isodate=0.6.0=py_1
  - jbig=2.1=h7f98852_2002
  - jinja2=2.11.3=pyh44b312d_0
  - jmespath=0.10.0=pyh9f0ad1d_0
  - jpeg=9d=h36c2ea0_0
  - jsonschema=3.2.0=pyhd8ed1ab_3
  - jupyter_core=4.7.1=py39hf3d152e_0
  - ld_impl_linux-64=2.35.1=hea4e1c9_2
  - libblas=3.9.0=8_openblas
  - libcblas=3.9.0=8_openblas
  - libcrc32c=1.1.1=h9c3ff4c_2
  - libffi=3.3=h58526e2_2
  - libgcc=7.2.0=h69d50b8_2
  - libgcc-ng=9.3.0=h2828fa1_18
  - libgd=2.3.2=h78a0170_0
  - libgfortran-ng=9.3.0=hff62375_18
  - libgfortran5=9.3.0=hff62375_18
  - libglib=2.68.1=h3e27bee_0
  - libgomp=9.3.0=h2828fa1_18
  - libiconv=1.16=h516909a_0
  - liblapack=3.9.0=8_openblas
  - libopenblas=0.3.12=pthreads_h4812303_1
  - libpng=1.6.37=h21135ba_2
  - libprotobuf=3.15.7=h780b84a_0
  - librsvg=2.50.3=hfa39831_1
  - libsodium=1.0.18=h36c2ea0_1
  - libstdcxx-ng=9.3.0=h6de172a_18
  - libtiff=4.2.0=hdc55705_0
  - libtool=2.4.6=h58526e2_1007
  - libuuid=2.32.1=h7f98852_1000
  - libwebp=1.2.0=h3452ae3_0
  - libwebp-base=1.2.0=h7f98852_2
  - libxcb=1.13=h7f98852_1003
  - libxml2=2.9.10=h72842e0_3
  - logmuse=0.2.6=pyh8c360ce_0
  - lz4-c=1.9.3=h9c3ff4c_0
  - markupsafe=1.1.1=py39h3811e60_3
  - more-itertools=8.7.0=pyhd8ed1ab_0
  - msrest=0.6.21=pyh44b312d_0
  - multidict=5.1.0=py39h3811e60_1
  - nbformat=5.1.3=pyhd8ed1ab_0
  - ncurses=6.2=h58526e2_4
  - networkx=2.3=py_0
  - numpy=1.20.2=py39hdbf815f_0
  - oauth2client=4.1.3=py_0
  - oauthlib=3.0.1=py_0
  - openjpeg=2.4.0=hf7af979_0
  - openssl=1.1.1k=h7f98852_0
  - packaging=20.9=pyh44b312d_0
  - pandas=1.2.3=py39hde0f152_0
  - pango=1.42.4=h69149e4_5
  - paramiko=2.7.2=pyh9f0ad1d_0
  - pcre=8.44=he1b5a44_0
  - peppy=0.31.0=pyh9f0ad1d_0
  - perl=5.32.0=h36c2ea0_0
  - pip=21.0.1=pyhd8ed1ab_0
  - pixman=0.40.0=h36c2ea0_0
  - pkg-config=0.29.2=h36c2ea0_1008
  - pluggy=0.13.1=py39hf3d152e_4
  - ply=3.11=py_1
  - prettytable=2.1.0=pyhd8ed1ab_0
  - protobuf=3.15.7=py39he80948d_0
  - psutil=5.8.0=py39h3811e60_1
  - pthread-stubs=0.4=h36c2ea0_1001
  - pulp=2.4=py39hf3d152e_0
  - py=1.10.0=pyhd3deb0d_0
  - pyasn1=0.4.8=py_0
  - pyasn1-modules=0.2.7=py_0
  - pycparser=2.20=pyh9f0ad1d_2
  - pygments=2.8.1=pyhd8ed1ab_0
  - pygraphviz=1.7=py39h78163bd_0
  - pyjwt=2.0.1=pyhd8ed1ab_1
  - pynacl=1.4.0=py39h3811e60_2
  - pyopenssl=20.0.1=pyhd8ed1ab_0
  - pyparsing=2.4.7=pyh9f0ad1d_0
  - pyrsistent=0.17.3=py39h3811e60_2
  - pysftp=0.2.9=py_1
  - pysocks=1.7.1=py39hf3d152e_3
  - pytest=6.2.3=py39hf3d152e_0
  - python=3.9.2=hffdb5ce_0_cpython
  - python-dateutil=2.8.1=py_0
  - python-irodsclient=0.8.6=pyhd8ed1ab_0
  - python_abi=3.9=1_cp39
  - pytz=2021.1=pyhd8ed1ab_0
  - pyyaml=5.4.1=py39h3811e60_0
  - ratelimiter=1.2.0=py_1002
  - readline=8.0=he28a2e2_2
  - requests=2.25.1=pyhd3deb0d_0
  - requests-oauthlib=1.3.0=pyh9f0ad1d_0
  - rsa=4.7.2=pyh44b312d_0
  - s3transfer=0.3.6=pyhd8ed1ab_0
  - setuptools=49.6.0=py39hf3d152e_3
  - simplejson=3.17.2=py39h3811e60_2
  - six=1.15.0=pyh9f0ad1d_0
  - slacker=0.14.0=py_0
  - smart_open=5.0.0=pyhd8ed1ab_0
  - smmap=3.0.5=pyh44b312d_0
  - snakemake=6.1.1=hdfd78af_0
  - snakemake-minimal=6.1.1=pyhdfd78af_0
  - sqlite=3.35.4=h74cdb3f_0
  - stone=3.2.1=pyhd8ed1ab_0
  - tk=8.6.10=h21135ba_1
  - toml=0.10.2=pyhd8ed1ab_0
  - toposort=1.6=pyhd8ed1ab_0
  - traitlets=5.0.5=py_0
  - typing-extensions=3.7.4.3=0
  - typing_extensions=3.7.4.3=py_0
  - tzdata=2021a=he74cb21_0
  - ubiquerg=0.6.1=pyh9f0ad1d_0
  - uritemplate=3.0.1=py_0
  - urllib3=1.26.4=pyhd8ed1ab_0
  - veracitools=0.1.3=py_0
  - wcwidth=0.2.5=pyh9f0ad1d_2
  - wheel=0.36.2=pyhd3deb0d_0
  - wrapt=1.12.1=py39h3811e60_3
  - xmlrunner=1.7.7=py_0
  - xorg-kbproto=1.0.7=h7f98852_1002
  - xorg-libice=1.0.10=h7f98852_0
  - xorg-libsm=1.2.3=hd9c2040_1000
  - xorg-libx11=1.7.0=h7f98852_0
  - xorg-libxau=1.0.9=h7f98852_0
  - xorg-libxdmcp=1.1.3=h7f98852_0
  - xorg-libxext=1.3.4=h7f98852_1
  - xorg-libxrender=0.9.10=h7f98852_1003
  - xorg-libxt=1.2.1=h7f98852_2
  - xorg-renderproto=0.11.1=h7f98852_1002
  - xorg-xextproto=7.3.0=h7f98852_1002
  - xorg-xproto=7.0.31=h7f98852_1007
  - xz=5.2.5=h516909a_1
  - yaml=0.2.5=h516909a_0
  - yarl=1.6.3=py39h3811e60_1
  - zipp=3.4.1=pyhd8ed1ab_0
  - zlib=1.2.11=h516909a_1010
  - zstd=1.4.9=ha95c52a_0

Hello Marteen, Thanks for your reply. I copied your list of dependencies and saved them in a snake.yaml file that I added in my epiGBS2 directory. I then proceeded with commands in this order:

conda install -c conda-forge mamba mamba env create -f snake.yaml conda activate snakemake mamba install -c bioconda snakemake=6.1.1

snakemake -n --use-conda #dry run snakemake -j 14 -p --use-conda

All seemed to be OK until the dry run but, when I launched the last command, I got this error:

(snakemake) bruno@Pleroma:~/Desktop/epiGBS2$ snakemake -j 14 -p --use-conda Building DAG of jobs... Creating conda environment src/env/stacks.yaml... Downloading and installing remote packages. CreateCondaEnvironmentException: Could not create conda environment from /home/bruno/Desktop/epiGBS2/src/rules/../env/stacks.yaml: Could not solve for environment specs Encountered problems while solving:

package perl-net-ssleay-1.74-0 requires perl >=5.22.0,<5.23.0, but none of the providers can be installed
package perl-ntlm-1.09-pl526_4 requires perl >=5.26.2,<5.26.3.0a0, but none of the providers can be installed
package python-3.10.0-h543edf9_2_cpython requires openssl >=3.0.0,<4.0a0, but none of the providers can be installed
package perl-lwp-protocol-https-6.07-pl526_4 requires perl-io-socket-ssl, but none of the providers can be installed
package perl-soap-lite-1.19-pl526_1 requires perl-io-socket-ssl, but none of the providers can be installed
package perl-libwww-perl-6.39-pl526_0 requires perl-net-http >=6.18, but none of the providers can be installed
package perl-xml-sax-1.02-pl526_0 requires perl >=5.26.2,<5.26.3.0a0, but none of the providers can be installed
package perl-role-tiny-2.000008-pl526_0 requires perl >=5.26.2,<5.26.3.0a0, but none of the providers can be installed

The environment can't be solved, aborting the operation File "/home/bruno/miniconda3/envs/snakemake/lib/python3.9/site-packages/snakemake/deployment/conda.py", line 388, in create

I tried to set the priority channel of conda to "strict" but I didn't solve the problem...What went wrong? Thanks again for your patience.

Bruno

Hi Bruno, Could you try replacing the contents of src/env/stacks.yaml with the content below?

name: stacksTest2
channels:
  - bioconda
  - conda-forge
  - defaults
  - r
dependencies:
  - _libgcc_mutex=0.1=conda_forge
  - _openmp_mutex=4.5=2_gnu
  - bzip2=1.0.8=h7f98852_4
  - c-ares=1.18.1=h7f98852_0
  - ca-certificates=2022.12.7=ha878542_0
  - cairo=1.16.0=h6cf1ce9_1008
  - expat=2.2.10=h9c3ff4c_0
  - fontconfig=2.13.96=ha180cfb_0
  - freetype=2.12.1=hca18f0e_1
  - fribidi=1.0.10=h36c2ea0_0
  - gettext=0.21.1=h27087fc_0
  - giflib=5.2.1=h36c2ea0_2
  - graphite2=1.3.13=h58526e2_1001
  - graphviz=2.42.3=h0511662_0
  - harfbuzz=2.9.1=h83ec7ef_1
  - htslib=1.16=h6bc39ce_0
  - icu=68.2=h9c3ff4c_0
  - jpeg=9e=h0b41bf4_3
  - keyutils=1.6.1=h166bdaf_0
  - krb5=1.20.1=hf9c8cef_0
  - ld_impl_linux-64=2.40=h41732ed_0
  - libcurl=7.87.0=h6312ad2_0
  - libdb=6.2.32=h9c3ff4c_0
  - libdeflate=1.13=h166bdaf_0
  - libedit=3.1.20191231=he28a2e2_2
  - libev=4.33=h516909a_1
  - libffi=3.4.2=h7f98852_5
  - libgcc-ng=12.2.0=h65d4601_19
  - libgd=2.2.5=h8e06009_4
  - libglib=2.74.1=h606061b_1
  - libgomp=12.2.0=h65d4601_19
  - libiconv=1.17=h166bdaf_0
  - libnghttp2=1.51.0=hdcd2b5c_0
  - libnsl=2.0.0=h7f98852_0
  - libpng=1.6.39=h753d276_0
  - libsqlite=3.40.0=h753d276_0
  - libssh2=1.10.0=haa6b8db_3
  - libstdcxx-ng=12.2.0=h46fd767_19
  - libtiff=4.1.0=hc3755c2_3
  - libtool=2.4.7=h27087fc_0
  - libuuid=2.32.1=h7f98852_1000
  - libwebp=1.0.2=h56121f0_5
  - libxcb=1.13=h7f98852_1004
  - libxml2=2.9.12=h72842e0_0
  - libxslt=1.1.33=h15afd5d_2
  - libzlib=1.2.13=h166bdaf_4
  - llvm-openmp=8.0.1=hc9558a2_0
  - lz4-c=1.9.3=h9c3ff4c_1
  - ncurses=6.3=h27087fc_1
  - openmp=8.0.1=0
  - openssl=1.1.1t=h0b41bf4_0
  - pango=1.42.4=h69149e4_5
  - pcre2=10.40=hc3806b6_0
  - perl=5.26.2=h36c2ea0_1008
  - perl-aceperl=1.92=pl526_2
  - perl-algorithm-munkres=0.08=pl526_1
  - perl-apache-test=1.40=pl526_1
  - perl-app-cpanminus=1.7044=pl526_1
  - perl-appconfig=1.71=pl526_1
  - perl-array-compare=3.0.1=pl526_1
  - perl-autoloader=5.74=pl526_2
  - perl-base=2.23=pl526_1
  - perl-bio-phylo=0.58=pl5262hdfd78af_3
  - perl-bioperl-core=1.007002=pl5262hdfd78af_3
  - perl-business-isbn=3.004=pl526_0
  - perl-business-isbn-data=20140910.003=pl526_0
  - perl-cache-cache=1.08=pl526_0
  - perl-carp=1.38=pl526_3
  - perl-cgi=4.44=pl526h14c3975_1
  - perl-class-inspector=1.34=pl526_0
  - perl-class-load=0.25=pl526_0
  - perl-class-load-xs=0.10=pl526h6bb024c_2
  - perl-class-method-modifiers=2.12=pl526_0
  - perl-clone=0.42=pl526h516909a_0
  - perl-common-sense=3.74=pl526_2
  - perl-compress-raw-zlib=2.087=pl526hc9558a2_0
  - perl-constant=1.33=pl526_1
  - perl-convert-binary-c=0.78=pl526h6bb024c_3
  - perl-convert-binhex=1.125=pl526_1
  - perl-crypt-rc4=2.02=pl526_1
  - perl-data-dumper=2.173=pl526_0
  - perl-data-optlist=0.110=pl526_2
  - perl-data-stag=0.14=pl526_1
  - perl-date-format=2.30=pl526_2
  - perl-db-file=1.855=pl526h516909a_0
  - perl-dbd-sqlite=1.64=pl526h516909a_0
  - perl-dbi=1.642=pl526_0
  - perl-devel-globaldestruction=0.14=pl526_0
  - perl-devel-overloadinfo=0.005=pl526_0
  - perl-devel-stacktrace=2.04=pl526_0
  - perl-digest-hmac=1.03=pl526_3
  - perl-digest-md5=2.55=pl526_0
  - perl-digest-perl-md5=1.9=pl526_1
  - perl-digest-sha1=2.13=pl526h6bb024c_1
  - perl-dist-checkconflicts=0.11=pl526_2
  - perl-dynaloader=1.25=pl526_1
  - perl-email-date-format=1.005=pl526_2
  - perl-encode=2.88=pl526_1
  - perl-encode-locale=1.05=pl526_6
  - perl-error=0.17027=pl526_1
  - perl-eval-closure=0.14=pl526h6bb024c_4
  - perl-exporter=5.72=pl526_1
  - perl-exporter-tiny=1.002001=pl526_0
  - perl-extutils-makemaker=7.36=pl526_1
  - perl-file-listing=6.04=pl526_1
  - perl-file-path=2.16=pl526_0
  - perl-file-slurp-tiny=0.004=pl526_1
  - perl-file-spec=3.48_01=pl526_1
  - perl-file-temp=0.2304=pl526_2
  - perl-file-which=1.23=pl526_0
  - perl-font-afm=1.20=pl526_2
  - perl-font-ttf=1.06=pl526_0
  - perl-gd=2.71=pl526he860b03_0
  - perl-getopt-long=2.50=pl526_1
  - perl-graph=0.9704=pl526_1
  - perl-graphviz=2.24=pl526h734ff71_0
  - perl-html-element-extended=1.18=pl526_1
  - perl-html-entities-numbered=0.04=pl526_1
  - perl-html-formatter=2.16=pl526_0
  - perl-html-parser=3.72=pl526h6bb024c_5
  - perl-html-tableextract=2.13=pl526_2
  - perl-html-tagset=3.20=pl526_3
  - perl-html-tidy=1.60=pl526_0
  - perl-html-tree=5.07=pl526_1
  - perl-html-treebuilder-xpath=0.14=pl526_1
  - perl-http-cookies=6.04=pl526_0
  - perl-http-daemon=6.01=pl526_1
  - perl-http-date=6.02=pl526_3
  - perl-http-message=6.18=pl526_0
  - perl-http-negotiate=6.01=pl526_3
  - perl-image-info=1.38=pl526_1
  - perl-image-size=3.300=pl526_2
  - perl-io-html=1.001=pl526_2
  - perl-io-sessiondata=1.03=pl526_1
  - perl-io-socket-ssl=2.066=pl526_0
  - perl-io-string=1.08=pl526_3
  - perl-io-stringy=2.111=pl526_1
  - perl-io-tty=1.12=pl526_1
  - perl-ipc-run=20180523.0=pl526_0
  - perl-ipc-sharelite=0.17=pl526h6bb024c_1
  - perl-jcode=2.07=pl526_2
  - perl-json=4.02=pl526_0
  - perl-json-xs=2.34=pl526h6bb024c_3
  - perl-lib=0.63=pl526_1
  - perl-libwww-perl=6.39=pl526_0
  - perl-libxml-perl=0.08=pl526_2
  - perl-list-moreutils=0.428=pl526_1
  - perl-list-moreutils-xs=0.428=pl526_0
  - perl-lwp-mediatypes=6.04=pl526_0
  - perl-lwp-protocol-https=6.07=pl526_4
  - perl-lwp-simple=6.15=pl526h470a237_4
  - perl-mailtools=2.21=pl526_0
  - perl-math-cdf=0.1=pl526h14c3975_5
  - perl-math-derivative=1.01=pl526_0
  - perl-math-random=0.72=pl526h14c3975_2
  - perl-math-spline=0.02=pl526_2
  - perl-mime-base64=3.15=pl526_1
  - perl-mime-lite=3.030=pl526_1
  - perl-mime-tools=5.508=pl526_1
  - perl-mime-types=2.17=pl526_0
  - perl-mldbm=2.05=pl526_1
  - perl-module-implementation=0.09=pl526_2
  - perl-module-runtime=0.016=pl526_1
  - perl-module-runtime-conflicts=0.003=pl526_0
  - perl-moo=2.003004=pl526_0
  - perl-moose=2.2011=pl526hf484d3e_1
  - perl-mozilla-ca=20180117=pl526_1
  - perl-mro-compat=0.13=pl526_0
  - perl-net-http=6.19=pl526_0
  - perl-net-ssleay=1.88=pl526h90d6eec_0
  - perl-ntlm=1.09=pl526_4
  - perl-ole-storage_lite=0.19=pl526_3
  - perl-package-deprecationmanager=0.17=pl526_0
  - perl-package-stash=0.38=pl526hf484d3e_1
  - perl-package-stash-xs=0.28=pl526hf484d3e_1
  - perl-params-util=1.07=pl526h6bb024c_4
  - perl-parent=0.236=pl526_1
  - perl-parse-recdescent=1.967015=pl526_0
  - perl-pathtools=3.75=pl526h14c3975_1
  - perl-pdf-api2=2.035=pl526_0
  - perl-pod-escapes=1.07=pl526_1
  - perl-pod-usage=1.69=pl526_1
  - perl-posix=1.38_03=pl526_1
  - perl-postscript=0.06=pl526_2
  - perl-role-tiny=2.000008=pl526_0
  - perl-scalar-list-utils=1.52=pl526h516909a_0
  - perl-set-scalar=1.29=pl526_2
  - perl-soap-lite=1.19=pl526_1
  - perl-socket=2.027=pl526_1
  - perl-sort-naturally=1.03=pl526_2
  - perl-spreadsheet-parseexcel=0.65=pl526_2
  - perl-spreadsheet-writeexcel=2.40=pl526_2
  - perl-statistics-descriptive=3.0702=pl526_0
  - perl-storable=3.15=pl526h14c3975_0
  - perl-sub-exporter=0.987=pl526_2
  - perl-sub-exporter-progressive=0.001013=pl526_0
  - perl-sub-identify=0.14=pl526h14c3975_0
  - perl-sub-install=0.928=pl526_2
  - perl-sub-name=0.21=pl526_1
  - perl-sub-quote=2.006003=pl526_1
  - perl-svg=2.84=pl526_0
  - perl-svg-graph=0.02=pl526_3
  - perl-task-weaken=1.06=pl526_0
  - perl-template-toolkit=2.26=pl526_1
  - perl-test=1.26=pl526_1
  - perl-test-harness=3.42=pl526_0
  - perl-test-leaktrace=0.16=pl526h14c3975_2
  - perl-test-requiresinternet=0.05=pl526_0
  - perl-tie-ixhash=1.23=pl526_2
  - perl-time-hires=1.9760=pl526h14c3975_1
  - perl-time-local=1.28=pl526_1
  - perl-timedate=2.30=pl526_1
  - perl-tree-dag_node=1.31=pl526_0
  - perl-try-tiny=0.30=pl526_1
  - perl-type-tiny=1.004004=pl526_0
  - perl-types-serialiser=1.0=pl526_2
  - perl-unicode-map=0.112=pl526h6bb024c_3
  - perl-uri=1.76=pl526_0
  - perl-www-robotrules=6.02=pl526_3
  - perl-xml-dom=1.46=pl526_0
  - perl-xml-dom-xpath=0.14=pl526_1
  - perl-xml-filter-buffertext=1.01=pl526_2
  - perl-xml-libxml=2.0132=pl526h7ec2d77_1
  - perl-xml-libxslt=1.94=pl526_1
  - perl-xml-namespacesupport=1.12=pl526_0
  - perl-xml-parser=2.44=pl526h4e0c4b3_7
  - perl-xml-regexp=0.04=pl526_2
  - perl-xml-sax=1.02=pl526_0
  - perl-xml-sax-base=1.09=pl526_0
  - perl-xml-sax-expat=0.51=pl526_3
  - perl-xml-sax-writer=0.57=pl526_0
  - perl-xml-simple=2.25=pl526_1
  - perl-xml-twig=3.52=pl526_2
  - perl-xml-writer=0.625=pl526_2
  - perl-xml-xpath=1.44=pl526_0
  - perl-xml-xpathengine=0.14=pl526_2
  - perl-xsloader=0.24=pl526_0
  - perl-yaml=1.29=pl526_0
  - pip=23.0.1=pyhd8ed1ab_0
  - pixman=0.40.0=h36c2ea0_0
  - pthread-stubs=0.4=h36c2ea0_1001
  - python=3.11.0=h10a6764_1_cpython
  - readline=8.1.2=h0f457ee_0
  - samtools=1.16.1=h6899075_1
  - setuptools=67.4.0=pyhd8ed1ab_0
  - stacks=2.55=h9a82719_1
  - tidyp=1.04=hec16e2b_4
  - tk=8.6.12=h27826a3_0
  - tzdata=2022g=h191b570_0
  - velvet=1.2.10=h7132678_5
  - wheel=0.38.4=pyhd8ed1ab_0
  - xorg-kbproto=1.0.7=h7f98852_1002
  - xorg-libice=1.0.10=h7f98852_0
  - xorg-libsm=1.2.3=hd9c2040_1000
  - xorg-libx11=1.7.2=h7f98852_0
  - xorg-libxau=1.0.9=h7f98852_0
  - xorg-libxdmcp=1.1.3=h7f98852_0
  - xorg-libxext=1.3.4=h0b41bf4_2
  - xorg-libxpm=3.5.13=h7f98852_0
  - xorg-libxrender=0.9.10=h7f98852_1003
  - xorg-libxt=1.2.1=h7f98852_2
  - xorg-renderproto=0.11.1=h7f98852_1002
  - xorg-xextproto=7.3.0=h0b41bf4_1003
  - xorg-xproto=7.0.31=h7f98852_1007
  - xz=5.2.6=h166bdaf_0
  - zlib=1.2.13=h166bdaf_4
  - zstd=1.4.9=ha95c52a_0
prefix: /scratch/conda/envs/stacksTest2

Hello Marteen, I'm not sure if I understood your instructions correctly, but I restarted with the installation procedure with these commands:

git clone https://github.com/nioo-knaw/epiGBS2.git cd epiGBS2 added snake.yaml (with indicated modifications in your last post) conda config --set channel_priority flexible mamba env create -f snake.yaml
conda activate stacksTest2 mamba install -c bioconda snakemake=6.1.1

However I got this error:

Could not solve for environment specs Encountered problems while solving:

package snakemake-6.1.1-hdfd78af_0 requires snakemake-minimal 6.1.1.*, but none of the providers can be installed

The environment can't be solved, aborting the operation

What it is wrong with the procedure? Sorry to bother you again and thanks for your kind help

Bruno

Hi Bruno, The correct procedure would be:

git clone https://github.com/nioo-knaw/epiGBS2.git
cd epiGBS2
add the snake.yaml with the contents in my first big content reply
replace the contents of `src/env/stacks.yaml' with the second content reply
mamba env create -f snake.yaml
conda activate snakemake
run the pipeline using
snakemake -j 14 -p --use-conda

The issue you encounter is that conda does not quite remember how to install all of the dependencies for the snakemake version 6.1.1 The contents of the first post contain all of required dependencies to install this specific version and the mamba env create -f command install these dependencies from the file. After this snakemake should be correctly installed. The second error is similar in that the contents of stacks.yaml seem to include an error causing an incorrect combination of packages to be installed. The second post with contents contains the content of a file that works for me to install the stacks pipeline. By replacing the contents of src/env/stacks.yaml snakemake will install the environment when running it using the new file contents These steps seem to work on my machine but before adding it to the pipeline I would like to check if these also work on your configuration I can add it to the pipeline itself. Greetings, Maarten

Hello Marteen, Thanks for your detailed and exhaustive explanation: now the overall scenario is clear and I finally completed the installation procedure, with correct dependencies and no conflicts. The dry run run smoothly so I tried a run with your trial data. The program performed 88 out 125 steps and produced several outputs even if some are missing. I attached the full snakemake .log file with details of process: 2023-03-01T154601.194915.snakemake.log What do you think can I correct to fully terminate the run? Thanks again for your support. Greetings,

Bruno

Hi Bruno, In need the following log to say more about what went wrong /home/bruno/Desktop/epiGBS2/output/output_denovo/make_reference.log Greetings, Maarten

Hello Marteen, Here the required log make_reference.log Thanks. Greetings,

Bruno

So this seems to be another issue with the environment... One of the libraries needed for the version of samtools does not get installed (although a backwards compatible version does get installed!) Unfortunately I am not able to reproduce this on my system but I have two possible temporary work arounds. could you try and run this command: ln -s .snakemake/conda/c0d3a528ab6f784a716676ee6ba38bdb/lib/libncurses.so.6.2 .snakemake/conda/c0d3a528ab6f784a716676ee6ba38bdb/lib/libncurses.so.5 This will create a link to the library with the name that is necessary for one of the programs to run. Afterwards you can try running the pipeline again.

The second option requires to install the libraries outside of conda which probably requires adminstration rights: To do this you can try and run this command or ask your server admin to install the libncurses package sudo apt-get install libncurses5-dev libncursesw5-dev

Hope this helps, Maarten

Hello Maarten, I followed your instruction, but it seems the link I created:

lrwxrwxrwx 1 bruno bruno 71 mar 2 11:27 libncurses.so.5 -> .snakemake/conda/c0d3a528ab6f784a716676ee6ba38bdb/lib/libncurses.so.6.2

is broken since it is missing the library libncurses.so.5. I then tried the second instruction, but it seems that, despite the command, it installs the latest version of the library:

(base) bruno@Pleroma:~/Desktop/epiGBS2/.snakemake/conda/c0d3a528ab6f784a716676ee6ba38bdb/lib$ sudo apt-get install libncurses5-dev libncursesw5-dev

Reading package lists... Done Building dependency tree... Done Reading state information... Done libncurses5-dev is already the newest version (6.3+20220423-2). libncursesw5-dev is already the newest version (6.3+20220423-2). 0 upgraded, 0 newly installed, 0 to remove and 3 not upgraded.

I suppose I need the .so.5: is that correct? How can I fix the old installation problem? Thanks in advance for your response and your kindness.

Bruno

I made a mistake in the link command: Could you try this: ln -s /home/bruno/Desktop/epiGBS2/.snakemake/conda/c0d3a528ab6f784a716676ee6ba38bdb/lib/libncurses.so.6.2 ./home/bruno/Desktop/epiGBS2/.snakemake/conda/c0d3a528ab6f784a716676ee6ba38bdb/lib/libncurses.so.5

In this way the path to the new libncurses.so.5 is always the same. More googling also suggested: sudo apt install libncurses5

Hello Maarten, I tried to install libncurses5 with:

sudo apt install libncurses5

but there is no way to make it successful and so the creation of the link is not possible...It seems that libncurses.so.6.2, that is necessary for some programs, couldn't coexists with libncurses.so.5 that is fundamental with Samtools...

H Bruno, You are correct, there is indeed a problem with the samtools version in the denovo creation step expecting libncurses.so.5. Samtools is used to create the index of the denovo reference. however this index does not seem to be used. As a work around I would try commenting out (placing a # infront of) or removing line 51 in src/rules/denovo.rules.

 index_ref=expand("{path}{file}", path=config["output_dir"], file="/output_denovo/consensus_cluster.renamed.fa.fai")

I would recommend removing the comma at the end of line 50 aswell. This should allow the pipeline to continue without creating the reference index using samtools. Greetings, Maarten

Hello Maarten, I disabled the command of Samtools in denovo .rules as you suggested and I got a huge improving in the overall process: the multiQC_report.html is produced along with all other expected directories and files, but except the snp_calling directory. This seems to be caused by a problem with /tmp directory, as confirmed by the error in bash (which I carelessly deleted and is not present in the snakemake log). Given that I have a separate partition for my home (which contains the directory epiGBS2 with all its installation), and the other linux directories (among which /tmp) are in another partition, I changed the usual setting of the file config.yaml:

tmpdir : "/home/bruno/Desktop/epiGBS2/tmp"

and all went smooth, with production of missing snp_calling directory and its files. I don't want to be tedious, but I reported the process for users that have a Linux installation with a separate partition for /home. I have an off topic question: did you use any particular software to create the nice figure of library structure in the article of epiGBS2? Thanks a lot for your kind support and patience!

Bruno

Hi Bruno, Thanks for being so patient with the issues you encountered. I will start work on updating the pipeline so all of the changes will be incorporated! All of the figures were created in R. (except for the conceptual figures) Kind regards, Maarten

nioo-knaw / epiGBS2

Problems with installation #17