search

carpentries-incubator / snakemake-novice-bioinformatics

Introduction to Snakemake for Bioinformatics
https://carpentries-incubator.github.io/snakemake-novice-bioinformatics
Other
16 stars 9 forks source link

Setup for Mac users - was conda: ResolvePackageNotFound #17

Open lexnederbragt opened 2 years ago

lexnederbragt commented 2 years ago

I am trying to install snakemake as per the instructions in https://carpentries-incubator.github.io/snakemake-novice-bioinformatics/setup.html. When I run conda env update --file conda_env_all.yaml using https://carpentries-incubator.github.io/snakemake-novice-bioinformatics/files/conda_env_all.yaml I get a ResolvePackageNotFound. error with a long list of packages:

Collecting package metadata (repodata.json): done
Solving environment: failed

ResolvePackageNotFound:
  - libedit==3.1.20191231=he28a2e2_2
  - openjdk==10.0.2=h14c3975_1015
  - _openmp_mutex==4.5=1_gnu
  - zlib==1.2.11=h516909a_1010
  - pynacl==1.4.0=py39h3811e60_2
  - libsodium==1.0.18=h36c2ea0_1
  - tornado==6.1=py39h3811e60_1
  - bzip2==1.0.8=h7f98852_4
  - curl==7.76.1=h979ede3_1
  - zstd==1.4.9=ha95c52a_0
  - lcms2==2.11=hcbb858e_1
  - libopenblas==0.3.15=pthreads_h8fe5266_0
  - coincbc==2.10.5=hcee13e7_1
  - tk==8.6.10=h21135ba_1
  - libprotobuf==3.15.8=h780b84a_0
  - rich==10.1.0=py39hf3d152e_0
  - yarl==1.5.1=py39h07f9747_0
  - kallisto==0.46.2=h60f4f9f_2
  - libssh2==1.9.0=ha56f1ee_6
  - pthread-stubs==0.4=h36c2ea0_1001
  - googleapis-common-protos==1.53.0=py39hf3d152e_0
  - pango==1.42.4=h69149e4_5
  - freetype==2.10.4=h0708190_1
  - libstdcxx-ng==9.3.0=h6de172a_19
  - xorg-libxau==1.0.9=h7f98852_0
  - pillow==7.2.0=py39h6f3857e_2
  - wrapt==1.12.1=py39h3811e60_3
  - pygraphviz==1.7=py39h78163bd_0
  - markupsafe==1.1.1=py39h3811e60_3
  - xorg-libxext==1.3.4=h7f98852_1
  - lz4-c==1.9.3=h9c3ff4c_0
  - icu==68.1=h58526e2_0
  - fastx_toolkit==0.0.14=0
  - tbb==2020.2=h4bd325d_4
  - aiohttp==3.7.4=py39h3811e60_0
  - xorg-libxpm==3.5.13=h7f98852_0
  - aioeasywebdav==2.4.0=py39hf3d152e_1001
  - protobuf==3.15.8=py39he80948d_0
  - pcre==8.44=he1b5a44_0
  - multidict==5.1.0=py39h3811e60_1
  - pysocks==1.7.1=py39hf3d152e_3
  - pytest==6.2.4=py39hf3d152e_0
  - libglib==2.68.1=h3e27bee_0
  - importlib-metadata==4.0.1=py39hf3d152e_0
  - libpng==1.6.37=h21135ba_2
  - pulp==2.4=py39hf3d152e_0
  - setuptools==49.6.0=py39hf3d152e_3
  - google-crc32c==1.1.2=py39hb81f231_0
  - readline==8.1=h46c0cb4_0
  - chardet==4.0.0=py39hf3d152e_1
  - xorg-libxdmcp==1.1.3=h7f98852_0
  - libgfortran-ng==9.3.0=hff62375_19
  - ncurses==6.2=h58526e2_4
  - giflib==5.2.1=h36c2ea0_2
  - c-ares==1.17.1=h7f98852_1
  - expat==2.3.0=h9c3ff4c_0
  - gettext==0.19.8.1=h0b5b191_1005
  - libxml2==2.9.10=h72842e0_4
  - libgtextutils==0.7=h1b792b2_7
  - pixman==0.40.0=h36c2ea0_0
  - certifi==2021.10.8=py39hf3d152e_0
  - libuuid==2.32.1=h7f98852_1000
  - datrie==0.8.2=py39h3811e60_2
  - libtool==2.4.6=h58526e2_1007
  - libcroco==0.6.13=h5080e8c_2
  - libgfortran5==9.3.0=hff62375_19
  - matplotlib-base==3.4.1=py39h2fa2bec_0
  - simplejson==3.17.2=py39h3811e60_2
  - graphviz==2.42.3=h0511662_0
  - docutils==0.17.1=py39hf3d152e_0
  - grpcio==1.37.1=py39hff7568b_0
  - pyyaml==5.4.1=py39h3811e60_0
  - libwebp==1.0.2=h56121f0_5
  - glib==2.68.1=h9c3ff4c_0
  - xorg-libxrender==0.9.10=h7f98852_1003
  - xorg-kbproto==1.0.7=h7f98852_1002
  - fftw==3.3.9=nompi_h74d3f13_101
  - xz==5.2.5=h516909a_1
  - numpy==1.20.2=py39hdbf815f_0
  - xorg-renderproto==0.11.1=h7f98852_1002
  - jpeg==9d=h36c2ea0_0
  - pkg-config==0.29.2=h36c2ea0_1008
  - glib-tools==2.68.1=h9c3ff4c_0
  - cryptography==3.4.7=py39hbca0aa6_0
  - libiconv==1.16=h516909a_0
  - humanfriendly==9.1=py39hf3d152e_0
  - bcrypt==3.2.0=py39h3811e60_1
  - libgomp==9.3.0=h2828fa1_19
  - libnghttp2==1.43.0=h812cca2_0
  - xorg-libsm==1.2.3=hd9c2040_1000
  - pyrsistent==0.17.3=py39h3811e60_2
  - xorg-xproto==7.0.31=h7f98852_1007
  - jemalloc==5.2.1=h9c3ff4c_5
  - libcrc32c==1.1.1=h9c3ff4c_2
  - yaml==0.2.5=h516909a_0
  - _libgcc_mutex==0.1=conda_forge
  - librsvg==2.46.4=h33a7fed_0
  - openssl==1.1.1k=h7f98852_0
  - jbig==2.1=h7f98852_2002
  - fribidi==1.0.10=h36c2ea0_0
  - graphite2==1.3.13=h58526e2_1001
  - cython==0.29.23=py39he80948d_0
  - xorg-libice==1.0.10=h7f98852_0
  - boost-cpp==1.74.0=hc6e9bd1_2
  - xorg-libxt==1.2.1=h7f98852_2
  - libffi==3.3=h58526e2_2
  - psutil==5.8.0=py39h3811e60_1
  - xorg-libx11==1.7.0=h7f98852_0
  - libcurl==7.76.1=hc4aaa36_1
  - gobject-introspection==1.68.0=py39hcb793ab_1
  - fontconfig==2.13.1=hba837de_1005
  - ca-certificates==2021.10.8=ha878542_0
  - libxcb==1.13=h7f98852_1003
  - jupyter_core==4.7.1=py39hf3d152e_0
  - pandas==1.2.4=py39hde0f152_0
  - future==0.18.2=py39hf3d152e_3
  - libgcc==7.2.0=h69d50b8_2
  - salmon==1.4.0=h84f40af_1
  - perl==5.26.2=h36c2ea0_1008
  - sqlite==3.35.5=h74cdb3f_0
  - brotlipy==0.7.0=py39h3811e60_1001
  - cffi==1.14.5=py39he32792d_0
  - hdf5==1.10.6=nompi_h6a2412b_1114
  - libgcc-ng==9.3.0=h2828fa1_19
  - cairo==1.16.0=h6cf1ce9_1008
  - libtiff==4.1.0=hc3755c2_3
  - kiwisolver==1.3.1=py39h1a9c180_1
  - ld_impl_linux-64==2.35.1=hea4e1c9_2
  - pluggy==0.13.1=py39hf3d152e_4
  - openjpeg==2.3.1=hf7af979_3
  - krb5==1.17.2=h926e7f8_0
  - libev==4.33=h516909a_1
  - harfbuzz==2.8.1=h83ec7ef_0
  - xorg-xextproto==7.3.0=h7f98852_1002
  - python==3.9.2=hffdb5ce_0_cpython
  - coloredlogs==15.0=py39hf3d152e_0

That is 136 out of 239 packages (!). What can be done to sove this?

Additionally, isn't it quite unusual to specify so many packages in detail? Is there a 'minimal' set of packages that are required?

tbooth commented 2 years ago

Hi. I'm looking into this now. As you noted there was some issue with the file.

I've now updated the links and the file so this file should work OK: https://carpentries-incubator.github.io/snakemake-novice-bioinformatics/files/conda_env.yaml

Regarding the number of packages, this is the minimal list.

The big list of packages was created by installing these into an environment, then doing "conda env export" to freeze the whole environment as conda_env.yaml. I believe this is the most reliable option since the decisions made by Conda regarding dependent packages will change over time, and my experience is that eventually these changes will break something.

However, there is now a pared-down version of the YAML file just containing the above list, if you want to install that and let Conda fill in the dependencies: https://carpentries-incubator.github.io/snakemake-novice-bioinformatics/files/conda_env_min.yaml

tbooth commented 2 years ago

Thanks @lexnederbragt for reporting this issue.

I believe this is fixed, but let me know if there are still problems.

lexnederbragt commented 2 years ago

Thanks! However, when I use https://carpentries-incubator.github.io/snakemake-novice-bioinformatics/files/conda_env_min.yaml as is, I get a lot (a lot!) of warnings about conflicts (Mac OS Big Sur, conda 4.12.0). When I instead used this, I got it to work (at least until episode 4).

name: snakemake_dash
channels:
  - conda-forge
  - bioconda
  - defaults
dependencies:
  - fastqc
  - fastx_toolkit
  - gnuplot
  - graphicsmagick
  - graphviz
  - kallisto
  - multiqc
  - salmon
  - snakemake

Versions I got in my environment are:

fastqc=0.11.9
fastx_toolkit=0.0.14
gnuplot=5.4.3
graphicsmagick=1.3.37
graphviz=3.0.0
kallisto=0.48.0
multiqc=1.12
salmon=1.8.0
snakemake=7.6.2

So quite a number of newer versions than in conda_env_min.yaml...

I am going through the lesson and will get back if I see anything that didn't work as expected.

tbooth commented 2 years ago

It does not surprise me that several things do not quite work on the Mac, and it's great that you are able to test and fix these. I'll hold off merging the fixes and pull requests just now, but when you are done I'll roll them into a big fix and then we'll be able to say our course is good for Mac users. (I'm not sure it will ever be good for Windows users!)

tbooth commented 2 years ago

Hi @lexnederbragt once you are happy you have a working Conda setup for Mac then if you can conda env export it and share it here I will add it as an alternative conda_env_macos.yaml file in the repo.