search

ythuang0522 / homopolish

High-quality Nanopore-only genome polisher
GNU General Public License v3.0
65 stars 12 forks source link

Mamba installation with strict channel priority fails for homopolish v0.4.1 #57

Closed ezherman closed 1 year ago

ezherman commented 1 year ago

I have found that the suggested mamba installation command fails for homopolish v0.4.1 under strict channel priority. Below I outline the steps that I took. I suggest adding the anaconda channel to the mamba command and adding that users should use flexible channel priority in order to install this version of homopolish, so long as the conflicting package (libedit) is not updated on conda-forge.

The default mamba command

The command suggested in the README by default installs homopolish v0.3.3.

mamba create -n homopolish -c conda-forge -c bioconda homopolish

The above command returns:

Looking for: ['homopolish']

conda-forge/linux-64                                        Using cache
conda-forge/noarch                                          Using cache
bioconda/linux-64                                           Using cache
bioconda/noarch                                             Using cache
pkgs/r/linux-64                                               No change
pkgs/main/linux-64                                            No change
pkgs/main/noarch                                              No change
pkgs/r/noarch                                                 No change
warning  libmamba Extracted package cache '/home/elh605/miniconda3/pkgs/more-itertools-8.4.0-py_0' has invalid size
warning  libmamba Extracted package cache '/home/elh605/miniconda3/pkgs/more-itertools-8.4.0-py_0' has invalid SHA-256 checksum
warning  libmamba Package tarball '/home/elh605/miniconda3/pkgs/more-itertools-8.4.0-py_0.tar.bz2' is invalid
warning  libmamba Extracted package cache '/home/elh605/miniconda3/pkgs/joblib-0.15.1-py_0' has invalid size
warning  libmamba Extracted package cache '/home/elh605/miniconda3/pkgs/joblib-0.15.1-py_0' has invalid SHA-256 checksum
warning  libmamba Package tarball '/home/elh605/miniconda3/pkgs/joblib-0.15.1-py_0.tar.bz2' is invalid
warning  libmamba Extracted package cache '/home/elh605/miniconda3/pkgs/urllib3-1.25.11-py_0' has invalid size
warning  libmamba Extracted package cache '/home/elh605/miniconda3/pkgs/urllib3-1.25.11-py_0' has invalid SHA-256 checksum
warning  libmamba Package tarball '/home/elh605/miniconda3/pkgs/urllib3-1.25.11-py_0.tar.bz2' is invalid
Transaction

  Prefix: /home/elh605/miniconda3/envs/homopolish

  Updating specs:

   - homopolish

  Package                    Version  Build                 Channel                    Size
─────────────────────────────────────────────────────────────────────────────────────────────
  Install:
─────────────────────────────────────────────────────────────────────────────────────────────

  + _libgcc_mutex                0.1  conda_forge           conda-forge/linux-64     Cached
  + _openmp_mutex                4.5  2_gnu                 conda-forge/linux-64     Cached
  + abseil-cpp            20210324.1  h9c3ff4c_0            conda-forge/linux-64     Cached
  + arrow-cpp                 0.17.1  py37h28458e4_33_cpu   conda-forge/linux-64     Cached
  + aws-c-cal                 0.5.11  h95a6274_0            conda-forge/linux-64     Cached
  + aws-c-common               0.6.2  h7f98852_0            conda-forge/linux-64     Cached
  + aws-c-event-stream         0.2.7  h3541f99_13           conda-forge/linux-64     Cached
  + aws-c-io                  0.10.5  hfb6a706_0            conda-forge/linux-64     Cached
  + aws-checksums             0.1.11  ha31a3da_7            conda-forge/linux-64     Cached
  + aws-sdk-cpp              1.8.186  hecaee15_4            conda-forge/linux-64     Cached
  + biopython                   1.76  py37h516909a_0        conda-forge/linux-64     Cached
  + boost-cpp                 1.74.0  hc6e9bd1_3            conda-forge/linux-64     Cached
  + brotli                     1.0.9  h166bdaf_8            conda-forge/linux-64     Cached
  + brotli-bin                 1.0.9  h166bdaf_8            conda-forge/linux-64     Cached
  + brotlipy                   0.7.0  py37h540881e_1004     conda-forge/linux-64     Cached
  + bzip2                      1.0.8  h7f98852_4            conda-forge/linux-64     Cached
  + c-ares                    1.18.1  h7f98852_0            conda-forge/linux-64     Cached
  + ca-certificates        2022.12.7  ha878542_0            conda-forge/linux-64     Cached
  + capnproto                 0.10.3  h6239696_0            conda-forge/linux-64     Cached
  + certifi                2022.12.7  pyhd8ed1ab_0          conda-forge/noarch       Cached
  + cffi                      1.15.1  py37h43b0acd_1        conda-forge/linux-64     Cached
  + chardet                    3.0.4  py37he5f6b98_1008     conda-forge/linux-64     Cached
  + cryptography              38.0.2  py37h38fbfac_1        conda-forge/linux-64     Cached
  + feather-format             0.4.1  pyh9f0ad1d_0          conda-forge/noarch       Cached
  + gflags                     2.2.2  he1b5a44_1004         conda-forge/linux-64     Cached
  + glog                       0.5.0  h48cff8f_0            conda-forge/linux-64     Cached
  + grpc-cpp                  1.37.1  h2519f57_2            conda-forge/linux-64     Cached
  + gsl                          2.4  h294904e_1006         conda-forge/linux-64     Cached
  + homopolish                 0.3.3  pyh5e36f6f_0          bioconda/noarch          Cached
  + icu                         68.2  h9c3ff4c_0            conda-forge/linux-64     Cached
  + idna                        2.10  pyh9f0ad1d_0          conda-forge/noarch       Cached
  + joblib                    0.15.1  py_0                  conda-forge/noarch        207kB
  + k8                         0.2.5  hd03093a_2            bioconda/linux-64        Cached
  + keyutils                   1.6.1  h166bdaf_0            conda-forge/linux-64     Cached
  + krb5                      1.19.3  h3790be6_0            conda-forge/linux-64     Cached
  + ld_impl_linux-64            2.39  hcc3a1bd_1            conda-forge/linux-64     Cached
  + libblas                    3.9.0  16_linux64_openblas   conda-forge/linux-64     Cached
  + libbrotlicommon            1.0.9  h166bdaf_8            conda-forge/linux-64     Cached
  + libbrotlidec               1.0.9  h166bdaf_8            conda-forge/linux-64     Cached
  + libbrotlienc               1.0.9  h166bdaf_8            conda-forge/linux-64     Cached
  + libcblas                   3.9.0  16_linux64_openblas   conda-forge/linux-64     Cached
  + libcurl                   7.86.0  h7bff187_1            conda-forge/linux-64     Cached
  + libedit             3.1.20191231  he28a2e2_2            conda-forge/linux-64     Cached
  + libev                       4.33  h516909a_1            conda-forge/linux-64     Cached
  + libevent                  2.1.10  h9b69904_4            conda-forge/linux-64     Cached
  + libffi                     3.4.2  h7f98852_5            conda-forge/linux-64     Cached
  + libgcc-ng                 12.2.0  h65d4601_19           conda-forge/linux-64     Cached
  + libgfortran-ng            12.2.0  h69a702a_19           conda-forge/linux-64     Cached
  + libgfortran5              12.2.0  h337968e_19           conda-forge/linux-64     Cached
  + libgomp                   12.2.0  h65d4601_19           conda-forge/linux-64     Cached
  + liblapack                  3.9.0  16_linux64_openblas   conda-forge/linux-64     Cached
  + libnghttp2                1.47.0  hdcd2b5c_1            conda-forge/linux-64     Cached
  + libnsl                     2.0.0  h7f98852_0            conda-forge/linux-64     Cached
  + libopenblas               0.3.21  pthreads_h78a6416_3   conda-forge/linux-64     Cached
  + libprotobuf               3.16.0  h780b84a_0            conda-forge/linux-64     Cached
  + libsqlite                 3.40.0  h753d276_0            conda-forge/linux-64     Cached
  + libssh2                   1.10.0  haa6b8db_3            conda-forge/linux-64     Cached
  + libstdcxx-ng              12.2.0  h46fd767_19           conda-forge/linux-64     Cached
  + libthrift                 0.14.1  he6d91bd_2            conda-forge/linux-64     Cached
  + libzlib                   1.2.13  h166bdaf_4            conda-forge/linux-64     Cached
  + lz4-c                      1.9.3  h9c3ff4c_1            conda-forge/linux-64     Cached
  + mash                       2.1.1  he518ae8_0            bioconda/linux-64        Cached
  + minimap2                    2.17  h5bf99c6_4            bioconda/linux-64        Cached
  + more-itertools             8.4.0  py_0                  conda-forge/noarch         36kB
  + ncurses                      6.3  h27087fc_1            conda-forge/linux-64     Cached
  + numpy                     1.18.5  py37h8960a57_0        conda-forge/linux-64     Cached
  + openssl                   1.1.1s  h0b41bf4_1            conda-forge/linux-64     Cached
  + orc                        1.6.7  h89a63ab_2            conda-forge/linux-64     Cached
  + pandas                    0.23.4  py37h637b7d7_1000     conda-forge/linux-64     Cached
  + parquet-cpp                1.5.1  2                     conda-forge/noarch       Cached
  + pip                       22.3.1  pyhd8ed1ab_0          conda-forge/noarch       Cached
  + pyarrow                   0.17.1  py37he56c5da_33_cpu   conda-forge/linux-64     Cached
  + pycparser                   2.21  pyhd8ed1ab_0          conda-forge/noarch       Cached
  + pycurl                    7.45.1  py37haaec8a5_2        conda-forge/linux-64     Cached
  + pyopenssl                 22.1.0  pyhd8ed1ab_0          conda-forge/noarch       Cached
  + pysocks                    1.7.1  py37h89c1867_5        conda-forge/linux-64     Cached
  + python                    3.7.12  hb7a2778_100_cpython  conda-forge/linux-64     Cached
  + python-dateutil            2.8.2  pyhd8ed1ab_0          conda-forge/noarch       Cached
  + python-wget                  3.2  py_0                  conda-forge/noarch       Cached
  + python_abi                   3.7  3_cp37m               conda-forge/linux-64     Cached
  + pytz                      2022.6  pyhd8ed1ab_0          conda-forge/noarch       Cached
  + re2                   2021.04.01  h9c3ff4c_0            conda-forge/linux-64     Cached
  + readline                   8.1.2  h0f457ee_0            conda-forge/linux-64     Cached
  + requests                  2.24.0  pyh9f0ad1d_0          conda-forge/noarch       Cached
  + s2n                       1.0.10  h9b69904_0            conda-forge/linux-64     Cached
  + scikit-learn              0.21.3  py37hcdab131_0        conda-forge/linux-64     Cached
  + scipy                      1.7.3  py37hf2a6cf1_0        conda-forge/linux-64     Cached
  + setuptools                59.8.0  py37h89c1867_1        conda-forge/linux-64     Cached
  + six                       1.16.0  pyh6c4a22f_0          conda-forge/noarch       Cached
  + snappy                     1.1.9  hbd366e4_2            conda-forge/linux-64     Cached
  + sqlite                    3.40.0  h4ff8645_0            conda-forge/linux-64     Cached
  + tk                        8.6.12  h27826a3_0            conda-forge/linux-64     Cached
  + urllib3                  1.25.11  py_0                  conda-forge/noarch         95kB
  + wheel                     0.38.4  pyhd8ed1ab_0          conda-forge/noarch       Cached
  + xz                         5.2.6  h166bdaf_0            conda-forge/linux-64     Cached
  + zlib                      1.2.13  h166bdaf_4            conda-forge/linux-64     Cached
  + zstd                       1.4.9  ha95c52a_0            conda-forge/linux-64     Cached

  Summary:

  Install: 97 packages

  Total download: 338kB

Specifying the latest homopolish version in the mamba command

Specifying the latest version results in a package incompatibility:

mamba create -n homopolish -c conda-forge -c bioconda homopolish=0.4.1

This returns:

Looking for: ['homopolish=0.4.1']

[+] 0.6s
pkgs/main/linux-64                                            No change
bioconda/linux-64                                             No change
pkgs/r/linux-64                                               No change
pkgs/r/noarch                                                 No change
bioconda/noarch                                               No change
pkgs/main/noarch                                              No change
conda-forge/noarch                                  10.6MB @   2.6MB/s  4.4s
conda-forge/linux-64                                28.3MB @   2.3MB/s 13.1s
Could not solve for environment specs
Encountered problems while solving:
  - package homopolish-0.4.1-pyhdfd78af_0 requires libedit >=3.1.20210910, but none of the providers can be installed

Including the anaconda channel for the libedit package

It seems that the libedit version that is required is not yet available on conda-forge. Instead, it's available on anaconda. However, specifying the anaconda channel does not resolve the issue, due to strict repo priority:

mamba create -n homopolish -c conda-forge -c bioconda -c anaconda homopolish=0.4.1 "anaconda::libedit >=3.1.20210910"

This returns:

Looking for: ['homopolish=0.4.1', "anaconda::libedit[version='>=3.1.20210910']"]

conda-forge/linux-64                                        Using cache
conda-forge/noarch                                          Using cache
bioconda/linux-64                                           Using cache
bioconda/noarch                                             Using cache
anaconda/linux-64                                           Using cache
anaconda/noarch                                             Using cache
anaconda/linux-64                                           Using cache
anaconda/noarch                                             Using cache
pkgs/main/noarch                                              No change
pkgs/r/linux-64                                               No change
pkgs/main/linux-64                                            No change
pkgs/r/noarch                                                 No change
Could not solve for environment specs
Encountered problems while solving:
  - package libedit-3.1.20210910-h7f8727e_0 is excluded by strict repo priority

Suggested solution

I suggest that users are asked to set channel priority to flexible prior to installing homopolish, as shown below. Note that the mamba create command takes a while due to the flexible solve. 

conda config --set channel_priority flexible

mamba create -n homopolish -c conda-forge -c bioconda -c anaconda homopolish

conda config --set channel_priority strict # optional return to default behaviour
SeaneryChang commented 1 year ago

@ezherman Thanks for your suggestion We found there are two version of homopolish 0.4.1 on bioconda and just one can build the env successfully. If set the channel_priority strict, it would get wrong dependencies version. We have sent the request for removing the wrong version from bioconda. The temparary solution is to set the certain build number when conda install. conda create -n homopolish_py1 -c conda-forge -c bioconda homopolish=0.4.1=pyhdfd78af_1

ezherman commented 1 year ago

Hi @SeaneryChang, thank you for looking into this! Great that you've found the source of the problem. I will close this issue, as it looks to have a solution now.

ezherman commented 1 year ago

Hi @SeaneryChang, unfortunately I am still getting the dependencies issue with the conda command that you provided. Would you be able to look into this again? And is there any way in which I can help?

Pohuii-Tzeng commented 1 year ago

Hi @ezherman, do you remember you run the command under the strict mode or the flexible mode?

ezherman commented 1 year ago

Hi @Pohuii-Tzeng, thanks for getting back to me! I ran the command under strict mode. This is a requirement for the workflow manager that I am using (Snakemake).

ezherman commented 1 year ago

Hi @Pohuii-Tzeng, I was wondering whether there has been any progress on resolving this issue with bioconda. Thank you!

Pohuii-Tzeng commented 1 year ago

Hello @ezherman. In order to make it downloadable in strict mode, I tried to deal with the dependency conflicts in homopolish these days. But still doesn't find a solution. We decide to suggest users install homopolish in flexible mode like what you suggested to us. But we will do it by adding the parameter in the install command which will only make the install command run in flexible. The command will look like this: conda create -n homopolish --no-channel-priority -c conda-forge -c bioconda homopolish=0.4.1=pyhdfd78af_1

And we will also upload a newer build to bioconda.

ezherman commented 1 year ago

Thanks for your reply @Pohuii-Tzeng! That's fair, I can see that a flexible install is a workable solution. It's not a perfect fix for my use case, where the Snakemake workflow manager requires all environments in the workflow to be installed with the same level of channel priority. But I have found a temporary workaround. I'll close this issue now, thanks again for getting back to me!