search

AstrobioMike / GToTree

A user-friendly workflow for phylogenomics
GNU General Public License v3.0
192 stars 25 forks source link

current version fails downloads with openssl / curl errors #74

Closed spleonard1 closed 1 year ago

spleonard1 commented 1 year ago

Hi! Trying to use the latest builds fails to download data with the following errors:

mamba create -y -n gtotree -c astrobiomik e -c conda-forge -c bioconda -c defaults gtotree

              __    __    __    __
             /  \  /  \  /  \  /  \
            /    \/    \/    \/    \

███████████████/ /██/ /██/ /██/ /████████████████████████ / / \ / \ / \ / \ _ / / _/ _/ _/ \ o _, / _/ ___/ ` |/ ███╗ ███╗ █████╗ ███╗ ███╗██████╗ █████╗ ████╗ ████║██╔══██╗████╗ ████║██╔══██╗██╔══██╗ ██╔████╔██║███████║██╔████╔██║██████╔╝███████║ ██║╚██╔╝██║██╔══██║██║╚██╔╝██║██╔══██╗██╔══██║ ██║ ╚═╝ ██║██║ ██║██║ ╚═╝ ██║██████╔╝██║ ██║ ╚═╝ ╚═╝╚═╝ ╚═╝╚═╝ ╚═╝╚═════╝ ╚═╝ ╚═╝

    mamba (1.1.0) supported by @QuantStack

    GitHub:  https://github.com/mamba-org/mamba
    Twitter: https://twitter.com/QuantStack

█████████████████████████████████████████████████████████████

Looking for: ['gtotree']

conda-forge/osx-64 No change bioconda/osx-64 No change bioconda/noarch No change astrobiomike/osx-64 No change astrobiomike/noarch No change pkgs/main/osx-64 No change pkgs/main/noarch No change pkgs/r/osx-64 No change pkgs/r/noarch No change conda-forge/noarch @ 4.4MB/s 3.1s Transaction

Prefix: /Users/leonard28/miniconda3/envs/gtotree

Updating specs:

───────────────────────────────────────────────────────────────────────────────────────

Downloading and Extracting Packages

Preparing transaction: done Verifying transaction: done Executing transaction: \ #################################################################################### kofamscan version 1.3.0-2 has been successfully installed!

This software needs a database which can be downloaded from ftp.genome.jp/pub/db/kofam/

For more details see ftp://ftp.genome.jp/pub/tools/kofamscan/README

done

To activate this environment, use

 $ mamba activate gtotree

To deactivate an active environment, use

 $ mamba deactivate

(base) houtx11:nancys_bugs leonard28$ conda activate gtotree (gtotree) houtx11:nancys_bugs leonard28$ gtt-hmms

               GToTree pre-packaged HMM SCG-sets

See github.com/AstrobioMike/GToTree/wiki/SCG-sets for more info

The environment variable GToTree_HMM_dir is set to: /Users/leonard28/miniconda3/envs/gtotree/share/gtotree/hmm_sets/

The 15 available pre-packaged HMM SCG-sets include:

   Actinobacteria                    (138 genes)
   Alphaproteobacteria               (117 genes)
   Archaea                            (76 genes)
   Bacteria                           (74 genes)
   Bacteria_and_Archaea               (25 genes)
   Bacteroidetes                      (90 genes)
   Betaproteobacteria                (203 genes)
   Chlamydiae                        (286 genes)
   Cyanobacteria                     (251 genes)
   Epsilonproteobacteria             (260 genes)
   Firmicutes                        (119 genes)
   Gammaproteobacteria               (172 genes)
   Proteobacteria                    (119 genes)
   Tenericutes                        (99 genes)
   Universal_Hug_et_al                (16 genes)

Details can be found in: /Users/leonard28/miniconda3/envs/gtotree/share/gtotree/hmm_sets/hmm-sources-and-info.tsv

(gtotree) houtx11:nancys_bugs leonard28$ gtt-test.sh

Downloading GToTree test data into the subdirectory GToTree-test-data/

Data being pulled from here: https://figshare.com/articles/dataset/GToTree_test_data/19372334

% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 curl: (35) OpenSSL/3.1.0: error:0A000152:SSL routines::unsafe legacy renegotiation disabled

Downloading the small test data failed for some reason :( You can try downloading it yourself from the link printed above and running the test as follows after unpacking it:

GToTree -a GToTree-test-data/ncbi_accessions.txt \ 
        -g GToTree-test-data/genbank_files.txt \ 
        -f GToTree-test-data/fasta_files.txt \ 
        -A GToTree-test-data/amino_acid_files.txt \ 
        -m GToTree-test-data/genome_to_id_map.tsv \ 
        -p GToTree-test-data/pfam_targets.txt \ 
        -H Universal -t -D -j 4 -o GToTree-test-output -F

Then you can compare the output to what is depicted here: https://github.com/AstrobioMike/GToTree/wiki/Installation#test-run

Exiting for now.

I can get a functional version of gtotree (GToTree v1.6.34) by forcing the install of python=3.7 in the environment as the alternate install option in the readme suggests. This will download things appropriately, but trying to upgrade to the latest version (1.7.08) gets me back to that original error.

Thanks for your help!

AstrobioMike commented 1 year ago

hmm, thanks for reporting this and for all the details, @spleonard1!

I'm not sure what's going on yet, i can recreate it on at least one ubuntu system i have access to (ubuntu 20.04), but not on a different one that is ubuntu 22.04, and i can't recreate it on either of my two macs running monterey 12.5.1 and ventura 13.2.1. And i can find some mention of a problem with later openssl versions and ubuntu (e.g., here and here). But i'm not finding any fix yet in my troubleshooting and testing things, whether using an earlier version of openssl or not. In fact, for me, even when i install with python 3.7 and using the older GToTree 1.6.34, things still aren't working for me...

This is a big problem that seems to be related to openssl communicating with the download target site, but i have no idea what to do about it yet.

In an active GToTree v1.6.34 environment where things worked for you, can you run conda list and paste the output for me here so i can see what the versions are of everything in there?

spleonard1 commented 1 year ago

Thanks for taking a look! I am on a Mac, Big Sur 11.7.4, which I unfortunately cannot upgrade because of IT restrictions

(gtotree) houtx11:nancys_bugs leonard28$ conda list

packages in environment at /Users/leonard28/miniconda3/envs/gtotree:

#

Name Version Build Channel

bc 1.07.1 h0d85af4_0 conda-forge biopython 1.79 py37h69ee0a8_2 conda-forge ca-certificates 2022.12.7 h033912b_0 conda-forge dos2unix 7.4.1 0 conda-forge entrez-direct 16.2 h193322a_1 bioconda fasttree 2.1.11 hdcdfbac_1 bioconda gdbm 1.18 h8a0c380_2 conda-forge gettext 0.21.1 h8a4c099_0 conda-forge gmp 6.2.1 h2e338ed_0 conda-forge gtotree 1.6.34 py37_0 astrobiomike gzip 1.12 h5eb16cf_0 conda-forge hmmer 3.3.2 h9722bc1_2 bioconda iqtree 2.2.0.3 h135ad0d_1 bioconda kofamscan 1.3.0 hdfd78af_2 bioconda libblas 3.9.0 16_osx64_openblas conda-forge libcblas 3.9.0 16_osx64_openblas conda-forge libcxx 16.0.0 h71dddab_0 conda-forge libffi 3.4.2 h0d85af4_5 conda-forge libgfortran 5.0.0 11_3_0_h97931a8_31 conda-forge libgfortran5 12.2.0 he409387_31 conda-forge libiconv 1.17 hac89ed1_0 conda-forge libidn2 2.3.4 hb7f2c08_0 conda-forge liblapack 3.9.0 16_osx64_openblas conda-forge libopenblas 0.3.21 openmp_h429af6e_3 conda-forge libsqlite 3.40.0 ha978bb4_0 conda-forge libunistring 0.9.10 h0d85af4_0 conda-forge libzlib 1.2.13 hfd90126_4 conda-forge llvm-openmp 16.0.0 h61d9ccf_0 conda-forge muscle 5.1 hb339e23_1 bioconda ncurses 6.3 h96cf925_1 conda-forge numpy 1.21.6 py37h345d48f_0 conda-forge openssl 3.1.0 hfd90126_0 conda-forge pandas 1.2.3 py37hff173d8_0 conda-forge parallel 20230322 h694c41f_0 conda-forge perl 5.32.1 2_h0d85af4_perl5 conda-forge pip 23.0.1 pyhd8ed1ab_0 conda-forge prodigal 2.6.3 ha5712d3_4 bioconda python 3.7.12 hf3644f1_100_cpython conda-forge python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge python_abi 3.7 3_cp37m conda-forge pytz 2023.3 pyhd8ed1ab_0 conda-forge readline 8.2 h9e318b2_1 conda-forge ruby 3.1.2 hb3742b3_1 conda-forge sed 4.7 h3efe00b_1000 conda-forge setuptools 59.8.0 py37hf985489_1 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge sqlite 3.40.0 h9ae0607_0 conda-forge taxonkit 0.14.1 h527b516_0 bioconda tk 8.6.12 h5dbffcc_0 conda-forge trimal 1.4.1 hcd10b59_6 bioconda wget 1.20.3 hd3787cc_1 conda-forge wheel 0.40.0 pyhd8ed1ab_0 conda-forge xz 5.2.6 h775f41a_0 conda-forge yaml 0.2.5 h0d85af4_2 conda-forge zlib 1.2.13 hfd90126_4 conda-forge

AstrobioMike commented 1 year ago

thanks!

and if you run this in that environment, it is actually able to download it successfully? 

curl -L --retry 10 --fail -o GToTree-test-data.tar.gz https://figshare.com/ndownloader/files/34404560

Or does that give you an error?

If it's successful, it should be about 7MB, e.g.:

ls -lh GToTree-test-data.tar.gz
# 7.0M Mar 31 17:42 GToTree-test-data.tar.gz
spleonard1 commented 1 year ago

"that environment" meaning the functional one? Yes it downloaded just fine (stepping away from this computer for the next 48 hrs so won't be able to try any more troubleshooting until next week :) )

curl -L --retry 10 --fail -o GToTree-test-data.tar.gz https://figshare.com/ndownloader/files/344045 60 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 7163k 100 7163k 0 0 2236k 0 0:00:03 0:00:03 --:--:-- 4605k

AstrobioMike commented 1 year ago

Just notes about this right now:

Still not sure how to deal with this yet, there is some chatter in curl issues (e.g., https://github.com/curl/curl/issues/8943 and https://github.com/curl/curl/issues/9158) about implementing an option that would enable circumventing this, but so far the devs don't want to do that. I'm going to keep poking to try to find another way to deal with it, and keep a hopeful eye on those issues.

Since it's working on some systems and not others (some ubuntu are fine, some macs are fine), that seems to suggest maybe it's not just the openssl configuration, but ¯_(ツ)_/¯ Hopefully it's not an issue for too many specific setups

AstrobioMike commented 1 year ago

Since nothing had improved on this yet, trying some things out i found the certificate problem doesn't happen with zenodo links. So i created versions of the files GToTree used to download from figshare (test data and pre-packages hmms) on zenodo now, and changed the target links. This shouldn't be a problem anymore on any systems as of v1.7.10.

Please anyone re-open this if they still have an issue 👍