Error in rule dram_download

jotech commented 1 year ago

[x] I checked and didn't found a related issue,e.g. while typing the title
[x] I got an error in the following rule(s): dram_download
[x] I checked the log files indicated indicated in the error message (and the cluster logs if submitted to a cluster)


Error in rule dram_download:
    jobid: 409
    output: [...]dat/db/atlas/DRAM/db, [...]dat/db/atlas/DRAM/DRAM.config
    log: logs/dram/download_dram.log (check log file(s) for error details)
    conda-env: [...]dat/db/atlas/conda_envs/912ad4abbd015a117d8bf7520e1ca315_
    shell:
         DRAM-setup.py prepare_databases  --output_dir [...]dat/db/atlas/DRAM/db  --threads 8  --verbose  --skip_uniref  &> logs/dram/download_dram.log  ;  DRAM-setup.py export_config --output_file [...]dat/db/atlas/DRAM/DRAM.config
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Here is the relevant log output:


Traceback (most recent call last):                                                                                                                                                                                                                                               
  File "[...]dat/db/atlas/conda_envs/912ad4abbd015a117d8bf7520e1ca315_/bin/DRAM-setup.py", line 5, in <module>
    from mag_annotator.database_processing import (prepare_databases, update_dram_forms,
  File "[...]dat/db/atlas/conda_envs/912ad4abbd015a117d8bf7520e1ca315_/lib/python3.10/site-packages/mag_annotator/database_processing.py", line 13, in <module>
    from skbio import read as read_sequence
  File "[...]dat/db/atlas/conda_envs/912ad4abbd015a117d8bf7520e1ca315_/lib/python3.10/site-packages/skbio/__init__.py", line 11, in <module>
    import skbio.io  # noqa
  File "[...]dat/db/atlas/conda_envs/912ad4abbd015a117d8bf7520e1ca315_/lib/python3.10/site-packages/skbio/io/__init__.py", line 247, in <module>
    import_module('skbio.io.format.lsmat')
  File "[...]dat/db/atlas/conda_envs/912ad4abbd015a117d8bf7520e1ca315_/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "[...]dat/db/atlas/conda_envs/912ad4abbd015a117d8bf7520e1ca315_/lib/python3.10/site-packages/skbio/io/format/lsmat.py", line 77, in <module>
    from skbio.stats.distance import DissimilarityMatrix, DistanceMatrix
  File "[...]dat/db/atlas/conda_envs/912ad4abbd015a117d8bf7520e1ca315_/lib/python3.10/site-packages/skbio/stats/distance/__init__.py", line 197, in <module>
    from ._mantel import mantel, pwmantel
  File "[...]dat/db/atlas/conda_envs/912ad4abbd015a117d8bf7520e1ca315_/lib/python3.10/site-packages/skbio/stats/distance/_mantel.py", line 16, in <module>
    from scipy.stats import PearsonRConstantInputWarning
ImportError: cannot import name 'PearsonRConstantInputWarning' from 'scipy.stats' (/zfshome/sukem066/.local/lib/python3.10/site-packages/scipy/stats/__init__.py)

Atlas version 2.16.3

related to #562 but error still exists in latest atlas version
maybe a special version of scikit-bio is needed? https://stackoverflow.com/questions/72744811/error-cannot-import-name-spearmanrconstantinputwarning-from-scipy-stats

SilasK commented 1 year ago

Thank you for creating the issue. I tried to fix it with the above commit Could you try install ing dev version from github or simply modifying the env/dram.yaml file accordingly?

jotech commented 1 year ago

hi @SilasK Thanks a lot for your fast response!

There seems to be another problem to find a fitting version of python:

mamba env create -p [...]/dat/db/atlas/conda_envs/912ad4abbd015a117d8bf7520e1ca315_ --file [...]/envs/atlas/lib/python3.10/site-packages/atlas/workflow/envs/dram.yaml

Looking for: ['python=3.11', "dram[version='>=1.4.6,<1.5']", "scikit-bio[version='>=0.5.8,<0.6']", "pandas[version='>=1.5,<2']"]

Could not solve for environment specs
The following packages are incompatible
├─ dram >=1.4.6,<1.5  is installable and it requires
│  └─ scipy <=1.8.1  with the potential options
│     ├─ scipy [0.17.1|0.18.0|...|1.2.1] would require
│     │  └─ python [2.7* |>=2.7,<2.8.0a0 ], which can be installed;
│     ├─ scipy [0.17.1|0.18.0|0.18.1|0.19.1] would require
│     │  └─ python 3.4* , which can be installed;
│     ├─ scipy [0.17.1|0.18.0|...|1.1.0] would require
│     │  └─ python [3.5* |>=3.5,<3.6.0a0 ], which can be installed;
│     ├─ scipy [0.18.1|0.19.0|0.19.1|1.0.0|1.0.1] would require
│     │  └─ python 3.6* , which can be installed;
│     ├─ scipy [0.19.1|1.0.0|...|1.5.2] would require
│     │  └─ python >=3.6,<3.7.0a0 , which can be installed;
│     ├─ scipy [1.1.0|1.2.0|...|1.7.3] would require
│     │  └─ python >=3.7,<3.8.0a0 , which can be installed;
│     ├─ scipy [1.3.1|1.3.2|...|1.7.3] would require
│     │  └─ python >=3.8,<3.9.0a0 , which can be installed;
│     ├─ scipy [1.4.1|1.5.0|1.5.1|1.5.2|1.5.3] would require
│     │  ├─ python >=3.6,<3.7.0a0 , which can be installed;
│     │  └─ python_abi 3.6.* *_cp36m, which can be installed;
│     ├─ scipy [1.4.1|1.5.0|1.5.1|1.5.2|1.5.3] would require
│     │  └─ python_abi 3.6 *_pypy36_pp73, which can be installed;
│     ├─ scipy [1.4.1|1.5.0|...|1.7.3] would require
│     │  ├─ python >=3.7,<3.8.0a0 , which can be installed;
│     │  └─ python_abi 3.7.* *_cp37m, which can be installed;
│     ├─ scipy [1.4.1|1.5.0|...|1.8.1] would require
│     │  ├─ python >=3.8,<3.9.0a0 , which can be installed;
│     │  └─ python_abi 3.8.* *_cp38, which can be installed;
│     ├─ scipy [1.5.2|1.5.3|...|1.8.1] would require
│     │  ├─ python >=3.9,<3.10.0a0 , which can be installed;
│     │  └─ python_abi 3.9.* *_cp39, which can be installed;
│     ├─ scipy [1.6.0|1.6.1|...|1.7.3] would require
│     │  └─ python_abi 3.7 *_pypy37_pp73, which can be installed;
│     ├─ scipy [1.7.2|1.7.3|1.8.0|1.8.1] would require
│     │  ├─ python >=3.10,<3.11.0a0 , which can be installed;
│     │  └─ python_abi 3.10.* *_cp310, which can be installed;
│     ├─ scipy 1.8.1 would require
│     │  └─ python_abi 3.8 *_pypy38_pp73, which can be installed;
│     ├─ scipy 1.8.1 would require
│     │  └─ python_abi 3.9 *_pypy39_pp73, which can be installed;
│     ├─ scipy [1.5.2|1.6.0|...|1.7.3] would require
│     │  └─ python >=3.9,<3.10.0a0 , which can be installed;
│     └─ scipy 1.7.3 would require
│        └─ python >=3.10,<3.11.0a0 , which can be installed;
└─ python 3.11**  is uninstallable because it conflicts with any installable versions previously reported.

SilasK commented 1 year ago

Ok let's remove the python requirement. Can you pull the latest commit and re-try?

Could you also telle me which versions you had in the initial env?

run:

conda activate [...]dat/db/atlas/conda_envs/912ad4abbd015a117d8bf7520e1ca315_
conda list python
conda list scipy
conda list scikit-learn

conda deactivate

jotech commented 1 year ago

unfortunately, removing the python requirement does not solve the incompatibility. The only valid solution for this I found was to allow an older version of scikit-bio (0.5.7), but this then causes the error reported initially.

Looking for: ['python', "dram[version='>=1.4.6,<1.5']", "scikit-bio[version='>=0.5.8,<0.6']", "pandas[version='>=1.5,<2']"]

Could not solve for environment specs
The following packages are incompatible
├─ dram >=1.4.6,<1.5  is installable and it requires
│  ├─ python >=3.6  with the potential options
│  │  ├─ python [3.8.0|3.8.1] would require
│  │  │  └─ python_abi * *_cp38, which can be installed;
│  │  ├─ python [3.8.0|3.8.1|...|3.8.8], which can be installed;
│  │  ├─ python [3.8.12|3.8.13|3.8.16] would require
│  │  │  └─ python_abi 3.8 *_pypy38_pp73, which conflicts with any installable versions previously reported;
│  │  ├─ python 3.8.5 would require
│  │  │  └─ python_abi 3.8.* *_graalpy223_38_native, which can be installed;
│  │  ├─ python [3.6.0|3.6.1|...|3.6.7] would require
│  │  │  └─ python_abi * *_cp36m, which can be installed;
│  │  ├─ python [3.6.10|3.6.11|...|3.6.9] would require
│  │  │  └─ python_abi 3.6.* *_cp36m, which can be installed;
│  │  ├─ python [3.6.10|3.6.12|...|3.6.9] conflicts with any installable versions previously reported;
│  │  ├─ python [3.7.0|3.7.1|3.7.2|3.7.3|3.7.6] would require
│  │  │  └─ python_abi * *_cp37m, which can be installed;
│  │  ├─ python [3.7.0|3.7.1|...|3.7.9] conflicts with any installable versions previously reported;
│  │  ├─ python [3.7.10|3.7.12|...|3.7.9] would require
│  │  │  └─ python_abi 3.7.* *_cp37m, which can be installed;
│  │  ├─ python [3.9.0|3.9.1|...|3.9.9] conflicts with any installable versions previously reported;
│  │  ├─ python [3.9.10|3.9.12|3.9.16] would require
│  │  │  └─ python_abi 3.9 *_pypy39_pp73, which conflicts with any installable versions previously reported;
│  │  ├─ python [3.10.0|3.10.1|...|3.10.9] conflicts with any installable versions previously reported;
│  │  └─ python [3.11.0|3.11.1|3.11.2|3.11.3|3.11.4] conflicts with any installable versions previously reported;
│  └─ scipy <=1.8.1  with the potential options
│     ├─ scipy [0.17.1|0.18.0|...|1.2.1] would require
│     │  └─ python [2.7* |>=2.7,<2.8.0a0 ], which conflicts with any installable versions previously reported;
│     ├─ scipy [0.17.1|0.18.0|0.18.1|0.19.1] would require
│     │  └─ python 3.4* , which conflicts with any installable versions previously reported;
│     ├─ scipy [0.17.1|0.18.0|...|1.1.0] would require
│     │  └─ python [3.5* |>=3.5,<3.6.0a0 ], which conflicts with any installable versions previously reported;
│     ├─ scipy [0.18.1|0.19.0|0.19.1|1.0.0|1.0.1] would require
│     │  └─ python 3.6*  but there are no viable options
│     │     ├─ python [3.6.0|3.6.1|...|3.6.7], which cannot be installed (as previously explained);
│     │     ├─ python [3.6.10|3.6.11|...|3.6.9], which cannot be installed (as previously explained);
│     │     ├─ python [3.6.10|3.6.12|...|3.6.9] conflicts with any installable versions previously reported;
│     │     └─ python [3.6.0a3|3.6.0a4|...|3.6.0rc1] conflicts with any installable versions previously reported;
│     ├─ scipy [0.19.1|1.0.0|...|1.5.2] would require
│     │  └─ python >=3.6,<3.7.0a0  but there are no viable options
│     │     ├─ python [3.6.0|3.6.1|...|3.6.7], which cannot be installed (as previously explained);
│     │     ├─ python [3.6.10|3.6.11|...|3.6.9], which cannot be installed (as previously explained);
│     │     └─ python [3.6.10|3.6.12|...|3.6.9] conflicts with any installable versions previously reported;
│     ├─ scipy [1.1.0|1.2.0|...|1.7.3] would require
│     │  └─ python >=3.7,<3.8.0a0  but there are no viable options
│     │     ├─ python [3.7.0|3.7.1|3.7.2|3.7.3|3.7.6], which cannot be installed (as previously explained);
│     │     ├─ python [3.7.0|3.7.1|...|3.7.9] conflicts with any installable versions previously reported;
│     │     └─ python [3.7.10|3.7.12|...|3.7.9], which cannot be installed (as previously explained);
│     ├─ scipy [1.3.1|1.3.2|...|1.8.1] would require
│     │  ├─ python >=3.8,<3.9.0a0  with the potential options
│     │  │  ├─ python [3.8.0|3.8.1], which can be installed (as previously explained);
│     │  │  ├─ python [3.8.0|3.8.1|...|3.8.8], which can be installed;
│     │  │  ├─ python [3.8.12|3.8.13|3.8.16], which cannot be installed (as previously explained);
│     │  │  └─ python 3.8.5, which can be installed (as previously explained);
│     │  └─ python_abi 3.8 *_pypy38_pp73, which conflicts with any installable versions previously reported;
│     ├─ scipy [1.4.1|1.5.0|1.5.1|1.5.2|1.5.3] would require
│     │  ├─ python >=3.6,<3.7.0a0  but there are no viable options
│     │  │  ├─ python [3.6.0|3.6.1|...|3.6.7], which cannot be installed (as previously explained);
│     │  │  ├─ python [3.6.10|3.6.11|...|3.6.9], which cannot be installed (as previously explained);
│     │  │  └─ python [3.6.10|3.6.12|...|3.6.9] conflicts with any installable versions previously reported;
│     │  └─ python_abi 3.6.* *_cp36m, which conflicts with any installable versions previously reported;
│     ├─ scipy [1.4.1|1.5.0|1.5.1|1.5.2|1.5.3] would require
│     │  ├─ python >=3.6,<3.7.0a0  but there are no viable options
│     │  │  ├─ python [3.6.0|3.6.1|...|3.6.7], which cannot be installed (as previously explained);
│     │  │  ├─ python [3.6.10|3.6.11|...|3.6.9], which cannot be installed (as previously explained);
│     │  │  └─ python [3.6.10|3.6.12|...|3.6.9] conflicts with any installable versions previously reported;
│     │  └─ python_abi 3.6 *_pypy36_pp73, which conflicts with any installable versions previously reported;
│     ├─ scipy [1.4.1|1.5.0|...|1.7.3] would require
│     │  ├─ python >=3.7,<3.8.0a0  but there are no viable options
│     │  │  ├─ python [3.7.0|3.7.1|3.7.2|3.7.3|3.7.6], which cannot be installed (as previously explained);
│     │  │  ├─ python [3.7.0|3.7.1|...|3.7.9] conflicts with any installable versions previously reported;
│     │  │  └─ python [3.7.10|3.7.12|...|3.7.9], which cannot be installed (as previously explained);
│     │  └─ python_abi 3.7.* *_cp37m, which conflicts with any installable versions previously reported;
│     ├─ scipy [1.4.1|1.5.0|...|1.8.1] would require
│     │  ├─ python >=3.8,<3.9.0a0  with the potential options
│     │  │  ├─ python [3.8.0|3.8.1], which can be installed (as previously explained);
│     │  │  ├─ python [3.8.0|3.8.1|...|3.8.8], which can be installed;
│     │  │  ├─ python [3.8.12|3.8.13|3.8.16], which cannot be installed (as previously explained);
│     │  │  └─ python 3.8.5, which can be installed (as previously explained);
│     │  └─ python_abi 3.8.* *_cp38, which conflicts with any installable versions previously reported;
│     ├─ scipy [1.5.2|1.5.3|...|1.8.1] would require
│     │  ├─ python >=3.9,<3.10.0a0  but there are no viable options
│     │  │  ├─ python [3.9.0|3.9.1|...|3.9.9] conflicts with any installable versions previously reported;
│     │  │  └─ python [3.9.10|3.9.12|3.9.16], which cannot be installed (as previously explained);
│     │  └─ python_abi 3.9.* *_cp39, which conflicts with any installable versions previously reported;
│     ├─ scipy [1.6.0|1.6.1|...|1.7.3] would require
│     │  ├─ python >=3.7,<3.8.0a0  but there are no viable options
│     │  │  ├─ python [3.7.0|3.7.1|3.7.2|3.7.3|3.7.6], which cannot be installed (as previously explained);
│     │  │  ├─ python [3.7.0|3.7.1|...|3.7.9] conflicts with any installable versions previously reported;
│     │  │  └─ python [3.7.10|3.7.12|...|3.7.9], which cannot be installed (as previously explained);
│     │  └─ python_abi 3.7 *_pypy37_pp73, which conflicts with any installable versions previously reported;
│     ├─ scipy [1.7.2|1.7.3|1.8.0|1.8.1] would require
│     │  ├─ python >=3.10,<3.11.0a0 , which conflicts with any installable versions previously reported;
│     │  └─ python_abi 3.10.* *_cp310, which can be installed;
│     ├─ scipy [1.10.0|1.10.1|...|1.9.3] would require
│     │  ├─ python >=3.9,<3.10.0a0  but there are no viable options
│     │  │  ├─ python [3.9.0|3.9.1|...|3.9.9] conflicts with any installable versions previously reported;
│     │  │  └─ python [3.9.10|3.9.12|3.9.16], which cannot be installed (as previously explained);
│     │  └─ python_abi 3.9 *_pypy39_pp73, which conflicts with any installable versions previously reported;
│     ├─ scipy [1.3.1|1.3.2|...|1.6.2] would require
│     │  ├─ mkl >=2019.4,<2021.0a0 , which can be installed;
│     │  └─ python >=3.8,<3.9.0a0  with the potential options
│     │     ├─ python [3.8.0|3.8.1], which can be installed (as previously explained);
│     │     ├─ python [3.8.0|3.8.1|...|3.8.8], which can be installed;
│     │     ├─ python [3.8.12|3.8.13|3.8.16], which cannot be installed (as previously explained);
│     │     └─ python 3.8.5, which can be installed (as previously explained);
│     └─ scipy 1.7.3 would require
│        └─ python >=3.10,<3.11.0a0 , which conflicts with any installable versions previously reported;
├─ pandas >=1.5,<2  is installable with the potential options
│  ├─ pandas [1.5.0|1.5.1|1.5.2|1.5.3] would require
│  │  └─ python >=3.10,<3.11.0a0 , which conflicts with any installable versions previously reported;
│  ├─ pandas [1.5.0|1.5.1|1.5.2|1.5.3] would require
│  │  └─ python >=3.8,<3.9.0a0  with the potential options
│  │     ├─ python [3.8.0|3.8.1], which can be installed (as previously explained);
│  │     ├─ python [3.8.0|3.8.1|...|3.8.8], which can be installed;
│  │     ├─ python [3.8.12|3.8.13|3.8.16], which cannot be installed (as previously explained);
│  │     └─ python 3.8.5, which can be installed (as previously explained);
│  ├─ pandas [1.5.0|1.5.1|1.5.2|1.5.3] would require
│  │  └─ python >=3.9,<3.10.0a0  but there are no viable options
│  │     ├─ python [3.9.0|3.9.1|...|3.9.9] conflicts with any installable versions previously reported;
│  │     └─ python [3.9.10|3.9.12|3.9.16], which cannot be installed (as previously explained);
│  └─ pandas [1.5.1|1.5.2|1.5.3] would require
│     └─ python >=3.11,<3.12.0a0 , which conflicts with any installable versions previously reported;
└─ scikit-bio >=0.5.8,<0.6  is uninstallable because there are no viable options
   ├─ scikit-bio 0.5.8 would require
   │  ├─ python [>=3.10,<3.11.0a0 |>=3.8,<3.9.0a0 |>=3.9,<3.10.0a0 ] with the potential options
   │  │  ├─ python [3.8.0|3.8.1], which can be installed (as previously explained);
   │  │  ├─ python [3.8.0|3.8.1|...|3.8.8], which can be installed;
   │  │  ├─ python [3.8.12|3.8.13|3.8.16], which cannot be installed (as previously explained);
   │  │  ├─ python 3.8.5, which can be installed (as previously explained);
   │  │  ├─ python [3.9.0|3.9.1|...|3.9.9] conflicts with any installable versions previously reported;
   │  │  ├─ python [3.9.10|3.9.12|3.9.16], which cannot be installed (as previously explained);
   │  │  └─ python [3.10.0|3.10.1|...|3.10.9] conflicts with any installable versions previously reported;
   │  ├─ python_abi [3.8.* *_cp38|3.9.* *_cp39], which conflicts with any installable versions previously reported;
   │  └─ scipy >=1.9.0  but there are no viable options
   │     ├─ scipy [1.10.0|1.10.1|...|1.9.3], which cannot be installed (as previously explained);
   │     ├─ scipy [1.9.1|1.9.3] would require
   │     │  ├─ mkl >=2021.4.0,<2022.0a0 , which conflicts with any installable versions previously reported;
   │     │  └─ python >=3.8,<3.9.0a0  with the potential options
   │     │     ├─ python [3.8.0|3.8.1], which can be installed (as previously explained);
   │     │     ├─ python [3.8.0|3.8.1|...|3.8.8], which can be installed;
   │     │     ├─ python [3.8.12|3.8.13|3.8.16], which cannot be installed (as previously explained);
   │     │     └─ python 3.8.5, which can be installed (as previously explained);
   │     ├─ scipy [1.10.0|1.10.1|...|1.9.3] would require
   │     │  └─ python >=3.10,<3.11.0a0 , which conflicts with any installable versions previously reported;
   │     ├─ scipy [1.10.0|1.10.1|...|1.9.3] would require
   │     │  ├─ python >=3.8,<3.9.0a0  with the potential options
   │     │  │  ├─ python [3.8.0|3.8.1], which can be installed (as previously explained);
   │     │  │  ├─ python [3.8.0|3.8.1|...|3.8.8], which can be installed;
   │     │  │  ├─ python [3.8.12|3.8.13|3.8.16], which cannot be installed (as previously explained);
   │     │  │  └─ python 3.8.5, which can be installed (as previously explained);
   │     │  └─ python_abi 3.8.* *_cp38, which conflicts with any installable versions previously reported;
   │     ├─ scipy [1.10.0|1.10.1|...|1.9.3] would require
   │     │  ├─ python >=3.9,<3.10.0a0  but there are no viable options
   │     │  │  ├─ python [3.9.0|3.9.1|...|3.9.9] conflicts with any installable versions previously reported;
   │     │  │  └─ python [3.9.10|3.9.12|3.9.16], which cannot be installed (as previously explained);
   │     │  └─ python_abi 3.9.* *_cp39, which conflicts with any installable versions previously reported;
   │     ├─ scipy [1.10.0|1.10.1|1.9.3] would require
   │     │  └─ python >=3.11,<3.12.0a0 , which conflicts with any installable versions previously reported;
   │     ├─ scipy [1.10.0|1.10.1] would require
   │     │  ├─ pypy3.8 >=7.3.11 , which conflicts with any installable versions previously reported;
   │     │  └─ python >=3.8,<3.9.0a0  with the potential options
   │     │     ├─ python [3.8.0|3.8.1], which can be installed (as previously explained);
   │     │     ├─ python [3.8.0|3.8.1|...|3.8.8], which can be installed;
   │     │     ├─ python [3.8.12|3.8.13|3.8.16], which cannot be installed (as previously explained);
   │     │     └─ python 3.8.5, which can be installed (as previously explained);
   │     ├─ scipy [1.10.0|1.10.1] would require
   │     │  ├─ pypy3.9 >=7.3.11 , which conflicts with any installable versions previously reported;
   │     │  └─ python >=3.9,<3.10.0a0  but there are no viable options
   │     │     ├─ python [3.9.0|3.9.1|...|3.9.9] conflicts with any installable versions previously reported;
   │     │     └─ python [3.9.10|3.9.12|3.9.16], which cannot be installed (as previously explained);
   │     ├─ scipy [1.10.0|1.10.1|1.9.1|1.9.3] would require
   │     │  └─ python >=3.8,<3.9.0a0  with the potential options
   │     │     ├─ python [3.8.0|3.8.1], which can be installed (as previously explained);
   │     │     ├─ python [3.8.0|3.8.1|...|3.8.8], which can be installed;
   │     │     ├─ python [3.8.12|3.8.13|3.8.16], which cannot be installed (as previously explained);
   │     │     └─ python 3.8.5, which can be installed (as previously explained);
   │     └─ scipy 1.9.3 would require
   │        ├─ mkl >=2023.1.0,<2024.0a0 , which can be installed;
   │        └─ python >=3.8,<3.9.0a0  with the potential options
   │           ├─ python [3.8.0|3.8.1], which can be installed (as previously explained);
   │           ├─ python [3.8.0|3.8.1|...|3.8.8], which can be installed;
   │           ├─ python [3.8.12|3.8.13|3.8.16], which cannot be installed (as previously explained);
   │           └─ python 3.8.5, which can be installed (as previously explained);
   └─ scikit-bio 0.5.8 would require
      └─ python >=3.11,<3.12.0a0 , which conflicts with any installable versions previously reported.

Version numbers from the old environment:

conda list python
#
# Name                    Version                   Build  Channel
ipython                   8.14.0             pyh41d4057_0    conda-forge
msgpack-python            1.0.5           py310hdf3cbec_0    conda-forge
python                    3.10.11         he550d4f_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python_abi                3.10                    3_cp310    conda-forge

conda list scipy
#
# Name                    Version                   Build  Channel
scipy                     1.8.1           py310hdfbd76f_3    conda-forge

conda list scikit-learn
#
# Name                    Version                   Build  Channel
scikit-learn              1.2.2           py310hf7d194e_2    conda-forge

SilasK commented 1 year ago

Can you try again please with the latest commit?

jotech commented 1 year ago

I tried a fresh install with the latest developmental version, but it still doesn't work. I would assume that the error does not occur in your case because the dram database might be already prepared?

Thank you for your ideas and then let's see what the dram people reply.

SilasK commented 1 year ago

I made a new bioconda release of dram (same version but a new build) that should handle all the dependencies. This will prpbalby take some minutes to be available.

I also simplified the environment in atlas on the master-branch. atlas v 2.17 If you prefer not to update to atlas v2.17 you can use the branch fix-664 that has the simplified environment but still atlas v2.16.*

jotech commented 1 year ago

hi @SilasK Thank you for the update! I tried the new version with the new dram, but unfortunately, as far as I can see, the error with the PearsonRConstantInputWarning persists.

conda activate [...]/atlas/conda_envs/bd5b0fdab3853fca343b76be3f6692de_

 conda list scikit-learn
# packages in environment at [...]/atlas/conda_envs/bd5b0fdab3853fca343b76be3f6692de_:
#
# Name                    Version                   Build  Channel
scikit-learn              1.2.2           py310hf7d194e_2    conda-forge

conda list scikit-learn
# packages in environment at [...]/atlas/conda_envs/bd5b0fdab3853fca343b76be3f6692de_:
#
# Name                    Version                   Build  Channel
scikit-learn              1.2.2           py310hf7d194e_2    conda-forge

conda list python
# packages in environment at [...]/atlas/conda_envs/bd5b0fdab3853fca343b76be3f6692de_:
#
# Name                    Version                   Build  Channel
ipython                   8.14.0             pyh41d4057_0    conda-forge
msgpack-python            1.0.5           py310hdf3cbec_0    conda-forge
python                    3.10.11         he550d4f_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python_abi                3.10                    3_cp310    conda-forge


>> from scipy.stats import PearsonRConstantInputWarning 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'PearsonRConstantInputWarning' from 'scipy.stats' ([...]/.local/lib/python3.10/site-packages/scipy/stats/__init__.py)

SilasK commented 1 year ago

I don't know why the package does not work on your server.

What you can try is to enter the conda env and reinstall dram:

conda activate [...]/atlas/conda_envs/bd5b0fdab3853fca343b76be3f6692de_
mamba uninstall dram
mamba install dram=1.4.6 

#test dram
DRAM-setup.py --help

conda deactivate

This should limit the version of scipy and scikit-bio (not scikit-learn)

could you test if this is the case.

scipy <=1.8.1 scikit-bio<=0.5.7

jotech commented 1 year ago

Unfortunately, it's still not working. I reinstalled it from scratch and also installed dram=1.4.6 on a fresh system, and the same error occurs.

But I'm still a bit confused, don't we need scikit-bio 0.5.8 as mentioned here scikit-bio/issues/1818

When I just install dram with scikit-bio=0.5.8 in a fresh conda, the error does not occur, although this installs dram-1.3.5 and scipy-1.11. At least then I can download the databases needed by dram, although I don't know if the dram=1.3.5 causes any errors within the atlas pipeline?

SilasK commented 1 year ago

I made a new dram build, and it passed all the tests. e.g. it run DRAM-setup.py --help without error.

Did you already get an error when running DRAM-setup.py --help ?

As you noticed there are some conflicts between versions of scipy and scikit-bio. I tried to solve this at multiple levels.

There are two solutions. keep everything at the latest version when it should work (that's what I tried. sckit-bio < 5.8, scipy < 1.8) https://github.com/WrightonLabCSU/DRAM/issues/191#issuecomment-1201699237

Or update to the most recent version when it should work again.

But scikit-bio 5.8 requires python 3.11, which I think is not compatible with everything.

jotech commented 1 year ago

Yes I'm getting the error already with DRAM-setup.py --help when using dram=1.4.6 and scikit-bio=0.5.7

with scikit-bio=0.5.8 and dram=1.3.5 it works but some downloads fail here because of the older dram version (ncbi removed some files).

I'm really stuck here, it is really strange that it is not reproducible although we both use conda and the same software versions.

My overall problem is that I cannot continue with the atlas pipeline because the failed dram download causes the whole pipeline to stop even when I exclude dram analysis from the config.yaml

SilasK commented 1 year ago

My overall problem is that I cannot continue with the atlas pipeline because the failed dram download causes the whole pipeline to stop even when I exclude dram analysis from the config.yaml

If you comment both kegg_modules and dram in the annotations file you should be able to run atlas run genomes

annotations:
  - gtdb_tree
  - gtdb_taxonomy
  - genes
 # - kegg_modules
 # - dram

SilasK commented 1 year ago

With dram=1.4.6 and scikit-bio=0.5.7 which version of scipy do you have?

SilasK commented 1 year ago

It could be that you re-installed the dram v1.4.6 build 0, and not my new build nr 1. Cou can check whcih build you have with conda list dram however I don't know if there is a way to specify the build. Usually, de-install and reinstall should work.

I managed to install dram on a colab using bioconda and also using the latest version of scikit-bio.. https://colab.research.google.com/drive/1q6OhFD12tZJJznFHCH6uNHQIJh0B4Dke?usp=sharing

jotech commented 1 year ago

With dram=1.4.6 and scikit-bio=0.5.7 which version of scipy do you have?

scipy=1.8.1

Cou can check whcih build you have with conda list dram

# Name                    Version                   Build  Channel
dram                      1.4.6              pyhdfd78af_1    bioconda

This should be the latest build, shouldn't it?

SilasK commented 1 year ago

yes it is the latest version. I don't know why it doesn't work. In the colab it works with exactly the same versions.

What you can try to delete all atlas-generated conda envs that contain dram, and try again.

to get list of al conda envs with dram:

grep dram [...]/atlas/conda_envs/*.y*

jotech commented 1 year ago

I already tried deleting all conda environments and also a fresh install on a different system, it made no difference.

I was wondering whether there could be another installation of the packages with a higher priority? But the error messages is pointing to the libraries from the conda environment.

Maybe we could try to have exact the same environment if you could provide the list of all packages? (conda list --export > requirements.txt)

SilasK commented 1 year ago

Have a look at the cobal link I send you. It produces the requirements.txt files it also gives a way to isntall Dram with the newest versions of scipy and scikit-bio.

conda config --set channel_priority strict might have an importance.

jotech commented 1 year ago

thanks for the advice! I will check it out and report back.

As a short reply to your previous advice, by commenting out dram and kegg_modules in config.yaml I can indeed continue with the pipeline.

jotech commented 1 year ago

Now I can confirm that using strict priorities indeed solved the issue! Many thanks for your assistance!

metagenome-atlas / atlas

Error in rule dram_download #664