search

popsim-consortium / stdpopsim

A library of standard population genetic models
GNU General Public License v3.0
122 stars 87 forks source link

msprime>=1.0.0a6 not available on conda #792

Closed andrewkern closed 3 years ago

andrewkern commented 3 years ago

this will break install directions for anyone following our current conda guide

jeromekelleher commented 3 years ago

Not much we can do about it, unfortunately, conda doesn't do pre-releases at all well. We'll have to update the docs for developer installs until msprime 1. comes out in April

Something like

Would it be possible to get a conda user to do this? I'm just guessing a lot of the time with conda stuff.

grahamgower commented 3 years ago

The following worked for me (probably).

conda create -n stdpopsim --yes python=3.8
conda activate stdpopsim
conda install --yes gsl
pip install --pre msprime
# using a fresh stdpopsim clone, just to be sure
git clone https://github.com/popsim-consortium/stdpopsim.git
cd stdpopsim
pip install -r requirements/development.txt
python setup.py install

There seems to be an issue with the SHA256 for the PonAbe recombination map(s) though. Did these files get modified recently?

$ pytest tests/
....
_______________________________________________ ERROR at setup of TestGeneticMap.test_cache_dirs _______________________________________________
[gw1] linux -- Python 3.8.8 /home/grg/miniconda3/envs/stp/bin/python3.8

    def setUpModule():
        destination = pathlib.Path("_test_cache/tarballs")
        for genetic_map in stdpopsim.all_genetic_maps():
            key = genetic_map.id
            local_file = destination / (key + ".tar.gz")
            if not local_file.exists():
                cache_dir = local_file.parent
                cache_dir.mkdir(exist_ok=True, parents=True)
                print("Downloading", genetic_map.url)
                utils.download(genetic_map.url, local_file)
            # This assertion could fail if we update a file on AWS,
            # or a developer creates a new genetic map with the wrong checksum
            # (in the latter case, this should at least be caught during CI tests).
>           assert utils.sha256(local_file) == genetic_map.sha256, (
                f"SHA256 for {local_file} doesn't match the SHA256 for "
                f"{genetic_map.id}. If you didn't add this SHA256 yourself, "
                f"try deleting {local_file} and restarting the tests."
            )
E           AssertionError: SHA256 for _test_cache/tarballs/NaterPP_PonAbe2.tar.gz doesn't match the SHA256 for NaterPP_PonAbe2. If you didn't add this SHA256 yourself, try deleting _test_cache/tarballs/NaterPP_PonAbe2.tar.gz and restarting the tests.
E           assert '990992bfccd0...cabf40dc1fc01' == 'f4858b7efe15...d943ca075bed7'
E             - f4858b7efe15abe28b9367e7e9dc16614dc614df9326f9eedbcd943ca075bed7
E             + 990992bfccd07a9b02c7f583dbdedef07ce9f5938588566a5f0cabf40dc1fc01

tests/test_genetic_maps.py:41: AssertionError

...

=========================================================== short test summary info ============================================================
ERROR tests/test_genetic_maps.py::TestGeneticMap::test_str - AssertionError: SHA256 for _test_cache/tarballs/NaterPP_PonAbe2.tar.gz doesn't m...
ERROR tests/test_genetic_maps.py::TestGeneticMapDownload::test_correct_url - AssertionError: SHA256 for _test_cache/tarballs/NaterPP_PonAbe2....
ERROR tests/test_genetic_maps.py::TestGeneticMapDownload::test_download_over_cache - AssertionError: SHA256 for _test_cache/tarballs/NaterPP_...
ERROR tests/test_genetic_maps.py::TestGeneticMapDownload::test_multiple_threads_downloading - AssertionError: SHA256 for _test_cache/tarballs...
ERROR tests/test_genetic_maps.py::TestAllGeneticMaps::test_ids - AssertionError: SHA256 for _test_cache/tarballs/NaterPP_PonAbe2.tar.gz doesn...
ERROR tests/test_genetic_maps.py::TestAllGeneticMaps::test_non_empty - AssertionError: SHA256 for _test_cache/tarballs/NaterPP_PonAbe2.tar.gz...
ERROR tests/test_genetic_maps.py::TestAllGeneticMaps::test_types - AssertionError: SHA256 for _test_cache/tarballs/NaterPP_PonAbe2.tar.gz doe...
ERROR tests/test_genetic_maps.py::TestGetChromosomeMap::test_known_chromosome - AssertionError: SHA256 for _test_cache/tarballs/NaterPP_PonAb...
ERROR tests/test_genetic_maps.py::TestGetChromosomeMap::test_unknown_chromosome - AssertionError: SHA256 for _test_cache/tarballs/NaterPP_Pon...
ERROR tests/test_genetic_maps.py::TestGetChromosomeMap::test_warning_for_long_recomb_map - AssertionError: SHA256 for _test_cache/tarballs/Na...
ERROR tests/test_genetic_maps.py::TestGetChromosomeMap::test_warning_from_no_mapped_chromosome - AssertionError: SHA256 for _test_cache/tarba...
ERROR tests/test_genetic_maps.py::TestGeneticMapTarball::test_no_args - AssertionError: SHA256 for _test_cache/tarballs/NaterPP_PonAbe2.tar.g...
ERROR tests/test_genetic_maps.py::TestGeneticMap::test_cache_dirs - AssertionError: SHA256 for _test_cache/tarballs/NaterPP_PonAbe2.tar.gz do...
====================================== 434 passed, 1 skipped, 2 xfailed, 45 warnings, 13 errors in 23.08s ======================================
jeromekelleher commented 3 years ago

python setup.py install

I don't think we should recommend this, it's not necessary, right? Once run stuff from the root directory, you'll always have stdpopsim available.

grahamgower commented 3 years ago

python setup.py install

I don't think we should recommend this, it's not necessary, right? Once run stuff from the root directory, you'll always have stdpopsim available.

Yeah, you're right, that should be avoided for development workflows.

andrewkern commented 3 years ago

here's what i'm suggesting for the workshop 

cd stdpopsim
conda create -n stdpopsim python=3.9 --yes
conda activate stdpopsim
pip install -r requirements/development.txt
python setup.py install
andrewkern commented 3 years ago

draft materials going up overe here https://github.com/popsim-consortium/workshops/pull/25

jeromekelleher commented 3 years ago

Actually, yes, that'll probably work because we have the binary wheels on pip. Cool, except delete the python setup.py install

andrewkern commented 3 years ago

will folks be able to use the maintenance cli without running install?

andrewkern commented 3 years ago

(btw using the binary wheels in pip with a conda environment feels like cheating but it works so well!!)

jeromekelleher commented 3 years ago

will folks be able to use the maintenance cli without running install?

Yep; once you run everything from the project root directory it all works fine. Gets very messy otherwise.

apragsdale commented 3 years ago

Would you need to run setup.py install to be able to run the tests? Or what’s the benefit of not running python setup.py install?

On 10 Mar 2021, at 9:16, Andrew Kern @.***> wrote:

 will folks be able to use the maintenance cli without running install?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

jeromekelleher commented 3 years ago

There is none - you should never do it, essentially. If you really want to install the in-development version, you should use pip install ., but it's a Bad Idea.

tsackton commented 3 years ago

Trying to get things installed on MacOS Catalina, the procedure suggested above by @andrewkern fails because pip fails to build scikit-allel, specifically because it cannot find stdio.h:

    clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /Users/tim/Applications/miniconda3/envs/stdpopsim/include -fPIC -O2 -isystem /Users/tim/Applications/miniconda3/envs/stdpopsim/include -I/Users/tim/Applications/miniconda3/envs/stdpopsim/include/python3.9 -I/Users/tim/Applications/miniconda3/envs/stdpopsim/lib/python3.9/site-packages/numpy/core/include -c allel/opt/model.c -o build/temp.macosx-10.9-x86_64-3.9/allel/opt/model.o
    In file included from allel/opt/model.c:17:
    /Users/tim/Applications/miniconda3/envs/stdpopsim/include/python3.9/Python.h:25:10: fatal error: 'stdio.h' file not found
    #include <stdio.h>
             ^~~~~~~~~
    1 error generated.
    error: command '/usr/local/clang6/bin/clang' failed with exit code 1
    ----------------------------------------

Here are the solutions that worked for me.

Option 1: use Python 3.8. 

cd stdpopsim
conda create -n stdpopsim python=3.8 --yes
conda activate stdpopsim
pip install -r requirements/development.txt

I'm not sure why exactly this works, but it seems to get around the failure to build scikit-allel.

Option 2: symlink the missing headers from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/ to /usr/local/include

sudo ln -s /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/* /usr/local/include/

Then, proceed with the Python 3.9 installation instructions. Now the scikit-allel build will succeed.

jeromekelleher commented 3 years ago

We sitll don't have a conda package, but msprime 1.0 full release is due soon and the pypi packages are a good workaround.