search

astropy / pytest-remotedata

Pytest plugin to control whether tests are run that have remote data
BSD 3-Clause "New" or "Revised" License
23 stars 15 forks source link

Issue installing astropy via conda #23

Closed lbdreyer closed 6 years ago

lbdreyer commented 6 years ago

I tried installing astropy with miniconda and noticed there was an issue with installing pytest-remotedata and pytest-openfiles.

The anaconda recipe for astropy 3.0.1 sets pytest-astropy as a runtime dependency, and pytest-astropy sets pytest-remotedata and pytest-openfiles as runtime dependencies. This means that when I try to install astropy into a conda environment, pytest-remotedata and pytest-openfiles (and pytest-astropy) also get installed, but there is an issue with installing both into a single environment due to their directory structures.

Both pytest-remotedata and pytest-openfiles have a top level tests directory which get installed into the lib/python3.6/site-packages/tests directory of the environment. So whichever package get installed second is trying overwrite whichever gets installed first. and this is causing problems.

Is pytest-astropy a runtime dependency of astropy? Or should the anaconda recipe for astropy be updated?

Below is the output I get when I try to install astropy with conda. See the ClobberError at the end for the error relating to pytest-remotedata and pytest-openfiles

$ conda create -n test_astropy_pytest astropy
Solving environment: done

## Package Plan ##

  environment location: .../miniconda3/envs/test_astropy_pytest

  added / updated specs: 
    - astropy

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    psutil-5.4.5               |   py36h14c3975_0         303 KB  defaults
    pluggy-0.6.0               |   py36hb689045_0          23 KB  defaults
    mkl-2018.0.2               |                1       205.2 MB  defaults
    pytest-openfiles-0.2.0     |           py36_0           9 KB  defaults
    pytest-remotedata-0.2.0    |           py36_0          16 KB  defaults
    attrs-17.4.0               |           py36_0          41 KB  defaults
    pytest-arraydiff-0.2       |           py36_0          14 KB  defaults
    libgfortran-ng-7.2.0       |       hdf63c60_3         1.2 MB  defaults
    intel-openmp-2018.0.0      |                8         620 KB  defaults
    setuptools-39.0.1          |           py36_0         549 KB  defaults
    astropy-3.0.1              |   py36h3010b51_1         6.8 MB  defaults
    pytest-3.5.0               |           py36_0         293 KB  defaults
    numpy-1.14.2               |   py36hdbf6ddf_1         4.1 MB  defaults
    ------------------------------------------------------------
                                           Total:       219.2 MB

The following NEW packages will be INSTALLED:

    astropy:            3.0.1-py36h3010b51_1  defaults
    attrs:              17.4.0-py36_0         defaults
    ca-certificates:    2018.03.07-0          defaults
    certifi:            2018.4.16-py36_0      defaults
    intel-openmp:       2018.0.0-8            defaults
    libedit:            3.1-heed3624_0        defaults
    libffi:             3.2.1-hd88cf55_4      defaults
    libgcc-ng:          7.2.0-hdf63c60_3      defaults
    libgfortran-ng:     7.2.0-hdf63c60_3      defaults
    libstdcxx-ng:       7.2.0-hdf63c60_3      defaults
    mkl:                2018.0.2-1            defaults
    mkl_fft:            1.0.1-py36h3010b51_0  defaults
    mkl_random:         1.0.1-py36h629b387_0  defaults
    more-itertools:     4.1.0-py36_0          defaults
    ncurses:            6.0-h9df7e31_2        defaults
    numpy:              1.14.2-py36hdbf6ddf_1 defaults
    openssl:            1.0.2o-h20670df_0     defaults
    pluggy:             0.6.0-py36hb689045_0  defaults
    psutil:             5.4.5-py36h14c3975_0  defaults
    py:                 1.5.3-py36_0          defaults
    pytest:             3.5.0-py36_0          defaults
    pytest-arraydiff:   0.2-py36_0            defaults
    pytest-astropy:     0.2.1-py36_0          defaults
    pytest-doctestplus: 0.1.2-py36_0          defaults
    pytest-openfiles:   0.2.0-py36_0          defaults
    pytest-remotedata:  0.2.0-py36_0          defaults
    python:             3.6.5-hc3d631a_0      defaults
    readline:           7.0-ha6073c6_4        defaults
    setuptools:         39.0.1-py36_0         defaults
    six:                1.11.0-py36h372c433_1 defaults
    sqlite:             3.23.1-he433501_0     defaults
    tk:                 8.6.7-hc745277_3      defaults
    xz:                 5.2.3-h55aa19d_2      defaults
    zlib:               1.2.11-ha838bed_2     defaults

Proceed ([y]/n)? y

Downloading and Extracting Packages
psutil 5.4.5########################################################################################################################################## | 100% 
pluggy 0.6.0########################################################################################################################################## | 100% 
mkl 2018.0.2########################################################################################################################################## | 100% 
pytest-openfiles 0.2.0################################################################################################################################ | 100% 
pytest-remotedata 0.2.0############################################################################################################################### | 100% 
attrs 17.4.0########################################################################################################################################## | 100% 
pytest-arraydiff 0.2################################################################################################################################## | 100% 
libgfortran-ng 7.2.0################################################################################################################################## | 100% 
intel-openmp 2018.0.0################################################################################################################################# | 100% 
setuptools 39.0.1##################################################################################################################################### | 100% 
astropy 3.0.1######################################################################################################################################### | 100% 
pytest 3.5.0########################################################################################################################################## | 100% 
numpy 1.14.2########################################################################################################################################## | 100% 
Preparing transaction: done
Verifying transaction: failed

CondaVerificationError: The package for py located at .../miniconda3/pkgs/py-1.5.3-py36_0
appears to be corrupted. The path 'lib/python3.6/site-packages/py/_code/__pycache__/_assertionold.cpython-36.pyc'
specified in the package manifest cannot be found.

CondaVerificationError: The package for py located at .../miniconda3/pkgs/py-1.5.3-py36_0
appears to be corrupted. The path 'lib/python3.6/site-packages/py/_code/__pycache__/code.cpython-36.pyc'
specified in the package manifest cannot be found.

CondaVerificationError: The package for py located at .../miniconda3/pkgs/py-1.5.3-py36_0
appears to be corrupted. The path 'lib/python3.6/site-packages/py/_code/__pycache__/source.cpython-36.pyc'
specified in the package manifest cannot be found.

CondaVerificationError: The package for py located at .../miniconda3/pkgs/py-1.5.3-py36_0
appears to be corrupted. The path 'lib/python3.6/site-packages/py/_code/_assertionold.py'
specified in the package manifest cannot be found.

CondaVerificationError: The package for py located at .../miniconda3/pkgs/py-1.5.3-py36_0
appears to be corrupted. The path 'lib/python3.6/site-packages/py/_code/code.py'
specified in the package manifest cannot be found.

CondaVerificationError: The package for py located at .../miniconda3/pkgs/py-1.5.3-py36_0
appears to be corrupted. The path 'lib/python3.6/site-packages/py/_code/source.py'
specified in the package manifest cannot be found.

CondaVerificationError: The package for py located at .../miniconda3/pkgs/py-1.5.3-py36_0
appears to be corrupted. The path 'lib/python3.6/site-packages/py/_io/capture.py'
specified in the package manifest cannot be found.

CondaVerificationError: The package for py located at .../miniconda3/pkgs/py-1.5.3-py36_0
appears to be corrupted. The path 'lib/python3.6/site-packages/py/_io/terminalwriter.py'
specified in the package manifest cannot be found.

CondaVerificationError: The package for py located at .../miniconda3/pkgs/py-1.5.3-py36_0
appears to be corrupted. The path 'lib/python3.6/site-packages/py/_path/__pycache__/common.cpython-36.pyc'
specified in the package manifest cannot be found.

CondaVerificationError: The package for py located at .../miniconda3/pkgs/py-1.5.3-py36_0
appears to be corrupted. The path 'lib/python3.6/site-packages/py/_path/__pycache__/local.cpython-36.pyc'
specified in the package manifest cannot be found.

CondaVerificationError: The package for py located at .../miniconda3/pkgs/py-1.5.3-py36_0
appears to be corrupted. The path 'lib/python3.6/site-packages/py/_path/__pycache__/svnurl.cpython-36.pyc'
specified in the package manifest cannot be found.

CondaVerificationError: The package for py located at .../miniconda3/pkgs/py-1.5.3-py36_0
appears to be corrupted. The path 'lib/python3.6/site-packages/py/_path/__pycache__/svnwc.cpython-36.pyc'
specified in the package manifest cannot be found.

CondaVerificationError: The package for py located at .../miniconda3/pkgs/py-1.5.3-py36_0
appears to be corrupted. The path 'lib/python3.6/site-packages/py/_path/common.py'
specified in the package manifest cannot be found.

CondaVerificationError: The package for py located at .../miniconda3/pkgs/py-1.5.3-py36_0
appears to be corrupted. The path 'lib/python3.6/site-packages/py/_path/local.py'
specified in the package manifest cannot be found.

CondaVerificationError: The package for py located at .../miniconda3/pkgs/py-1.5.3-py36_0
appears to be corrupted. The path 'lib/python3.6/site-packages/py/_path/svnurl.py'
specified in the package manifest cannot be found.

CondaVerificationError: The package for py located at .../miniconda3/pkgs/py-1.5.3-py36_0
appears to be corrupted. The path 'lib/python3.6/site-packages/py/_path/svnwc.py'
specified in the package manifest cannot be found.

CondaVerificationError: The package for py located at .../miniconda3/pkgs/py-1.5.3-py36_0
appears to be corrupted. The path 'lib/python3.6/site-packages/py/_vendored_packages/__init__.py'
specified in the package manifest cannot be found.

ClobberError: This transaction has incompatible packages due to a shared path.
  packages: defaults::pytest-openfiles-0.2.0-py36_0, defaults::pytest-remotedata-0.2.0-py36_0
  path: 'lib/python3.6/site-packages/tests/__init__.py'

ClobberError: This transaction has incompatible packages due to a shared path.
  packages: defaults::pytest-openfiles-0.2.0-py36_0, defaults::pytest-remotedata-0.2.0-py36_0
  path: 'lib/python3.6/site-packages/tests/__pycache__/__init__.cpython-36.pyc'
kalefranz commented 6 years ago 
CondaVerificationError: The package for py located at .../miniconda3/pkgs/py-1.5.3-py36_0
appears to be corrupted. The path 'lib/python3.6/site-packages/py/_code/__pycache__/_assertionold.cpython-36.pyc'
specified in the package manifest cannot be found.

Means you have a corrupt package in your package cache. Use conda clean --packages, and if that doesn't work, just remove it manually.

ClobberError: This transaction has incompatible packages due to a shared path.
  packages: defaults::pytest-openfiles-0.2.0-py36_0, defaults::pytest-remotedata-0.2.0-py36_0
  path: 'lib/python3.6/site-packages/tests/__init__.py'

Means that conda is alerting you to packages that have overlapping paths, that will very likely become problematic at some later point. The behavior of conda here is configured via the path_conflict configuration parameter. The parameter can have values of clobber, warn, or prevent. Currently, because we know we have several problems in defaults channels, the default value is clobber. So I assume conda config --show-sources for this user will show that the configuration parameter has been changed to warn or prevent.

bsipocz commented 6 years ago

@kalefranz - Thank you very much for the explanation.

@lbdreyer - Could you please check the suggestions above and whether that solves your issue? We haven't run into it though extensively run tests in conda environments for several packages that depend on astropy.

drdavella commented 6 years ago

@lbdreyer, @kalefranz thanks for the report. This is indeed a packaging problem with those repositories. I'll fix it and will try to get a release going in the near future.

drdavella commented 6 years ago

This should be fixed for conda users once https://github.com/conda-forge/pytest-astropy-feedstock/pull/1 is merged.

bsipocz commented 6 years ago

(note: should be fixed for conda-forge, but I don't think it's propagated to the default channels before we have the new astropy bugfix release out next week)