search

fatiando / rockhound

NOTICE: This library is no longer being developed. Use Ensaio instead (https://www.fatiando.org/ensaio). -- Download geophysical models/datasets and load them in Python
BSD 3-Clause "New" or "Revised" License
34 stars 15 forks source link

ImportError : fetch_bedmap2 does not work with 1.0.25 of rasterio #53

Closed ChetGoerzen closed 2 years ago

ChetGoerzen commented 5 years ago

When running the test suite the test for bedmap2 fails with the current version of rasterio. The issue appears to be in line 102 of fetch_bedmap2.py. A call to xr.open_rasterio requires an import from rasterio._base import gdal_version which fails. This problem can be resolved by using version 1.0.24 of rasterio.

Full code that generated the error

def test_bedmap2_multiple_datasets():
    "Load multiple datasets"
    # Test with two datasets with same shape
    grid = fetch_bedmap2(["bed", "surface"])
    assert set(grid.data_vars) == set(["bed", "surface"])
    # Test with "bed" and "lakemask_vostok"
    grid = fetch_bedmap2(["bed", "lakemask_vostok"])
    assert set(grid.data_vars) == set(["bed", "lakemask_vostok"])
    # Test with "bed" and "thickness_uncertainty_5km"
    grid = fetch_bedmap2(["bed", "thickness_uncertainty_5km"])
    assert set(grid.data_vars) == set(["bed", "thickness_uncertainty_5km"])

Full error message

________________________ test_bedmap2_multiple_datasets ________________________

    def test_bedmap2_multiple_datasets():
        "Load multiple datasets"
        # Test with two datasets with same shape
>       grid = fetch_bedmap2(["bed", "surface"])

../rockhound/tests/test_bedmap2.py:37: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../rockhound/bedmap2.py:102: in fetch_bedmap2
    array = xr.open_rasterio(get_fname(dataset, fnames), chunks=chunks, **kwargs)
/home/chet/anaconda3/envs/rockhound/lib/python3.7/site-packages/xarray/backends/rasterio_.py:214: in open_rasterio
    import rasterio
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    """Rasterio"""

    from __future__ import absolute_import

    from collections import namedtuple
    from contextlib import contextmanager
    import logging

    try:
        from pathlib import Path
    except ImportError:  # pragma: no cover
        class Path:
            pass

    try:
        from logging import NullHandler
    except ImportError:  # pragma: no cover
        class NullHandler(logging.Handler):
            def emit(self, record):
                pass

>   from rasterio._base import gdal_version
E   ImportError: libtiledb.so.1.6.0: cannot open shared object file: No such file or directory

/home/chet/anaconda3/envs/rockhound/lib/python3.7/site-packages/rasterio/__init__.py:22: ImportError

System information

output of conda list 
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
affine                    2.2.2                      py_0    conda-forge
alabaster                 0.7.12                     py_0    conda-forge
appdirs                   1.4.3                      py_1    conda-forge
asn1crypto                0.24.0                py37_1003    conda-forge
astroid                   2.2.5                    py37_0    conda-forge
atomicwrites              1.3.0                      py_0    conda-forge
attrs                     19.1.0                     py_0    conda-forge
babel                     2.7.0                      py_0    conda-forge
black                     19.3b0                     py_0    conda-forge
bokeh                     1.3.4                    py37_0    conda-forge
boost-cpp                 1.70.0               h8e57a91_2    conda-forge
bzip2                     1.0.8                h516909a_0    conda-forge
ca-certificates           2019.6.16            hecc5488_0    conda-forge
cairo                     1.16.0            hfb77d84_1002    conda-forge
cartopy                   0.17.0          py37he1be148_1005    conda-forge
certifi                   2019.6.16                py37_1    conda-forge
cffi                      1.12.3           py37h8022711_0    conda-forge
cfitsio                   3.470                hb60a0a2_2    conda-forge
chardet                   3.0.4                 py37_1003    conda-forge
click                     7.0                        py_0    conda-forge
click-plugins             1.1.1                      py_0    conda-forge
cligj                     0.5.0                      py_0    conda-forge
cloudpickle               1.2.1                      py_0    conda-forge
cmocean                   2.0                        py_1    conda-forge
colorspacious             1.1.2              pyh24bf2e0_0    conda-forge
coverage                  4.5.4            py37h516909a_0    conda-forge
cryptography              2.7              py37h72c5cf5_0    conda-forge
curl                      7.65.3               hf8cf82a_0    conda-forge
cycler                    0.10.0                     py_1    conda-forge
cytoolz                   0.10.0           py37h516909a_0    conda-forge
dask                      2.3.0                      py_0    conda-forge
dask-core                 2.3.0                      py_0    conda-forge
dbus                      1.13.6               he372182_0    conda-forge
distributed               2.3.0                      py_0    conda-forge
docutils                  0.15.2                   py37_0    conda-forge
entrypoints               0.3                   py37_1000    conda-forge
expat                     2.2.5             he1b5a44_1003    conda-forge
flake8                    3.7.8                    py37_1    conda-forge
fontconfig                2.13.1            h86ecdb6_1001    conda-forge
freetype                  2.10.0               he983fc9_1    conda-forge
freexl                    1.0.5             h14c3975_1002    conda-forge
fsspec                    0.4.1                      py_0    conda-forge
geos                      3.7.2                he1b5a44_1    conda-forge
geotiff                   1.5.1                h560c3f3_2    conda-forge
gettext                   0.19.8.1          hc5be6a0_1002    conda-forge
giflib                    5.1.7                h516909a_1    conda-forge
glib                      2.58.3            h6f030ca_1002    conda-forge
gst-plugins-base          1.14.5               h0935bb2_0    conda-forge
gstreamer                 1.14.5               h36ae1b5_0    conda-forge
hdf4                      4.2.13            h9a582f1_1002    conda-forge
hdf5                      1.10.5          nompi_h3c11f04_1100    conda-forge
heapdict                  1.0.0                 py37_1000    conda-forge
icu                       64.2                 he1b5a44_0    conda-forge
idna                      2.8                   py37_1000    conda-forge
imagesize                 1.1.0                      py_0    conda-forge
importlib_metadata        0.18                     py37_0    conda-forge
isort                     4.3.21                   py37_0    conda-forge
jinja2                    2.10.1                     py_0    conda-forge
jpeg                      9c                h14c3975_1001    conda-forge
json-c                    0.13.1            h14c3975_1001    conda-forge
kealib                    1.4.10            h58c409b_1005    conda-forge
kiwisolver                1.1.0            py37hc9558a2_0    conda-forge
krb5                      1.16.3            h05b26f9_1001    conda-forge
lazy-object-proxy         1.4.1            py37h516909a_0    conda-forge
libblas                   3.8.0               12_openblas    conda-forge
libcblas                  3.8.0               12_openblas    conda-forge
libcurl                   7.65.3               hda55be3_0    conda-forge
libdap4                   3.20.2            hd48c02d_1000    conda-forge
libedit                   3.1.20170329      hf8c457e_1001    conda-forge
libffi                    3.2.1             he1b5a44_1006    conda-forge
libgcc-ng                 9.1.0                hdf63c60_0  
libgdal                   3.0.1                hbef8c27_6    conda-forge
libgfortran-ng            7.3.0                hdf63c60_0  
libiconv                  1.15              h516909a_1005    conda-forge
libkml                    1.3.0             h4fcabce_1010    conda-forge
liblapack                 3.8.0               12_openblas    conda-forge
libnetcdf                 4.6.2             h056eaf5_1002    conda-forge
libopenblas               0.3.7                h6e990d7_1    conda-forge
libpng                    1.6.37               hed695b0_0    conda-forge
libpq                     11.5                 hd9ab2ff_0    conda-forge
libspatialite             4.3.0a            he1bb1e1_1029    conda-forge
libssh2                   1.8.2                h22169c7_2    conda-forge
libstdcxx-ng              9.1.0                hdf63c60_0  
libtiff                   4.0.10            h57b8799_1003    conda-forge
libuuid                   2.32.1            h14c3975_1000    conda-forge
libxcb                    1.13              h14c3975_1002    conda-forge
libxml2                   2.9.9                hee79883_2    conda-forge
locket                    0.2.0                      py_2    conda-forge
lz4-c                     1.8.3             he1b5a44_1001    conda-forge
markupsafe                1.1.1            py37h14c3975_0    conda-forge
matplotlib                3.1.1                    py37_1    conda-forge
matplotlib-base           3.1.1            py37he7580a8_1    conda-forge
mccabe                    0.6.1                      py_1    conda-forge
more-itertools            7.2.0                      py_0    conda-forge
msgpack-python            0.6.1            py37h6bb024c_0    conda-forge
ncurses                   6.1               hf484d3e_1002    conda-forge
numpy                     1.17.0           py37h95a1406_0    conda-forge
numpydoc                  0.9.1                      py_0    conda-forge
olefile                   0.46                       py_0    conda-forge
openjpeg                  2.3.1                h58a6597_0    conda-forge
openssl                   1.1.1c               h516909a_0    conda-forge
owslib                    0.18.0                     py_0    conda-forge
packaging                 19.0                       py_0    conda-forge
pandas                    0.25.0           py37hb3f55d8_0    conda-forge
partd                     1.0.0                      py_0    conda-forge
pcre                      8.41              hf484d3e_1003    conda-forge
pillow                    6.1.0            py37h6b7be26_1    conda-forge
pip                       19.2.2                   py37_0    conda-forge
pixman                    0.38.0            h516909a_1003    conda-forge
pluggy                    0.12.0                     py_0    conda-forge
pooch                     0.5.2                    py37_0    conda-forge
poppler                   0.67.0               ha967d66_7    conda-forge
poppler-data              0.4.9                         1    conda-forge
postgresql                11.5                 hc63931a_0    conda-forge
proj4                     6.1.0                he751ad9_2    conda-forge
psutil                    5.6.3            py37h516909a_0    conda-forge
pthread-stubs             0.4               h14c3975_1001    conda-forge
py                        1.8.0                      py_0    conda-forge
pycodestyle               2.5.0                      py_0    conda-forge
pycparser                 2.19                     py37_1    conda-forge
pyepsg                    0.4.0                      py_0    conda-forge
pyflakes                  2.1.1                      py_0    conda-forge
pygments                  2.4.2                      py_0    conda-forge
pykdtree                  1.3.1           py37h3010b51_1002    conda-forge
pylint                    2.3.1                    py37_0    conda-forge
pyopenssl                 19.0.0                   py37_0    conda-forge
pyparsing                 2.4.2                      py_0    conda-forge
pyproj                    2.2.1            py37hc44880f_0    conda-forge
pyqt                      5.9.2            py37hcca6a23_2    conda-forge
pyshp                     2.1.0                      py_0    conda-forge
pysocks                   1.7.0                    py37_0    conda-forge
pytest                    5.1.0                    py37_0    conda-forge
pytest-cov                2.7.1                      py_0    conda-forge
python                    3.7.3                h33d41f4_1    conda-forge
python-dateutil           2.8.0                      py_0    conda-forge
pytz                      2019.2                     py_0    conda-forge
pyyaml                    5.1.2            py37h516909a_0    conda-forge
qt                        5.9.7                h0c104cb_3    conda-forge
rasterio                  1.0.25           py37h900e953_2    conda-forge
readline                  8.0                  hf8c457e_0    conda-forge
requests                  2.22.0                   py37_1    conda-forge
rockhound                 0.1.0+20.gddc8b2d           dev_0    
scipy                     1.3.1            py37h921218d_2    conda-forge
setuptools                41.0.1                   py37_0    conda-forge
shapely                   1.6.4           py37hec07ddf_1006    conda-forge
sip                       4.19.8          py37hf484d3e_1000    conda-forge
six                       1.12.0                py37_1000    conda-forge
snowballstemmer           1.9.0                      py_0    conda-forge
snuggs                    1.4.6                      py_0    conda-forge
sortedcontainers          2.1.0                      py_0    conda-forge
sphinx                    1.8.5                    py37_0    conda-forge
sphinx-gallery            0.4.0                    py37_0    conda-forge
sphinx_rtd_theme          0.4.3                      py_0    conda-forge
sphinxcontrib-websupport  1.1.2                      py_0    conda-forge
sqlite                    3.29.0               hcee41ef_0    conda-forge
tbb                       2018.0.5             h2d50403_0    conda-forge
tblib                     1.4.0                      py_0    conda-forge
tiledb                    1.6.2                h69c774e_0    conda-forge
tk                        8.6.9             hed695b0_1002    conda-forge
toml                      0.10.0                     py_0    conda-forge
toolz                     0.10.0                     py_0    conda-forge
tornado                   6.0.3            py37h516909a_0    conda-forge
tzcode                    2019a             h516909a_1002    conda-forge
urllib3                   1.25.3                   py37_0    conda-forge
viscm                     0.7                pyh24bf2e0_0    conda-forge
wcwidth                   0.1.7                      py_1    conda-forge
wheel                     0.33.4                   py37_0    conda-forge
wrapt                     1.11.2           py37h516909a_0    conda-forge
xarray                    0.12.3                     py_0    conda-forge
xerces-c                  3.2.2             h8412b87_1004    conda-forge
xorg-kbproto              1.0.7             h14c3975_1002    conda-forge
xorg-libice               1.0.10               h516909a_0    conda-forge
xorg-libsm                1.2.3             h84519dc_1000    conda-forge
xorg-libx11               1.6.8                h516909a_0    conda-forge
xorg-libxau               1.0.9                h14c3975_0    conda-forge
xorg-libxdmcp             1.1.3                h516909a_0    conda-forge
xorg-libxext              1.3.4                h516909a_0    conda-forge
xorg-libxrender           0.9.10            h516909a_1002    conda-forge
xorg-renderproto          0.11.1            h14c3975_1002    conda-forge
xorg-xextproto            7.3.0             h14c3975_1002    conda-forge
xorg-xproto               7.0.31            h14c3975_1007    conda-forge
xz                        5.2.4             h14c3975_1001    conda-forge
yaml                      0.1.7             h14c3975_1001    conda-forge
zict                      1.0.0                      py_0    conda-forge
zipp                      0.5.2                      py_0    conda-forge
zlib                      1.2.11            h516909a_1005    conda-forge
zstd                      1.4.0                h3b9ef0a_0    conda-forge
welcome[bot] commented 5 years ago

👋 Thanks for opening your first issue here! Please make sure you filled out the template with as much detail as possible.

You might also want to take a look at our Contributing Guide and Code of Conduct.

santisoler commented 5 years ago

Thanks @ChetGoerzen for opening this issue.

This error is raised when xarray tries to import gdal_version from rasterio._base, so I think we should open an issue on xarray repository reporting this. Meanwhile would be better to pin the rasterio version on Rockhound environment.yml and requirements.txt files. Would you like to open a PR with these changes?

CC @leouieda

ChetGoerzen commented 5 years ago

Hi @santisoler,

I've pinned the rasterio version to 1.0.24 in requirements.txt and the environment.yml file in the Fetch Models WIP (#52 ) pull request. However, the build pipeline fails for Mac Python 37 and Mac Python 36. The failure occurs in the Setup Miniconda phase of the build. The error appears to be with a file called /Users/vsts/.conda/pkgs/urls.txt. According to Run pipelines with Anaconda environments, on Hosted macOS the agent user doesn't have ownership of Miniconda's installation directory. I suspect that this is why the Mac builds fail while the Windows builds pass. Are administrator privileges required to update the packages installed by conda in the build?

Error message
Creating the 'testing' environment with python=3.7 ======================================================================== Collecting package metadata (current_repodata.json): ...working... failed \# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<< Traceback (most recent call last): File "/usr/local/miniconda/lib/python3.7/site-packages/conda/gateways/disk/update.py", line 107, in touch mkdir_p_sudo_safe(dirpath) File "/usr/local/miniconda/lib/python3.7/site-packages/conda/gateways/disk/__init__.py", line 84, in mkdir_p_sudo_safe os.mkdir(path) FileExistsError: [Errno 17] File exists: '/Users/vsts/.conda/pkgs' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/miniconda/lib/python3.7/site-packages/conda/core/subdir_data.py", line 185, in _load mtime = getmtime(self.cache_path_json) File "/usr/local/miniconda/lib/python3.7/site-packages/conda/core/subdir_data.py", line 153, in cache_path_json return self.cache_path_base + '.json' File "/usr/local/miniconda/lib/python3.7/site-packages/conda/core/subdir_data.py", line 144, in cache_path_base create_cache_dir(), File "/usr/local/miniconda/lib/python3.7/site-packages/conda/core/subdir_data.py", line 645, in create_cache_dir cache_dir = join(PackageCacheData.first_writable(context.pkgs_dirs).pkgs_dir, 'cache') File "/usr/local/miniconda/lib/python3.7/site-packages/conda/core/package_cache_data.py", line 162, in first_writable created = create_package_cache_directory(package_cache.pkgs_dir) File "/usr/local/miniconda/lib/python3.7/site-packages/conda/gateways/disk/create.py", line 435, in create_package_cache_directory touch(join(pkgs_dir, PACKAGE_CACHE_MAGIC_FILE), mkdir=True, sudo_safe=sudo_safe) File "/usr/local/miniconda/lib/python3.7/site-packages/conda/gateways/disk/update.py", line 125, in touch raise NotWritableError(path, e.errno, caused_by=e) conda.exceptions.NotWritableError: The current user does not have write permissions to a required path. path: /Users/vsts/.conda/pkgs/urls.txt uid: 501 gid: 20 If you feel that permissions on this path are set incorrectly, you can manually change them by executing $ sudo chown 501:20 /Users/vsts/.conda/pkgs/urls.txt In general, it's not advisable to use 'sudo conda'. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/miniconda/lib/python3.7/site-packages/conda/exceptions.py", line 1062, in __call__ return func(*args, **kwargs) File "/usr/local/miniconda/lib/python3.7/site-packages/conda/cli/main.py", line 84, in _main exit_code = do_call(args, p) File "/usr/local/miniconda/lib/python3.7/site-packages/conda/cli/conda_argparse.py", line 82, in do_call exit_code = getattr(module, func_name)(args, parser) File "/usr/local/miniconda/lib/python3.7/site-packages/conda/cli/main_create.py", line 37, in execute install(args, parser, 'create') File "/usr/local/miniconda/lib/python3.7/site-packages/conda/cli/install.py", line 256, in install force_reinstall=context.force_reinstall or context.force, File "/usr/local/miniconda/lib/python3.7/site-packages/conda/core/solve.py", line 112, in solve_for_transaction force_remove, force_reinstall) File "/usr/local/miniconda/lib/python3.7/site-packages/conda/core/solve.py", line 150, in solve_for_diff force_remove) File "/usr/local/miniconda/lib/python3.7/site-packages/conda/core/solve.py", line 249, in solve_final_state ssc = self._collect_all_metadata(ssc) File "/usr/local/miniconda/lib/python3.7/site-packages/conda/common/io.py", line 88, in decorated return f(*args, **kwds) File "/usr/local/miniconda/lib/python3.7/site-packages/conda/core/solve.py", line 389, in _collect_all_metadata index, r = self._prepare(prepared_specs) File "/usr/local/miniconda/lib/python3.7/site-packages/conda/core/solve.py", line 974, in _prepare self.subdirs, prepared_specs, self._repodata_fn) File "/usr/local/miniconda/lib/python3.7/site-packages/conda/core/index.py", line 214, in get_reduced_index repodata_fn=repodata_fn) File "/usr/local/miniconda/lib/python3.7/site-packages/conda/core/subdir_data.py", line 91, in query_all result = tuple(concat(executor.map(subdir_query, channel_urls))) File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 586, in result_iterator yield fs.pop().result() File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 425, in result return self.__get_result() File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result raise self._exception File "/usr/local/miniconda/lib/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/usr/local/miniconda/lib/python3.7/site-packages/conda/core/subdir_data.py", line 87, in package_ref_or_match_spec)) File "/usr/local/miniconda/lib/python3.7/site-packages/conda/core/subdir_data.py", line 96, in query self.load() File "/usr/local/miniconda/lib/python3.7/site-packages/conda/core/subdir_data.py", line 160, in load _internal_state = self._load() File "/usr/local/miniconda/lib/python3.7/site-packages/conda/core/subdir_data.py", line 188, in _load self.cache_path_json) File "/usr/local/miniconda/lib/python3.7/site-packages/conda/core/subdir_data.py", line 153, in cache_path_json return self.cache_path_base + '.json' File "/usr/local/miniconda/lib/python3.7/site-packages/conda/core/subdir_data.py", line 144, in cache_path_base create_cache_dir(), File "/usr/local/miniconda/lib/python3.7/site-packages/conda/core/subdir_data.py", line 646, in create_cache_dir mkdir_p_sudo_safe(cache_dir) File "/usr/local/miniconda/lib/python3.7/site-packages/conda/gateways/disk/__init__.py", line 84, in mkdir_p_sudo_safe os.mkdir(path) FileExistsError: [Errno 17] File exists: '/Users/vsts/.conda/pkgs/cache'

CC @leouieda

santisoler commented 5 years ago

Hi @ChetGoerzen! I got a response on https://github.com/pydata/xarray/issues/3230. Seems the problem was somehow related with the conda installation of rasterio. I tried to install rasterio 1.0.25 through pip and had no problem. So, I tried to reinstall rasterio with conda from scratch, and had no problem either.

So, please try the following:

conda activate rockhound
conda uninstall rasterio
conda install rasterio

Then try:

python -c "import rasterio"

If you don't get any error, then it's very likely you can run fetch_bedmap2 without any problem. If this solves the issue, then we don't need to pin rasterio to 1.0.24.

santisoler commented 5 years ago

Update

I rerun Travis for the last commit on master and it fails when importing rasterio, so I think there's still a problem with the conda installation of rasterio.

santisoler commented 5 years ago

Regarding the pipelines problem, I don't think it is intimately related with this issue, so maybe we could discuss it on #52 or even open a new issue for it. Nevertheless, the builds install the packages on every run from either creating a conda environment from the current environment.yml file or through pip, getting the dependencies from requirements.txt. So you don't need administration privileges.

santisoler commented 5 years ago

@leouieda Could you throw some light on this when you have some time?