search

fatiando / verde

Processing and gridding spatial data, machine-learning style
https://www.fatiando.org/verde
BSD 3-Clause "New" or "Revised" License
602 stars 72 forks source link

make_xarray_grid fails after passing DataArrays #330

Closed santisoler closed 3 years ago

santisoler commented 3 years ago

Description of the problem

When passing xarray objects as arguments to vd.make_xarray_grid we end up on some xarray errors down the line.

Full code that generated the error

import numpy as np
import xarray as xr
import verde as vd

x = np.linspace(0, 10, 11)
y = np.linspace(-4, 4, 8)
xx, yy = np.meshgrid(x, y)
data = xx + yy

dims = ("y", "x")
coords = {"x": x, "y": y}
da = xr.DataArray(data, coords=coords, dims=dims)

vd.make_xarray_grid(coordinates=(da.x, da.y), data_names="data", data=da)

Full error message

Traceback (most recent call last):
  File "/home/santi/verde_bug.py", line 15, in <module>
    vd.make_xarray_grid(coordinates=(da.x, da.y), data_names="data", data=da)
  File "/home/santi/.miniforge3/envs/default/lib/python3.9/site-packages/verde/utils.py", line 370, in make_xarray_grid
    return xr.Dataset(data_vars, coords)
  File "/home/santi/.miniforge3/envs/default/lib/python3.9/site-packages/xarray/core/dataset.py", line 754, in __init__
    variables, coord_names, dims, indexes, _ = merge_data_and_coords(
  File "/home/santi/.miniforge3/envs/default/lib/python3.9/site-packages/xarray/core/merge.py", line 484, in merge_data_and_coords
    return merge_core(
  File "/home/santi/.miniforge3/envs/default/lib/python3.9/site-packages/xarray/core/merge.py", line 632, in merge_core
    collected = collect_variables_and_indexes(aligned)
  File "/home/santi/.miniforge3/envs/default/lib/python3.9/site-packages/xarray/core/merge.py", line 294, in collect_variables_and_indexes
    variable = as_variable(variable, name=name)
  File "/home/santi/.miniforge3/envs/default/lib/python3.9/site-packages/xarray/core/variable.py", line 121, in as_variable
    raise TypeError(
TypeError: Using a DataArray object to construct a variable is ambiguous, please extract the data using the .data property.

System information

output of conda list 

# packages in environment at /home/santi/.miniforge3/envs/default:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       1_gnu    conda-forge
affine                    2.3.0                      py_0    conda-forge
aiohttp                   3.7.4            py39h3811e60_0    conda-forge
alsa-lib                  1.2.3                h516909a_0    conda-forge
anyio                     3.0.1            py39hf3d152e_0    conda-forge
appdirs                   1.4.4              pyh9f0ad1d_0    conda-forge
argon2-cffi               20.1.0           py39h3811e60_2    conda-forge
async-timeout             3.0.1                   py_1000    conda-forge
async_generator           1.10                       py_0    conda-forge
attrs                     21.2.0             pyhd8ed1ab_0    conda-forge
babel                     2.9.1              pyh44b312d_0    conda-forge
babelfish                 0.5.5                    pypi_0    pypi
backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
backports                 1.0                        py_2    conda-forge
backports.functools_lru_cache 1.6.4              pyhd8ed1ab_0    conda-forge
beautifulsoup4            4.9.3                    pypi_0    pypi
black                     21.7b0             pyhd8ed1ab_0    conda-forge
bleach                    3.3.0              pyh44b312d_0    conda-forge
bokeh                     2.3.2            py39hf3d152e_0    conda-forge
boost-cpp                 1.74.0               hc6e9bd1_3    conda-forge
boule                     0.2.0                      py_0    conda-forge
brotlipy                  0.7.0           py39h3811e60_1001    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.17.1               h7f98852_1    conda-forge
ca-certificates           2021.5.30            ha878542_0    conda-forge
cairo                     1.16.0            h6cf1ce9_1008    conda-forge
cartopy                   0.19.0.post1     py39h3b23250_0    conda-forge
certifi                   2021.5.30        py39hf3d152e_0    conda-forge
cffi                      1.14.5           py39he32792d_0    conda-forge
cfitsio                   3.470                hb418390_7    conda-forge
cftime                    1.4.1            py39hce5d2b2_0    conda-forge
chardet                   4.0.0            py39hf3d152e_1    conda-forge
click                     7.1.2              pyh9f0ad1d_0    conda-forge
click-plugins             1.1.1                      py_0    conda-forge
cligj                     0.7.1              pyhd8ed1ab_0    conda-forge
cloudpickle               1.6.0                      py_0    conda-forge
colorama                  0.4.4              pyh9f0ad1d_0    conda-forge
cryptography              3.4.7            py39hbca0aa6_0    conda-forge
curl                      7.76.1               hea6ffbf_2    conda-forge
cycler                    0.10.0                     py_2    conda-forge
cytoolz                   0.11.0           py39h3811e60_3    conda-forge
dask                      2021.5.0           pyhd8ed1ab_0    conda-forge
dask-core                 2021.5.0           pyhd8ed1ab_0    conda-forge
dask-labextension         5.1.0              pyhd8ed1ab_0    conda-forge
dataclasses               0.8                pyhc8e2a94_1    conda-forge
dbus                      1.13.6               h48d8840_2    conda-forge
decorator                 5.0.9              pyhd8ed1ab_0    conda-forge
defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
distributed               2021.5.0         py39hf3d152e_0    conda-forge
docrep                    0.3.2              pyh44b312d_0    conda-forge
entrypoints               0.3             pyhd8ed1ab_1003    conda-forge
expat                     2.3.0                h9c3ff4c_0    conda-forge
flake8                    3.9.2              pyhd8ed1ab_0    conda-forge
fontconfig                2.13.1            hba837de_1005    conda-forge
freetype                  2.10.4               h0708190_1    conda-forge
freexl                    1.0.6                h7f98852_0    conda-forge
fsspec                    2021.5.0           pyhd8ed1ab_0    conda-forge
future                    0.18.2           py39hf3d152e_3    conda-forge
geos                      3.9.1                h9c3ff4c_2    conda-forge
geotiff                   1.6.0                h2b14fbe_4    conda-forge
gettext                   0.19.8.1          h0b5b191_1005    conda-forge
giflib                    5.2.1                h36c2ea0_2    conda-forge
gitdb                     4.0.7              pyhd8ed1ab_0    conda-forge
gitpython                 3.1.17             pyhd8ed1ab_0    conda-forge
glib                      2.68.2               h9c3ff4c_0    conda-forge
glib-tools                2.68.2               h9c3ff4c_0    conda-forge
gmp                       6.2.1                h58526e2_0    conda-forge
gmpy2                     2.1.0b1          py39hf9ed83c_1    conda-forge
gst-plugins-base          1.18.4               hf529b03_2    conda-forge
gstreamer                 1.18.4               h76c114f_2    conda-forge
guessit                   3.3.1                    pypi_0    pypi
harmonica                 0.2.1              pyhd8ed1ab_0    conda-forge
hdf4                      4.2.15               h10796ff_2    conda-forge
hdf5                      1.10.6          nompi_h6a2412b_1114    conda-forge
heapdict                  1.0.1                      py_0    conda-forge
icu                       68.1                 h58526e2_0    conda-forge
idna                      2.10               pyh9f0ad1d_0    conda-forge
importlib-metadata        4.0.1            py39hf3d152e_0    conda-forge
ipykernel                 5.5.5            py39hef51801_0    conda-forge
ipython                   7.27.0           py39hef51801_0    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
ipywidgets                7.6.3              pyhd3deb0d_0    conda-forge
jedi                      0.18.0           py39hf3d152e_2    conda-forge
jinja2                    3.0.0              pyhd8ed1ab_0    conda-forge
joblib                    1.0.1              pyhd8ed1ab_0    conda-forge
jpeg                      9d                   h36c2ea0_0    conda-forge
json-c                    0.15                 h98cffda_0    conda-forge
json5                     0.9.5              pyh9f0ad1d_0    conda-forge
jsonschema                3.2.0              pyhd8ed1ab_3    conda-forge
jupyter                   1.0.0            py39hf3d152e_6    conda-forge
jupyter-server-mathjax    0.2.2              pyhd8ed1ab_0    conda-forge
jupyter-server-proxy      3.0.2              pyhd8ed1ab_0    conda-forge
jupyter_client            6.1.12             pyhd8ed1ab_0    conda-forge
jupyter_console           6.4.0              pyhd8ed1ab_0    conda-forge
jupyter_core              4.7.1            py39hf3d152e_0    conda-forge
jupyter_server            1.7.0            py39hf3d152e_1    conda-forge
jupyterlab                3.1.9              pyhd8ed1ab_0    conda-forge
jupyterlab_pygments       0.1.2              pyh9f0ad1d_0    conda-forge
jupyterlab_server         2.5.2              pyhd8ed1ab_0    conda-forge
jupyterlab_widgets        1.0.0              pyhd8ed1ab_1    conda-forge
jupytext                  1.11.4             pyhd0ecf6b_0    conda-forge
kealib                    1.4.14               hcc255d8_2    conda-forge
kiwisolver                1.3.1            py39h1a9c180_1    conda-forge
krb5                      1.19.1               hcc1bbae_0    conda-forge
lcms2                     2.12                 hddcbb42_0    conda-forge
ld_impl_linux-64          2.35.1               hea4e1c9_2    conda-forge
libblas                   3.9.0                9_openblas    conda-forge
libcblas                  3.9.0                9_openblas    conda-forge
libclang                  11.1.0          default_ha53f305_1    conda-forge
libcurl                   7.76.1               h2574ce0_2    conda-forge
libdap4                   3.20.6               hd7c4107_2    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libevent                  2.1.10               hcdb4288_3    conda-forge
libffi                    3.3                  h58526e2_2    conda-forge
libgcc-ng                 9.3.0               h2828fa1_19    conda-forge
libgdal                   3.2.1                h38ff51b_7    conda-forge
libgfortran-ng            9.3.0               hff62375_19    conda-forge
libgfortran5              9.3.0               hff62375_19    conda-forge
libglib                   2.68.2               h3e27bee_0    conda-forge
libgomp                   9.3.0               h2828fa1_19    conda-forge
libiconv                  1.16                 h516909a_0    conda-forge
libkml                    1.3.0             h238a007_1013    conda-forge
liblapack                 3.9.0                9_openblas    conda-forge
libllvm10                 10.0.1               he513fc3_3    conda-forge
libllvm11                 11.1.0               hf817b99_2    conda-forge
libnetcdf                 4.7.4           nompi_h56d31a8_107    conda-forge
libnghttp2                1.43.0               h812cca2_0    conda-forge
libogg                    1.3.4                h7f98852_1    conda-forge
libopenblas               0.3.15          pthreads_h8fe5266_1    conda-forge
libopus                   1.3.1                h7f98852_1    conda-forge
libpng                    1.6.37               h21135ba_2    conda-forge
libpq                     13.3                 hd57d9b9_0    conda-forge
librttopo                 1.1.0                h1185371_6    conda-forge
libsodium                 1.0.18               h36c2ea0_1    conda-forge
libspatialite             5.0.1                he52d314_3    conda-forge
libssh2                   1.9.0                ha56f1ee_6    conda-forge
libstdcxx-ng              9.3.0               h6de172a_19    conda-forge
libtiff                   4.2.0                hbd63e13_2    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libuv                     1.41.0               h7f98852_0    conda-forge
libvorbis                 1.3.7                h9c3ff4c_0    conda-forge
libwebp-base              1.2.0                h7f98852_2    conda-forge
libxcb                    1.13              h7f98852_1003    conda-forge
libxkbcommon              1.0.3                he3ba5ed_0    conda-forge
libxml2                   2.9.12               h72842e0_0    conda-forge
livereload                2.6.3              pyh9f0ad1d_0    conda-forge
llvmlite                  0.36.0           py39h1bbdace_0    conda-forge
locket                    0.2.0                      py_2    conda-forge
lz4-c                     1.9.3                h9c3ff4c_0    conda-forge
markdown-it-py            1.1.0              pyhd8ed1ab_0    conda-forge
markupsafe                2.0.0            py39h3811e60_0    conda-forge
matplotlib                3.4.2            py39hf3d152e_0    conda-forge
matplotlib-base           3.4.2            py39h2fa2bec_0    conda-forge
matplotlib-inline         0.1.2              pyhd8ed1ab_2    conda-forge
mccabe                    0.6.1                      py_1    conda-forge
mdit-py-plugins           0.2.8              pyhd8ed1ab_0    conda-forge
mistune                   0.8.4           py39h3811e60_1003    conda-forge
mpc                       1.1.0             h04dde30_1009    conda-forge
mpfr                      4.0.2                he80fd80_1    conda-forge
mpmath                    1.2.1              pyhd8ed1ab_0    conda-forge
msgpack-python            1.0.2            py39h1a9c180_1    conda-forge
multidict                 5.1.0            py39h3811e60_1    conda-forge
mypy_extensions           0.4.3            py39hf3d152e_3    conda-forge
mysql-common              8.0.23               ha770c72_2    conda-forge
mysql-libs                8.0.23               h935591d_2    conda-forge
nb_conda                  2.2.1            py39hf3d152e_4    conda-forge
nb_conda_kernels          2.3.1            py39hf3d152e_0    conda-forge
nbclassic                 0.2.8              pyhd8ed1ab_0    conda-forge
nbclient                  0.5.3              pyhd8ed1ab_0    conda-forge
nbconvert                 6.0.7            py39hf3d152e_3    conda-forge
nbdime                    3.1.0              pyhd8ed1ab_0    conda-forge
nbformat                  5.1.3              pyhd8ed1ab_0    conda-forge
ncurses                   6.2                  h58526e2_4    conda-forge
nest-asyncio              1.5.1              pyhd8ed1ab_0    conda-forge
netcdf4                   1.5.6           nompi_py39h36800e2_102    conda-forge
nodejs                    15.14.0              h92b4a50_0    conda-forge
notebook                  6.4.0              pyha770c72_0    conda-forge
nspr                      4.30                 h9c3ff4c_0    conda-forge
nss                       3.65                 hb5efdd6_0    conda-forge
numba                     0.53.1           py39h56b8d98_1    conda-forge
numpy                     1.21.1           py39hdbf815f_0    conda-forge
olefile                   0.46               pyh9f0ad1d_1    conda-forge
openjpeg                  2.4.0                hb52868f_1    conda-forge
openssl                   1.1.1k               h7f98852_0    conda-forge
packaging                 20.9               pyh44b312d_0    conda-forge
pandas                    1.2.4            py39hde0f152_0    conda-forge
pandoc                    2.13                 h7f98852_0    conda-forge
pandocfilters             1.4.2                      py_1    conda-forge
parso                     0.8.2              pyhd8ed1ab_0    conda-forge
partd                     1.2.0              pyhd8ed1ab_0    conda-forge
pathspec                  0.8.1              pyhd3deb0d_0    conda-forge
pcre                      8.44                 he1b5a44_0    conda-forge
pexpect                   4.8.0              pyh9f0ad1d_2    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pillow                    8.2.0            py39hf95b381_1    conda-forge
pip                       21.2.4             pyhd8ed1ab_0    conda-forge
pixman                    0.40.0               h36c2ea0_0    conda-forge
pooch                     1.5.1              pyhd8ed1ab_0    conda-forge
poppler                   0.89.0               h2de54a5_5    conda-forge
poppler-data              0.4.10                        0    conda-forge
postgresql                13.3                 h2510834_0    conda-forge
proj                      7.2.0                h277dcde_2    conda-forge
prometheus_client         0.10.1             pyhd8ed1ab_0    conda-forge
prompt-toolkit            3.0.18             pyha770c72_0    conda-forge
prompt_toolkit            3.0.18               hd8ed1ab_0    conda-forge
psutil                    5.8.0            py39h3811e60_1    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
pycodestyle               2.7.0              pyhd8ed1ab_0    conda-forge
pycparser                 2.20               pyh9f0ad1d_2    conda-forge
pyflakes                  2.3.1              pyhd8ed1ab_0    conda-forge
pygments                  2.9.0              pyhd8ed1ab_0    conda-forge
pyopenssl                 20.0.1             pyhd8ed1ab_0    conda-forge
pyparsing                 2.4.7              pyh9f0ad1d_0    conda-forge
pyproj                    3.1.0            py39h50a64a9_3    conda-forge
pyqt                      5.12.3           py39hf3d152e_7    conda-forge
pyqt-impl                 5.12.3           py39h0fcd23e_7    conda-forge
pyqt5-sip                 4.19.18          py39he80948d_7    conda-forge
pyqtchart                 5.12             py39h0fcd23e_7    conda-forge
pyqtwebengine             5.12.1           py39h0fcd23e_7    conda-forge
pyrsistent                0.17.3           py39h3811e60_2    conda-forge
pyshp                     2.1.3              pyh44b312d_0    conda-forge
pysocks                   1.7.1            py39hf3d152e_3    conda-forge
python                    3.9.5           h49503c6_0_cpython    conda-forge
python-dateutil           2.8.1                      py_0    conda-forge
python_abi                3.9                      1_cp39    conda-forge
pytz                      2021.1             pyhd8ed1ab_0    conda-forge
pyyaml                    5.4.1            py39h3811e60_0    conda-forge
pyzmq                     22.0.3           py39h37b5a0c_1    conda-forge
qt                        5.12.9               hda022c4_4    conda-forge
qtconsole                 5.1.0              pyhd8ed1ab_0    conda-forge
qtpy                      1.9.0                      py_0    conda-forge
rasterio                  1.2.3            py39h63973eb_0    conda-forge
readline                  8.1                  h46c0cb4_0    conda-forge
rebulk                    3.0.1                    pypi_0    pypi
regex                     2021.4.4         py39h3811e60_0    conda-forge
requests                  2.25.1             pyhd3deb0d_0    conda-forge
requests-cache            0.5.2                    pypi_0    pypi
rockhound                 0.2.0                      py_0    conda-forge
rope                      0.19.0             pyhd8ed1ab_0    conda-forge
scikit-learn              0.24.2           py39h4dfa638_0    conda-forge
scipy                     1.7.0            py39hee8e79c_1    conda-forge
send2trash                1.5.0                      py_0    conda-forge
setuptools                49.6.0           py39hf3d152e_3    conda-forge
shapely                   1.7.1            py39ha61afbd_4    conda-forge
simpervisor               0.4                pyhd8ed1ab_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
smmap                     3.0.5              pyh44b312d_0    conda-forge
sniffio                   1.2.0            py39hf3d152e_1    conda-forge
snuggs                    1.4.7                      py_0    conda-forge
sortedcontainers          2.4.0              pyhd8ed1ab_0    conda-forge
soupsieve                 2.2.1                    pypi_0    pypi
sqlite                    3.35.5               h74cdb3f_0    conda-forge
subdivx-download          0.6                      pypi_0    pypi
sympy                     1.8              py39hf3d152e_0    conda-forge
tblib                     1.7.0              pyhd8ed1ab_0    conda-forge
terminado                 0.9.4            py39hf3d152e_0    conda-forge
testpath                  0.5.0              pyhd8ed1ab_0    conda-forge
threadpoolctl             2.1.0              pyh5ca1d4c_0    conda-forge
tiledb                    2.2.9                h91fcb0e_0    conda-forge
tk                        8.6.10               h21135ba_1    conda-forge
toml                      0.10.2             pyhd8ed1ab_0    conda-forge
tomli                     1.2.1              pyhd8ed1ab_0    conda-forge
toolz                     0.11.1                     py_0    conda-forge
tornado                   6.1              py39h3811e60_1    conda-forge
traitlets                 5.0.5                      py_0    conda-forge
tvdb-api                  3.1.0                    pypi_0    pypi
tvnamer                   3.0.4                    pypi_0    pypi
typed-ast                 1.4.3            py39h3811e60_0    conda-forge
typing-extensions         3.7.4.3                       0    conda-forge
typing_extensions         3.7.4.3                    py_0    conda-forge
tzcode                    2021a                h7f98852_1    conda-forge
tzdata                    2021a                he74cb21_0    conda-forge
urllib3                   1.26.4             pyhd8ed1ab_0    conda-forge
verde                     1.6.1              pyhd8ed1ab_0    conda-forge
wcwidth                   0.2.5              pyh9f0ad1d_2    conda-forge
webencodings              0.5.1                      py_1    conda-forge
websocket-client          0.57.0           py39hf3d152e_4    conda-forge
wheel                     0.36.2             pyhd3deb0d_0    conda-forge
widgetsnbextension        3.5.1            py39hf3d152e_4    conda-forge
xarray                    0.19.0             pyhd8ed1ab_1    conda-forge
xerces-c                  3.2.3                h9d8b166_2    conda-forge
xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
xorg-libice               1.0.10               h7f98852_0    conda-forge
xorg-libsm                1.2.3             hd9c2040_1000    conda-forge
xorg-libx11               1.7.0                h7f98852_0    conda-forge
xorg-libxau               1.0.9                h7f98852_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xorg-libxext              1.3.4                h7f98852_1    conda-forge
xorg-libxrender           0.9.10            h7f98852_1003    conda-forge
xorg-renderproto          0.11.1            h7f98852_1002    conda-forge
xorg-xextproto            7.3.0             h7f98852_1002    conda-forge
xorg-xproto               7.0.31            h7f98852_1007    conda-forge
xrft                      0.3.0              pyhd8ed1ab_0    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
yaml                      0.2.5                h516909a_0    conda-forge
yarl                      1.6.3            py39h3811e60_1    conda-forge
zeromq                    4.3.4                h9c3ff4c_0    conda-forge
zict                      2.0.0                      py_0    conda-forge
zipp                      3.4.1              pyhd8ed1ab_0    conda-forge
zlib                      1.2.11            h516909a_1010    conda-forge
zstd                      1.4.9                ha95c52a_0    conda-forge
leouieda commented 3 years ago

That does feel like something we should handle automatically. What happens if you always run the variable through np.asarray? It only copies when it needs to and ideally DataArrays should support this conversion without a copy.

santisoler commented 3 years ago

I've been rethinking about this issue and concluded that I blamed Verde too quickly for a bug in Harmonica. I think that the make_xarray_grid is intended to be used on a Numpy -> Xarray workflow, so letting it accept xarray objects doesn't sound too clever. In fact, Xarray objects offers more robust ways to merge xarray objects into a single Dataset.

The origin of the bug in Harmonica is on the prism_layer function, which is intended to work both with Numpy arrays and with Xarray objects. I'll fix that in a PR in Harmonica.

For this reason I'm closing this issue. If you or anyone thinks otherwise, please feel free to comment and/or reopen it if it's necessary.

santisoler commented 3 years ago

See https://github.com/fatiando/harmonica/pull/243