intake / xrviz

Interactive visualisation interface for Xarrays
https://xrviz.readthedocs.io
BSD 3-Clause "New" or "Revised" License
105 stars 21 forks source link

TextInput None Error #74

Closed pgierz closed 3 years ago

pgierz commented 3 years ago

Hello,

I'm trying to use xrviz with a multi-file Xarray dataset containing netcdf data. Originally, the data looks like this: Screen Shot 2021-04-20 at 6 07 31 PM

I have various data variables, defined in Lon/lat/time (and sometimes vertical level). The output unfortunately has rather bad time-stamping. For example, the date 3700-01-31 is stored as a float 37000131.75 so I have a small function to clean that up. Note that these are hypothetical dates, and could just as easily just count months upward from an arbitrary year 1:

import cftime
from intake_xarray.netcdf import NetCDFSource
import xarray

class Echam5MainMM(NetCDFSource):
    name = "echam5-main-mm"
    version = "0.1.0"
    partition_access = True

    #TODO(PG)
    # def load
    # def fixup

    def _preprocess_ds(self, ds_in):
        ts_in_arr = list(ds_in.coords["time"].data)
        ts_in = ds_in.coords["time"]
        for idx, ts_in in enumerate(ts_in):
            ts_out = self._echam_timestamp_to_cftime(ts_in)
            ts_in_arr[idx] = ts_out
        ds_out = ds_in.assign_coords({"time": ts_in_arr})
        ds_out.time.attrs.update(ds_in.time.attrs)
        return ds_out

    @staticmethod
    def _echam_timestamp_to_cftime(ts_in):
        if not isinstance(ts_in, xarray.core.dataarray.DataArray):
            raise TypeError("Please pass timesteps in after being loaded into xarray")
        assert hasattr(ts_in, "units")
        ts_in = str(ts_in.data).split(".")[0]
        year = ts_in[:4]
        month = ts_in[4:6]
        day = ts_in[6:9]
        ts_out = cftime.datetime(int(year), int(month), int(day))
        return ts_out

My eventual goal is to build an intake driver, and I'm learning more about how that works. The first step would be to have the dates automatically cleaned up. I haven't hooked up the intake driver yet and am still running the preprocess method by hand (if there are any hints how to do that, I'd be very grateful!)

So, I have something like:

>>> data_source = Echam5MainMM("example-data/conpi_echam5_main_mm_*.nc")
>>> xrds = data_source.read_chunked()
>>> xrds_out = conpi_echam5_main._preprocess_ds(xrds)
>>> import xrviz
>>> db = xrviz.dashboard.Dashboard(xrds_out)
>>> db.show()

I noticed that the read_chunked takes a while, even though it should just be loaded in the metadata rather than the actual arrays. If there are any ideas how to speed that up, I'd be interested.

The xrviz dashboard loads up and shows me a list of variables, which seems correct.

image

However, the plot button is greyed out, and I can find the following error in the console:

Found valid latitude/longitude coordinates, assuming latitude_longitude for projection grid_mapping variable
tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOMainLoop object at 0x10af69b20>>, <Task finished name='Task-88' coro=<_needs_document_lock.<locals>._needs_document_lock_wrapper() done, defined at /Users/pgierz/anaconda3/envs/paleodyn-database-viewer/lib/python3.9/site-packages/bokeh/server/session.py:51> exception=ValueError("failed to validate TextInput(id='1170', ...).value: expected a value of type str, got None of type NoneType")>)
Traceback (most recent call last):
  File "/Users/pgierz/anaconda3/envs/paleodyn-database-viewer/lib/python3.9/site-packages/tornado/ioloop.py", line 741, in _run_callback
    ret = callback()
  File "/Users/pgierz/anaconda3/envs/paleodyn-database-viewer/lib/python3.9/site-packages/tornado/ioloop.py", line 765, in _discard_future_result
    future.result()
  File "/Users/pgierz/anaconda3/envs/paleodyn-database-viewer/lib/python3.9/site-packages/bokeh/server/session.py", line 71, in _needs_document_lock_wrapper
    result = await result
  File "/Users/pgierz/anaconda3/envs/paleodyn-database-viewer/lib/python3.9/site-packages/tornado/gen.py", line 216, in wrapper
    result = ctx_run(func, *args, **kwargs)
  File "/Users/pgierz/anaconda3/envs/paleodyn-database-viewer/lib/python3.9/site-packages/panel/reactive.py", line 252, in _change_coroutine
    self._change_event(doc)
  File "/Users/pgierz/anaconda3/envs/paleodyn-database-viewer/lib/python3.9/site-packages/panel/reactive.py", line 262, in _change_event
    self._process_events(events)
  File "/Users/pgierz/anaconda3/envs/paleodyn-database-viewer/lib/python3.9/site-packages/panel/reactive.py", line 245, in _process_events
    self.param.set_param(**self._process_property_change(events))
  File "/Users/pgierz/anaconda3/envs/paleodyn-database-viewer/lib/python3.9/site-packages/param/parameterized.py", line 1472, in set_param
    self_._batch_call_watchers()
  File "/Users/pgierz/anaconda3/envs/paleodyn-database-viewer/lib/python3.9/site-packages/param/parameterized.py", line 1611, in _batch_call_watchers
    self_._execute_watcher(watcher, events)
  File "/Users/pgierz/anaconda3/envs/paleodyn-database-viewer/lib/python3.9/site-packages/param/parameterized.py", line 1573, in _execute_watcher
    watcher.fn(*args, **kwargs)
  File "/Users/pgierz/anaconda3/envs/paleodyn-database-viewer/lib/python3.9/site-packages/xrviz/sigslot.py", line 76, in _signal
    self._emit(self._map[wn], event.new)
  File "/Users/pgierz/anaconda3/envs/paleodyn-database-viewer/lib/python3.9/site-packages/xrviz/sigslot.py", line 95, in _emit
    if callback(value) is False:
  File "/Users/pgierz/anaconda3/envs/paleodyn-database-viewer/lib/python3.9/site-packages/xrviz/style.py", line 95, in setup
    self.lower_limit.value = None
  File "/Users/pgierz/anaconda3/envs/paleodyn-database-viewer/lib/python3.9/site-packages/param/parameterized.py", line 318, in _f
    return f(self, obj, val)
  File "/Users/pgierz/anaconda3/envs/paleodyn-database-viewer/lib/python3.9/site-packages/param/parameterized.py", line 936, in __set__
    obj.param._call_watcher(watcher, event)
  File "/Users/pgierz/anaconda3/envs/paleodyn-database-viewer/lib/python3.9/site-packages/param/parameterized.py", line 1591, in _call_watcher
    self_._execute_watcher(watcher, (event,))
  File "/Users/pgierz/anaconda3/envs/paleodyn-database-viewer/lib/python3.9/site-packages/param/parameterized.py", line 1573, in _execute_watcher
    watcher.fn(*args, **kwargs)
  File "/Users/pgierz/anaconda3/envs/paleodyn-database-viewer/lib/python3.9/site-packages/panel/reactive.py", line 238, in _param_change
    self._apply_update(events, msg, model, ref)
  File "/Users/pgierz/anaconda3/envs/paleodyn-database-viewer/lib/python3.9/site-packages/panel/reactive.py", line 192, in _apply_update
    self._update_model(events, msg, root, model, doc, comm)
  File "/Users/pgierz/anaconda3/envs/paleodyn-database-viewer/lib/python3.9/site-packages/panel/reactive.py", line 205, in _update_model
    model.update(**msg)
  File "/Users/pgierz/anaconda3/envs/paleodyn-database-viewer/lib/python3.9/site-packages/bokeh/core/has_props.py", line 441, in update
    setattr(self, k, v)
  File "/Users/pgierz/anaconda3/envs/paleodyn-database-viewer/lib/python3.9/site-packages/bokeh/core/has_props.py", line 298, in __setattr__
    super().__setattr__(name, value)
  File "/Users/pgierz/anaconda3/envs/paleodyn-database-viewer/lib/python3.9/site-packages/bokeh/core/property/descriptors.py", line 552, in __set__
    self._internal_set(obj, value, setter=setter)
  File "/Users/pgierz/anaconda3/envs/paleodyn-database-viewer/lib/python3.9/site-packages/bokeh/core/property/descriptors.py", line 784, in _internal_set
    value = self.property.prepare_value(obj, self.name, value)
  File "/Users/pgierz/anaconda3/envs/paleodyn-database-viewer/lib/python3.9/site-packages/bokeh/core/property/bases.py", line 350, in prepare_value
    raise ValueError(f"failed to validate {obj_repr}.{name}: {error}")
ValueError: failed to validate TextInput(id='1170', ...).value: expected a value of type str, got None of type NoneType

Also, the online documentation seems to have options for setting up the projection. I can't find those.

Any hints would be great! Thanks! Paul

pgierz commented 3 years ago

Note that setting these two lines to empty strings seems to solve that problem: https://github.com/intake/xrviz/blob/b5969232b14aad408cbceceea41469dae6cf3235/xrviz/style.py#L93-L96

However, I am not sure if that causes other problems. I'm happy to send a PR for that if the developers agree.

martindurant commented 3 years ago

You can set CRS as detailed on this page, but there is no automatic way to extract or guess this.

Your proposed solution seems like a reasonable thing to do - text boxes should have string values, not None. However, it's been a while since I've looked at this code.

@hdsingh , do you agree with this change? More generally, are you likely to have the time to respond to any issues that crop up here?

hdsingh commented 3 years ago

Yes, it looks reasonable. Although we are converting the input into string later on here, not sure what's exactly causing this issue. @pgierz Can you please also verify if this works fine for the sample dataset as well.

@martindurant Please feel free to tag me in the issues, I will try my best to respond when I have some bandwidth.

martindurant commented 3 years ago

Thanks, @hdsingh

My eventual goal is to build an intake driver, and I'm learning more about how that works.

@pgierz , don't hesitate to ask on the intake tracker.

pgierz commented 3 years ago

Hi @hdsingh, sure. I did a clean install just to be safe. I basically did the Conda-variant of the TravisCI config:

$ mkdir -p ~/Code/intake
$ cd ~/Code/intake
$ git clone https://github.com/intake/xrviz.git
$ cd xrviz
$ conda create -c conda-forge -n xrviz-dev python=3.7 pytest --file requirements.txt
$ conda activate xrviz-dev
$ pip install . --no-deps
$ pytest

That resulted in all tests passing:

============================= test session starts ==============================
platform darwin -- Python 3.7.10, pytest-6.2.3, py-1.10.0, pluggy-0.13.1
rootdir: /Users/pgierz/Documents/Code/github.com/intake/xrviz
collected 50 items / 1 skipped / 49 selected                                   

xrviz/tests/test_control.py .                                            [  2%]
xrviz/tests/test_coord_setter.py .                                       [  4%]
xrviz/tests/test_dashboard.py ...........ss........s                     [ 48%]
xrviz/tests/test_describe.py .                                           [ 50%]
xrviz/tests/test_display.py ...                                          [ 56%]
xrviz/tests/test_example.py .                                            [ 58%]
xrviz/tests/test_fields.py ......                                        [ 70%]
xrviz/tests/test_style.py ..........                                     [ 90%]
xrviz/tests/test_utils.py .....                                          [100%]

=============================== warnings summary ===============================
../../../../../anaconda3/envs/xrviz-dev/lib/python3.7/site-packages/fsspec/__init__.py:47
  /Users/pgierz/anaconda3/envs/xrviz-dev/lib/python3.7/site-packages/fsspec/__init__.py:47: DeprecationWarning: SelectableGroups dict interface is deprecated. Use select.
    for spec in entry_points.get("fsspec.specs", []):

xrviz/tests/test_dashboard.py::test_2d_variable_for_dims
  /Users/pgierz/anaconda3/envs/xrviz-dev/lib/python3.7/site-packages/holoviews/operation/datashader.py:5: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
    from collections import Callable

xrviz/tests/test_dashboard.py::test_create_taps_and_series_graph_for_2d_coords
  /Users/pgierz/anaconda3/envs/xrviz-dev/lib/python3.7/site-packages/xarray/core/dataarray.py:4226: DeprecationWarning: Behaviour of argmin/argmax with neither dim nor axis argument will change to return a dict of indices of each dimension. To get a single, flat index, please use np.argmin(da.data) or np.argmax(da.data) instead of da.argmin() or da.argmax().
    result = self.variable.argmin(dim, axis, keep_attrs, skipna)

-- Docs: https://docs.pytest.org/en/stable/warnings.html
================== 47 passed, 4 skipped, 3 warnings in 35.41s ==================

Running xrviz show xrviz/sample_data/great_lakes.nc technically works, yet has the same grayed-out plot button as in the example using my own data. My proposed change fixes the example as well as my own data, see #75

@martindurant, I may come back to you with some questions, thank you. For now, I need to puzzle around a bit first and learn the library