Astropy support

[HealthyPear]

Edit by @maximlt : issue originally opened on hvPlot.

Is your feature request related to a problem? Please describe.

I'd like to use hvplot (but more generally the whole holoviz ecosystem) on data structures defined by the astropy project. Such data structures can be simple tables, but also Quantity Tables with units (see their docs).

Describe the solution you'd like

Essentially an analogue of what is already available for other libraries like Pandas

Describe alternatives you've considered

Astropy can export its tables into a pandas dataframe, but in doing so you lose the units metadata.

[ahuang11]

Can astropy translate to xarray? If so, does it keep units?

[maximlt]

@Hoxbro do you think this discussion should take place on HoloViews' tracker? Although hvPlot could have a simple implementation like it does for polars as an easy entry point. @HealthyPear do you happen to know if HoloViz is already used in the Astropy community?

[HealthyPear]

Can astropy translate to xarray? If so, does it keep units?

No, only to Pandas (losing the units)

do you happen to know if HoloViz is already used in the Astropy community?

I have never seen an official or affiliated package using this framework. I think adding this feature is exactly what might spark interest in the community because any astronomical data source from the supported data formats can be loaded by a QTable.

[maximlt]

Thanks @HealthyPear for your feedback, I'm going to bring this up in our next community calls https://holoviz.org/community.html, feel free to join!

[HealthyPear]

Now that I think about it, I opened this issue on a completely personal basis but I am not one of the maintainers 😅

I don't know if it can be of interest to them so I ping them with no obligation whatsoever @taldcroft @astrofrog - sorry I only interacted with them in the past, maybe they know specific people who are competent on a possible interface implementation!

[maximlt]

What we can do at the hvPlot level is implement a small interface that under the hood would convert astropy objects in objects hvPlot/HoloViews know how to deal with (Pandas, Xarray, etc.). This should be quite simple to implement, we've done that recently for polars (https://github.com/holoviz/hvplot/pull/1129). However, this requires some conversion so it's not efficient and will be lossy (units lost). A more serious integration would require to add Astropy as one of the supported backends in HoloViews, except that this is much more involved and no one has volunteered to take that challenge :)

[HealthyPear]

A more serious integration would require to add Astropy as one of the supported backends in HoloViews,

Unfortunately, this is the only approach that might make sense to implement concerning this issue because it is already possible to convert an astropy table to a Pandas dataframe with the previously mentioned limitations. The only point of adding support to astropy would be to keep units and table metadata once loaded by HoloViews backend.

[HealthyPear]

Is there a developer tutorial on how to develop a new backend? that might help to ease the development of such an interface

[HealthyPear]

another limitation of pandas wrt to astropy tables is that a pandas dataframe does not support multidimensional columns

see https://docs.astropy.org/en/stable/table/pandas.html

[taldcroft]

As a project Astropy is certainly interested in inter-operability with other community packages, so I'm happy to answer questions from the astropy Table perspective.

I'm not familiar with hvplot, but the question asking for a tutorial on how to develop a new backend seems on point. Basically what does the data source backend need to provide to work with hvplot?

[taldcroft]

And certainly the hope would be to avoid converting Table to pandas DataFrame since the loss of units and N-d columns would be a problem for astronomers.

[ahuang11]

Perhaps hvplot can support pint-xarray (if it doesn't already), then it's just a matter of converting astropy -> pint-xarray -> hvplot? https://pint-xarray.readthedocs.io/en/stable/creation.html

[maximlt]

HoloViews has no tutorial for how to develop a new backend. They all live in this subpackage: https://github.com/holoviz/holoviews/tree/main/holoviews/core/data.

HoloViews has some very basic support to display the unit set on a dimension, however, full unit support hasn't yet been implemented (https://github.com/holoviz/holoviews/issues/21).

Since this discussion revolves mostly about HoloViews and that there's no interest in a basic support in hvPlot, I'm migrating it to HoloView's repo :)

[HealthyPear]

from hvplot pandas interface I see

try:
        import pandas as pd
except:
    raise ImportError('Could not patch plotting API onto pandas. '
                      'Pandas could not be imported.')

analogously in holoviews/core/data/pandas.py you need to import pandas, which to me means that in order for holoviz to interface with another library that library needs to be installed.

If this is true, at least for astropy the units would be automatically supported via the interface because you could just use the units module of astropy (in the interface).

[HealthyPear]

HoloViews has no tutorial for how to develop a new backend.

is there at least an API "skeleton" each interface is supposed to follow?

[ahuang11]

I'm not sure, but I asked on the HoloViz Discord to see if others with more HoloViews internals know.