leouieda
commented
6 years ago Hi @djhoese et al., thanks for staring the conversation! I've been looking at pyresample and there are a lot of similarities (and some really cool examples!). It seems that the main use case is to go from a grid in a projection/region/orientation to another grid. Is that right? In this sense, it makes sense that satpy uses xarray as an internal data format.
For Verde, the main use case is unevenly sampled data to grids. So xarray is mostly an output format for me. The accessor pattern is really cool but it only works if you're starting out with an xarray Dataset. We usually have pandas.DataFrame or numpy 1D arrays. This is why the inputs to fit are coordinates and data separately. The same applies to having projection information etc. It would be cool to have Verde include that in the output grid but we get input data in all kinds of formats (usually text files with bad headers, or none at all).
I've been focusing more on the interface and worrying about efficiency later. That said, I started using pykdtree and it's really fast! Sadly, that's not what we use for interpolation. The gridders we have are more like radial basis functions so our bottleneck is building the sensitivity/jacobian/feature matrix and then solving the least squares problem. Right now, I'm using dumb numpy operations to build the matrix and scikit-learn to do the fitting. This is really not ideal but I've been bitten by premature optimization before :wink:
Verde still has some way to go in terms of features. Right now, I have the bare minimum. This all started because I'm working on a vector interpolation method and needed some utilities and a basis for comparison. Much of what is currently here has been in GMT for a long time (greenspline and blockmean). I reimplemented them because I wanted to tweak things without having to dive into C code and use bash. In the end, I like the verde interface better and plan on adding different Green's functions for spherical gridding, etc.
One thing that I've been wanting and pyresample already has is filtering. I sometimes need to downsample gridded data and running a lowpass filter first is crucial. I have some other projects that use gridded data more often and will definitely have use cases for pyresample.
It would be really great to have a similar API at least. I'm not opposed to merging if there is significant overlap. The Verde interface is not terribly complex (see BaseGridder). A class that implements fit, predict, and filter can be used with all other classes (through Chain). For example, a spatial filtering class could have the following interface:
class GaussianFilter():
def __init__(self, width):
self.width = width
def filter(self, coordinates, data, weights=None):
"""
Run the filter on the data and output new coordinates and data.
Can assume that data is 2D and raise exception if not.
"""
...It could then be used to filter some gridded data before resampling in a pipeline:
data = xarray.open_dataset(...)
chain = vd.Chain([
("filter", GaussianFilter(width=100e3)),
("resampler", Resampler(area_def)),
])
# Fit will filter the data before passing it to Resampler
chain.fit((data.longitude, data.latitude), data.scalars)
# The filter doesn't impact the data prediction so it's not called when gridding
new_grid = chain.grid(region=..., spacing=...)Is this something that would be interesting to you?
djhoese
I am one of the maintainers of the pyresample project (https://github.com/pytroll/pyresample) along with @mraspaud @adybbroe @pnuu. We do some very similar operations to what you have in verde, but with different interfaces and sometimes different purposes. I think our two projects could benefit from each other.
Pyresample/SatPy
Pyresample has been around for a long time and is used by many meteorological institutes/organizations via the satpy library (https://github.com/pytroll/satpy) and mpop before that. I also use satpy in my command line tools under the polar2grid and geo2grid project and joined the PyTroll/SatPy team after attending conferences and realizing that we were solving exactly the same problems.
Pyresample and satpy now use xarray and dask for all of their operations, but pyresample is in need of a new interface now that xarray accessors are more popular and the ease of attaching geolocation information to an xarray object (see https://github.com/pydata/xarray/issues/2288). Pyresample offers (or will very soon) three different resampling algorithms: nearest neighbor using pykdtree, elliptical weighted averaging for scan-based satellite instrument data, and bilinear interpolation using pykdtree for neighbor calculations. The old pyresample API also includes the ability to use your own weighting functions, but we haven't migrated that to work with the xarray/dask functionality.
The use cases that pyresample were originally used for and what people were used to doing involved having predefined 500m-2km grids in various projections. These grids can also be dependent on the visualization tool used to view the satellite data where only certain projections can be used or certain resolutions are too dense for satellite raster images. This is my opinion/experience, the other pyresample developers may see differently. Anyway, a lot of pyresample's interfaces assume you have a predefined grid ("AreaDefinition") that you want to resample your data to. It is coming up more and more that we need an easier interface for making these definitions from other existing definitions like lon/lat arrays or adjustments to existing AreaDefinitions (change resolution). While this is all possible, it isn't always the easiest to do or access or document.
Verde
I spoke with you @leouieda at SciPy 2018 and on gitter which lead to #92 where verde now uses pykdtree if it is available for better performance. You also mentioned in your lightning talk I believe that you'd like to switch to dask where possible but haven't done so yet. This is something pyresample's experience could help with since dask-ifying a kdtree is not easy, but I think we've done the best we can without rewriting the C code.
You are developing some really nice interfaces in verde from what I've seen and make certain use cases extremely easy. Like the last paragraph I described for pyresample above. Pyresample could benefit from interfaces like these I think or both projects could benefit by adding these interfaces to xarray accessors.
Collaboration
One library, two libraries, one library that depends on another, two libraries that depend on a third, or just similar interfaces...I'm not sure what would be best. I do think that we could benefit from working together. This issue isn't requesting that either project be absorbed by the other, but to make each aware of the other.
@leouieda Where do you see verde's features overlapping with pyresample? How many more features do you still want to add to verde (assuming verde is young-ish and seeing how many PRs you are making every day)?
How far do you see verde's feature set going? What else do you want it to do? Does it/will it do more than gridding and the related utilities?