search

SpatioTemporal / pystare

The Python interface for the SpatioTemporal Adaptive Resolution Encoding (STARE), a unified indexing for geolocated data.
https://pystare.readthedocs.io/en/latest/
12 stars 2 forks source link

parallelize conversions/lookup with dask/xarray ufuncs #95

Open NiklasPhabian opened 2 years ago

NiklasPhabian commented 2 years ago

It would seem like dasks makes things as easy as:

import dask.distributed
import pystare

sids = xarray.apply_ufunc(pystare.from_latlon_2d, lats_x, lons_x, dask="parallelized", kwargs={'level': level})
with dask.distributed.Client(n_workers=60, threads_per_worker=2, memory_limit='2GB') as client:
    sids = compute.compute()
sids = xarray.DataArray.to_numpy(sids)

We might want to wrap all the bottleneck functions