Possible merge with xarray-dsp and other projects

smartass101 commented 6 years ago

As we have discussed in the mailing group, it might be reasonable to merge this project with my smaller project smartass101/xarray-dsp.

The xarray_dsp package in the project currently features mostly functions for frequency and other filters and spectral analysis and a few other wrappers. It mostly targets time-domain analysis.

There are also a few wrappers for wavelets from pycwt, but I guess those wouldn't really fit into xr-scipy since pycwt is not included in scipy. It's also a question whether you want to restrict this project to scipy functions.

I also found kaipak/xrsigproc which also uses some smoothing functions from scipy.ndimage and scipy.convolve for (probably) spatial filtering. There is also serazing/xscale which also does some signal processing and FFT using mostly scipy again. Maybe we could contact these projects as well.

fujiisoup commented 6 years ago

Thanks for contacting me.

Also, I know crusaderky/xarray_extras. There must be much more small packages that have been developed individually. For a limited list, see FAQ page of xarray's document page

It is a good idea to merge these packages into one, which would reduce the maintenance and development cost. Howerver, it is also possible that developers would be flustrated when there is any disagreement in the development direction.

So, I think it would be nice to decide a project goal before contacting to other project managers. My idea is

Only implement functions that are in numpy or scipy.
Try to keep API and package structure as similar to those of numpy and scipy as possible.

But honestly, current development and maintenance status are far from satisfactory yet, because of lack of resources. For example, interpolate module should be deprecated as xarray now natively supports it. fft module should be updated to follow the recent xarray update (#3).

I am really welcome your possible contribution to this project. I think xarray_dsp could be implemented into signal module as similar to scipy.signal module.

Any thought is welcome, for the implementation or the project goal.

smartass101 commented 6 years ago

I'm not sure how closely to follow the scipy/numpy API, because their functions usually expect separate arrays wheres when using xarray many new "higher-level" concepts become easily accessible. For instance, xarray-dsp mostly follows the scip.signal API for the spectral analysis tools, but is a bit different for the filter routines, because it becomes easy to automatically construct and apply a filter based on the information in the time axis of an array. This is a more of a "high-level" approach than to design filter coefficients separately.

Xscale also seems to focus on a higher-level API for windowing.

Ultimately, I'd focus mostly on what functions are commonly used in actual real-world analysis of data.

fujiisoup commented 6 years ago

Ultimately, I'd focus mostly on what functions are commonly used in actual real-world analysis of data.

I looked at your filter module, and find high level functions such as highpass are really neat. On the other hand, an advantage of the similar API to scipy or numpy is that we can greatly lsimplify the document; users do not need to read our document in its details if they know how to use scipy's counterpart. It would be also important and we may need to balance these two.

Can we prepare high level functions as well as scipy's counterpart?

OriolAbril commented 3 years ago

Hi!

I have created a library that has wrappers for scipy.stats (among other things): https://xarray-einstats.readthedocs.io/en/latest/.

It currently has:

wrappers for many numpy.linalg functions
wrappers for scipy.stats distributions plus a few functions
wrappers for [einops]() functions
a numba.guvectorize-decorated version of numpy.histogram for dataarrays

I have mostly added wrappers for things I personally use, also trying to not overlap with your library nor xarray-extras, so it might not make much sense as a group, but the modules are completely independent between them and could be reorganized into independent packages or merged into existing ones. I don't know what your roadmap is nor if there is interest within the xarray for scipy wrappers or similar things, but it could be interesting to collaborate.

In my case, I might add some very thin wrappers on a few special functions that I use often (i.e. expit, logsumexp), and some more guvectorized things, but the things above are mostly what I'll use and what I am willing to maintain.

Also, feedback very welcome!

fujiisoup commented 3 years ago

Hi. Thank you for the suggestion.

Honestly speaking, i have nowadays no time to maintain this project🥲 And furthermore, most of the features of this package is now implemented into the xarray.

If anyone is interested in developing this package, I'm happy to give the ownership of this repository. It should be better for the open source community.

andersy005 commented 3 years ago

Honestly speaking, i have nowadays no time to maintain this project🥲 If anyone is interested in developing this package, I'm happy to give the ownership of this repository. It should be better for the open source community.

@fujiisoup, would you be up for transferring the project to the xarray-contrib organization? I am thinking that It would help increase the overall visibility of the project and in the process, some folks may chip in to help with the development/maintenance of the project moving forward if need be.

fujiisoup commented 3 years ago

would you be up for transferring the project to the xarray-contrib organization?

@andersy005, Yes, if it is OK to transferring buggy one