Suggestions for additional algorithms

[jscargle]

Hi ... it is great to see this development! I have gotten sidetracked matters of some gravity, and have postponed development and testing of some ideas that may be useful ...

I think there is no mention of co-spectra in your list ... maybe this is because it is more or less limited to NuSTAR data?

In any case, I would like to suggest a few things that are in development and in various stages:

• a Fourier transform (amplitude AND phase) for unevenly spaced data
• time-frequency analysis of event data and other unevenly spaced data
• co-spectra of event data
• novel methods for de-noising LIGO strain time series

[matteobachetti]

Hi @jscargle, sorry for the late reply, and thanks a lot for your suggestions! Co-spectra are just included in cross-spectra, that is why we don't mention them explicitly. We do need to add the new work we've done on their statistics though :).

We have been discussing about adding some of those features but we are a small group and time is always... sparse! Can you point us to the docs or some open-source code (not necessarily in Python) that implements the actual algorithms?

[jscargle]

Thanks! ... I can point you to a short and simple MatLab script that essentially updates the FORTRAN (do you remember FORTRAN?) in "Studies in astronomical time series analysis. III - Fourier transforms, autocorrelation functions, and cross-correlation functions of unevenly spaced data," 1989ApJ.,343..874 [simple to say, but a lot of sweat and tears testing and dealing with the singular case of evenly spaced data, done as a sanity check ... :-) ] This generalizes the infamous LS to the full complex FT, enabling much neglected phase analysis viewpoints. Let me first consult with a few others with whom I have discussed githubbing this algorithm and then get back to you.

[dhuppenkothen]

Hi @jscargle. Thank you for your suggestions! I'm curious: is the implementation in that paper different from e.g. the implementation of the Lomb-Scargle periodogram in astropy or gatspy?

I've sort of implicitly left out methods for unevenly sampled data, since the Lomb-Scargle periodogram is implemented in astropy, but if there are generalizations that aren't implemented and that you think the community would benefit from, then I'd love to see them implemented in Stingray!

[jscargle]

Hi ... the main difference is that it computes the full Fourier transform of arbitrarily spaced data. The absolute square is then equivalent to LS ... but the implementation is different from that in astropy, but maybe only in irrelevant details connected with MatLab vs. Python. So the main "value added" is the phase spectrum. Let me know if you would like to see the code.

[jscargle]

... oh, and you can also do the co-spectrum if you have the full FT! In this way I have implemented a bin-free co-spectrum estimator for event (time-tagged photon) data.

[matteobachetti]

@jscargle finally, after almost three years from this kind offer from you, we are working on extending Stingray to periodograms and cross spectra for unevenly sampled datasets! I think it will be extremely useful, given the number of instruments in low-Earth orbit with very frequent occultations or SAA passages.

@dhruv9vats is the Google Summer of Code student who has implemented the Multi-taper periodogram for us, and he will work now on extending the Powerspectrum and Crossspectrum classes to unevenly sampled data. Was there any development in the Githubization of the code you mention above? Let us know, and thanks again!

[jscargle]

Hi Matteo ... good to hear from you.

First: I do not check my nasa mail account very often and mostly use this gmail account ...

I forget the details of our correspondence, but I would be glad if there were some interest in an algorithm for the (complex) Fourier transform of unevenly sampled data. I have updated an old FORTRAN (sic!) program to implement this, translating it to MatLab. The absolute-square of the output is exactly the Lomb-Scargle periodogram, but it includes the phase spectrum. In many applications the Fourier transform might be of use.

I would be glad to provide the code, including a test script, if someone would be interested, say, in translating it to Python (should be very simple) and adding it to Stingray -- if that makes sense on your end.

Cheers, Jeff

PS. Your mention of the SAA reminded me that a while ago I used this algorithm to compute power spectra of Fermi photon data (that is, time-tagged events, another form of uneven sampling) and discovered that the spectra of these gamma ray data have a forest of lines corresponding to the orbit of the spacecraft -- a precessional modulation, the orbital frequency of course, a subtle modulation due to the SAA passages, as well as harmonics of, and beats between, these frequencies.

Jeff Scargle Astrobiology and Space Science Division NASA Ames Research Center @.*** 415 385 4297 (cell)

[matteobachetti]

Hi Jeff, You are receiving these messages from Github, where you can find the original conversation: https://github.com/StingraySoftware/stingray/issues/368 I don't have your Gmail account (it's hidden behind the Github notification address), feel free to send me a private email at matteo dot bachetti at inaf dot it .

We would be absolutely happy to translate the code to Python. This would be a great addition to our codebase and I suspect the people that develop Astropy.timeseries and other libraries for astronomical data analysis would be interested as well.

Thanks a lot, Matteo

[matteobachetti]

This should be resolved by #737. Thanks a lot @jscargle for pointing out the cross spectrum algorithm and providing the example code!