search

simonsobs / sotodlib

Simons Observatory: Time-Ordered Data processing library.
MIT License
15 stars 18 forks source link

Wavelet Jumps #797

Open skhrg opened 6 months ago

skhrg commented 6 months ago

Mostly just a looming todo item for me so that I can feel ok about putting this on the back burner and work on more pressing things.

Finding jumps with a matched filter seems to mostly work but there are some issues namely:

  1. A fourier space convolution doesn't really like the sharp discontinuity
  2. Real space convolution are slow...

But wavelets make a lot of sense here since they are good at dealing with these sorts of localized features.

A naive attempt of looking at the CWT with a gaus1 wavelet seems to work well, but CWTs are probably more expensive than we need. This paper uses a DWT dydatic scales and seems to be the basis of a lot of modern approaches (though I cannot find a python implementation of the wavelet they use). This paper is a good followup with more information about dealing with noise.

Regardless of the exact technique used I think my takeaway is that using methods like this to then get an estimate of the Lipshitz regularity would be a very powerful technique in our flagging toolkit (since that also get us glitches at the same time but in a way where we know what is what). This paper is a good overview of estimating alpha.

I can imagine two modes here that we could use:

  1. Estimate alpha at each point in the TOD and then use that to flag jumps and gltiches.
  2. Do a cheap but perhaps false positive prone selection (ie: a thresh on the gradient, this can be done in the wavelett basis as well) and then estimate alpha in each of the flagged zones to kill false positives.

Overall I think that these sorts of wavelet techniques have the potential to be the final boss of giltch and jump flagging and merit a detailed look (the literature on this specific application seems deep). Will try to get something worthy of a PR cobbled together in the coming weeks.

msilvafe commented 4 weeks ago

Can we get an update on this @skhrg ?

skhrg commented 4 weeks ago

Had a student work on this over the summer and by jove it worked! I'd like to have him clean up and speed up the code before its ready for others to run but I'll send you his poster on slack.