We need better bandpower definitions (and choices of Q).

[aewallwi]

Our power spectrum estimators define bandpowers as the square of FFT modes (or very close to FFT modes with a sinc multiplication). This choice of bandpower definition has the problem that intrinsic foregrounds occupy all bandpowers at a significant level and the foreground covariance is far from diagonal, causing demixing machinery to fail (see section 7.1 of https://www.overleaf.com/read/krkyhmqzghzj ). We should investigate and implement bandpower definitions that better diagonalize the signal covariance and isolate foregrounds from signal.

Rather then a specific single feature request, this is a call for work to be performed on an open research project that will lead to a series of hera_pspec features.

[aewallwi]

Here are some potential directions of investigation (I think these would be nice student projects).

One idea: bandpowers are the eigenmodes of an Analytic covariance matrix similar to sinc downweighting but with a diagonal that is not flat in continuous delay space but is instead enveloped by a simple analytic FT-able function such as a cosine (to avoid degeneracy issues).

Another idea: band-powers are the eigenvectors of a covariance matrix derived from a large ensemble of simulated 21cm / foreground models.

[acliu]

This is an interesting thing to discuss. I am not yet sure I am on board with something like the second idea. I strongly prefer bandpowers that at least have some level of localization in Fourier space, so that the window functions are at least somewhat peaked. (Although I think there's a possibility that @jaguirre disagrees with me on this, based on my fuzzy memory of some past conversations).

Suppose I take an extreme example where the foregrounds look like RFI, i.e., a delta function in frequency. In the limit where this foreground is very strong, one of the eigenvectors will just be that delta function. If this is then one of the bandpowers, the Fourier space footprint (i.e., the window function will be spread over all k's). That makes the power spectrum a difficult thing to interpret as an intermediate result. One could make the argument that maybe we don't care, and any quadratic statistic is ok as long as we can fit theory parameters. I'm a little uncomfortable with that. I'd like the power spectrum as an intermediate step.

It may, of course, turn out to be the case that realistic foregrounds don't suffer from this problem.

Note that if the bandpowers are the eigenvectors, I think the power spectrum estimation problem becomes a trivial one. I think one simply has to put the data in the eigenbasis and then it's just a squaring. So this is really a generalized mapmaking problem rather than a power spectrum estimation. It's pretty similar to expressing the data in KL modes like they did in some of the preprocessing steps of the SDSS P(k) pipeline back in the day. If we think of the problem in this way, it allows us to retain a lot of flexibility. Once the data are mapped into KL modes, if you want to treat these modes as the bandpowers, you just square. Or, you use these modes as an intermediate step and write another Q that maps them to traditional Fourier bandpowers.