This PR implements two improvements for the lanczos regridder and adds new tasks to combine the regridding and stacking processes.
Lanczos updates
Signal variance estimate
Rather than using a single scalar as an estimate of the gridded signal variance, this PR adds the option to use an actual diagonal signal covariance derived from another dataset. Testing shows that using a proper covariance estimate improves interpolation around bright point sources while also reducing excess noise and artifacts introduced when using an excessively high variance on quieter parts of the sky. This PR:
Adds a simple task to estimate the diagonal covariance from a sidereal stream (this is just the squared, mean-subtracted power with a windowed mean applied)
Adds a container to hold a band diagonal covariance dataset in sidereal time
Adds an optional signal_cov argument in the setup method of transform.Regridder and, if provided, uses this container to make the inverse signal covariance for the interpolation
Cleans up the transform.Regridder and sidereal.SiderealRegridder code a bit
Artifact masking
Using a proper signal covariance seems to reduce/remove the bulk of regridding artifacts, but there are still some residual artifacts at the edge of heavily masked areas. From testing, this seems to occur when the main lobe of the regridding kernel is centred on flagged time samples. The estimate of the gridded sample comes only from the kernel side lobes, where small numerical errors end up being significant. This PR addresses this by projecting the data mask through the absolute value of the gridding kernel and masking samples below some fractional threshold.
Combined regridding and stacking
This is still somewhat experimental, and I have not yet tested whether the resulting stack is better using this method instead of just using the standard quarterstacking.
Having more time samples present in the Wiener filter regridder should in theory produce a better estimate of the underlying signal. Rather than gridding each individual day and taking a weighted average, as is done in the standard quarterstack pipeline, this PR adds tasks to split the gridding into steps which produce
A noise-weighted projection of the time series onto the RA grid
A band-diagonal noise covariance matrix
The idea is to produce these two datasets for several days, sum them over all days, add an estimate of the gridded signal covariance, and then solve the deconvolution problem. This should have two advantages:
Fewer to no gaps in the data, reducing or eliminating gridding artifacts
A slightly better estimate of the underlying signal
In order to make this process computationally feasible, the weights (noise) dataset, and thereby the noise covariance, is factorized into a frequency-time array and a frequency-baseline array. This PR does the folllwing:
Adds containers for a factorized timestream and a partially gridded sidereal stream
Adds a task to factorize noise weights
Refactors the regrid.band_wiener code to separate the gridding process
Adds task to perform the "dirty" sidereal gridding, deconvolution, and stack-and-deconvolution processes
Adds a feature to core.io.Truncate to get the truncation weight dataset from a class property, allowing that dataset to be produced on the fly
This PR implements two improvements for the lanczos regridder and adds new tasks to combine the regridding and stacking processes.
Lanczos updates
Signal variance estimate
Rather than using a single scalar as an estimate of the gridded signal variance, this PR adds the option to use an actual diagonal signal covariance derived from another dataset. Testing shows that using a proper covariance estimate improves interpolation around bright point sources while also reducing excess noise and artifacts introduced when using an excessively high variance on quieter parts of the sky. This PR:
signal_covargument in thesetupmethod oftransform.Regridderand, if provided, uses this container to make the inverse signal covariance for the interpolationtransform.Regridderandsidereal.SiderealRegriddercode a bitArtifact masking
Using a proper signal covariance seems to reduce/remove the bulk of regridding artifacts, but there are still some residual artifacts at the edge of heavily masked areas. From testing, this seems to occur when the main lobe of the regridding kernel is centred on flagged time samples. The estimate of the gridded sample comes only from the kernel side lobes, where small numerical errors end up being significant. This PR addresses this by projecting the data mask through the absolute value of the gridding kernel and masking samples below some fractional threshold.
Combined regridding and stacking
This is still somewhat experimental, and I have not yet tested whether the resulting stack is better using this method instead of just using the standard quarterstacking. Having more time samples present in the Wiener filter regridder should in theory produce a better estimate of the underlying signal. Rather than gridding each individual day and taking a weighted average, as is done in the standard quarterstack pipeline, this PR adds tasks to split the gridding into steps which produce
In order to make this process computationally feasible, the weights (noise) dataset, and thereby the noise covariance, is factorized into a frequency-time array and a frequency-baseline array. This PR does the folllwing:
regrid.band_wienercode to separate the gridding processcore.io.Truncateto get the truncation weight dataset from a class property, allowing that dataset to be produced on the fly