Use np.ndarray instead of mpiarray.MPIArray in MaskBaselines. This was causing an error in python 3.11+ because ufunc where is not defined for distributed arrays
Change the minimization method used by minimize_scalar in MaskFreq. The previous method golden would fail with newer versions of scipy because it could not find a valid bracket. I believe that this is because the function being minimized is roughly monotonically decreasing, so the criteria func(xb) < func(xa) and func(xb) < func(xc) could not be met. (See: https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize_scalar.html). I also tried using a standard scipyoptimize.minimize here. The only method which would converge properly was COBYLA, but it sometimes produced a slightly different mask even with appx. the same minimum at a 1e-4 tolerance.
Added a proper nsigma initial threshold for the sumthreshold algorithm in RFIStokesI and a variance estimate. Previously, the default threshold of the 95th percentile was being used, which causes problems when previous masks have been applied. The variance is calculated in a slightly wonky way, but it works well to apply sumthreshold over daytime data. A variance boost of 1e4 is added during solar transit, and max_m is increased for better masking.
Fixes a misnamed variable in sumthreshold
With these changes, we can run the pipeline with the newest version of python available on Cedar (3.11.5), which has a built module chime/python/2024.04
Some miscellaneous bug fixes:
np.ndarray
instead ofmpiarray.MPIArray
inMaskBaselines
. This was causing an error in python 3.11+ because ufuncwhere
is not defined for distributed arraysminimize_scalar
inMaskFreq
. The previous methodgolden
would fail with newer versions of scipy because it could not find a valid bracket. I believe that this is because the function being minimized is roughly monotonically decreasing, so the criteriafunc(xb) < func(xa) and func(xb) < func(xc)
could not be met. (See: https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize_scalar.html). I also tried using a standardscipyoptimize.minimize
here. The only method which would converge properly wasCOBYLA
, but it sometimes produced a slightly different mask even with appx. the same minimum at a1e-4
tolerance.nsigma
initial threshold for thesumthreshold
algorithm inRFIStokesI
and a variance estimate. Previously, the default threshold of the 95th percentile was being used, which causes problems when previous masks have been applied. The variance is calculated in a slightly wonky way, but it works well to applysumthreshold
over daytime data. A variance boost of1e4
is added during solar transit, andmax_m
is increased for better masking.sumthreshold
With these changes, we can run the pipeline with the newest version of python available on Cedar (3.11.5), which has a built module
chime/python/2024.04