Use np.ndarray instead of mpiarray.MPIArray in MaskBaselines. This was causing an error in python 3.11+ because ufunc where is not defined for distributed arrays
Change the minimization method used by minimize_scalar in MaskFreq. The previous method golden would fail with newer versions of scipy because it could not find a valid bracket. I believe that this is because the function being minimized is roughly monotonically decreasing, so the criteria func(xb) < func(xa) and func(xb) < func(xc) could not be met. (See: https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize_scalar.html). I also tried using a standard scipyoptimize.minimize here. The only method which would converge properly was COBYLA, but it sometimes produced a slightly different mask even with appx. the same minimum at a 1e-4 tolerance.
Added a proper nsigma initial threshold for the sumthreshold algorithm in RFIStokesI and a variance estimate. Previously, the default threshold of the 95th percentile was being used, which causes problems when previous masks have been applied. The variance is calculated in a slightly wonky way, but it works well to apply sumthreshold over daytime data. A variance boost of 1e4 is added during solar transit, and max_m is increased for better masking.
Fixes a misnamed variable in sumthreshold
With these changes, we can run the pipeline with the newest version of python available on Cedar (3.11.5), which has a built module chime/python/2024.04
Some miscellaneous bug fixes:
np.ndarrayinstead ofmpiarray.MPIArrayinMaskBaselines. This was causing an error in python 3.11+ because ufuncwhereis not defined for distributed arraysminimize_scalarinMaskFreq. The previous methodgoldenwould fail with newer versions of scipy because it could not find a valid bracket. I believe that this is because the function being minimized is roughly monotonically decreasing, so the criteriafunc(xb) < func(xa) and func(xb) < func(xc)could not be met. (See: https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize_scalar.html). I also tried using a standardscipyoptimize.minimizehere. The only method which would converge properly wasCOBYLA, but it sometimes produced a slightly different mask even with appx. the same minimum at a1e-4tolerance.nsigmainitial threshold for thesumthresholdalgorithm inRFIStokesIand a variance estimate. Previously, the default threshold of the 95th percentile was being used, which causes problems when previous masks have been applied. The variance is calculated in a slightly wonky way, but it works well to applysumthresholdover daytime data. A variance boost of1e4is added during solar transit, andmax_mis increased for better masking.sumthresholdWith these changes, we can run the pipeline with the newest version of python available on Cedar (3.11.5), which has a built module
chime/python/2024.04