Closed ortk95 closed 1 month ago
Could do locally derived value fixing with something along the lines of
def clean(img: np.ndarray) -> np.ndarray:
nans = np.isnan(img)
cleaned = img.copy()
cleaned[nans] = np.nanmedian(img)
to_fix = nans & ~scipy.ndimage.uniform_filter(nans, size=3)
for i, j in np.argwhere(to_fix):
cleaned[i, j] = np.nanmean(img[max(i - 1, 0) : i + 2, max(j - 1, 0) : j + 2])
return cleaned
griddata
clearly performs best at keeping original variation with no influence from NaNs, so think we should use that as the default method. Can keep RectBivariateSpline
as an option for e.g. if smoothing is wanted.
From futher testing, griddata
doesn't seem as great:
I think this probably therefore rules out griddata
, as e.g. we don't want unstable mapping results to be interpreted as changes in the target etc.
Scipy's RegularGridInterpolator
seems to perform really nicely (and quickly) when paired with our custom _replace_nans_with_interpolated_values
, so this is probably the way to go.
Currently, we use
RectBivariateSpline
, so have to replace NaN pixels with some value before interpolating (currently just set to 0). Even though we later mask any projected NaN pixels (with_should_propagate_nan_to_map
), this can produce edge effects near NaN pixels when using e.g. cubic splines.It would make more sense therefore to either:
nanmean
of 3x3 footprint around each pixel. This would cost more performance, but provide better results for variable data.bisplrep
orgriddata
.