Closed ajutila closed 4 years ago
Patched in the awi_fix_surface branch.
Smoothing is now available to deal with outliers. It uses a median filter with a prescribed window. Seems to handle the outlier end traces and NaN traces (with additional handling) https://github.com/kingjml/pySnowRadar/blob/591df7c588b571594e4f745f94d939937857ba58/pySnowRadar/snowradar.py#L191-L192
For now we need additional handling where the entire trace is nan. The exception triggers infilling of the radar data with 0 values. This does not modify the snow radar object and must be used in conjunction with the smoothing to create a decent surface. It will displace a warning when this happens. https://github.com/kingjml/pySnowRadar/blob/591df7c588b571594e4f745f94d939937857ba58/pySnowRadar/snowradar.py#L191-L192
This is all a bit too temporary and needs to be addressed in the long term as part of https://github.com/kingjml/pySnowRadar/issues/2
I'd rather not roll these into master until the QA is finalized.
Great, this indeed partly fixes the special surface issues with AWI data. Unfortunately, there's (at least) a third case, where due to descend/ascend of the plane the surf_bin
at some part of the frame reaches fast time bin values where some of the traces are NaNs. I dare you @kingjml to try the frame 20170410_01_002/010, if you still have them. I haven't noticed this earlier, as my hack approach of using elevation compensation prior to choosing the subset with a threshold mostly got it decently right.
Huh, and I guess smoothing doesn't do really well if the weird surf_bin
values are too close to the start/end of the frame? Case in point: Data_img_01_20190410_03_009.mat from the TVC flight, although that frame is otherwise really bad example due to really noisy signal (plane turning around).
Long term solution for this is to change the way processing.py handles individual radar traces. This will require it to drop np.nan values and evaluate only the valid data. The problem here is that we need a method to track how we are manipulating the arrays so that the picker index returns make sense. We'll tackle this before the alpha is done.
Masking might be the key.
@kingjml & @m9brady, something that I noticed this afternoon. I guess the null space removal in the function get_bounds
triggered by the m_above = None
option won't work properly.
https://github.com/kingjml/pySnowRadar/blob/5a2100d4f2b97ac6e98e3006ecb05f0c966a993e/pySnowRadar/snowradar.py#L187-L208
That's because of zero-patching all-NaN traces in get_surface
introduced in awi_fix_surface branch
https://github.com/kingjml/pySnowRadar/blob/591df7c588b571594e4f745f94d939937857ba58/pySnowRadar/snowradar.py#L176-L196
That got me thinking, could we then just ignore the zero surf_bin
values when assigning the bounds? Just simply
null_lower = self.surf_bin[self.surf_bin > 0].max() + (m_below / self.dfr).astype(int)
null_upper = self.surf_bin[self.surf_bin > 0].min() - (m_above / self.dfr).astype(int)
This should cover many cases, if not all? Or is this too simple to work? As it seems that backflipping (pun intended) nyquist zone crossings in connection to the EM-Bird ascends/descends in AWI data might cause bigger all-NaN patches, smoothing surf_bin
in get_surface
won't work anymore either. So going from this (elevation compensated, but not nyquist zone backflipped)
via nyquist zone backflipping (not elevation compensated yet)
to this (now it's both elevation compensated and nyquist zone backflipped)
Of course the picker still doesn't like NaN data...
(Oh and for the time being ignore the radar data quality on the nyquist zone backflipped bit. I guess it should be flipped before coherent noise removal and deconvolution, so yay...)
But in essence, would this make sense? If/when switching to dynamic handling of individual radar traces this becomes obsolete, right?
This emails an AWI issue with calibration flights. Will move to close as a general issue.
AWI data in its current state includes All-NaN traces and random traces where the last value has the highest value. That screws up the function
get_surface
, which in turn messes upget_bounds
. Therefore theradar_subset
has wrong boundaries, which allows radar data with NaN values to emerge, which in turn ends up ruining the interface picker.So far my hack has been using the elevation compensation, assigning 0 values for NaNs
surf_bin = np.nanargmax(np.nan_to_num(self.data_radar), axis=0)
, and finding the approximate modal value of the surface bin and using a threshold around it for boundaries