localdevices / pyorc

Surface velocity, object tracking, and river flow measurements in an open-source API
GNU Affero General Public License v3.0
134 stars 32 forks source link

filter for areas with too little samples #70

Closed hcwinsemius closed 1 year ago

hcwinsemius commented 2 years ago

Idea to base this on a quantile of nr of available samples, estimated from locations where number of values is not zero.

Pseudo-code

# count the amount of valid samples after filtering
sample_count = np.isfinite(ds_filt["v_x"]).sum(dim="time")
# turn into a flattened array
count_array = sample_count.values.flatten()
# find the quantile that is treated as non reliable after removing zero-sample areas (0.25)
count_array = count_array[count_array>0]
threshold = np.quantile(count_array, 0.25)

# apply sample threshold and return
ds_copy = copy.deepcopy(ds_filt)
ds_copy["v_x"] = ds_filt["v_x"].where(sample_count > threshold)
ds_copy["v_y"] = ds_filt["v_y"].where(sample_count > threshold)

return ds_copy