Closed mihaiboldeanu closed 9 months ago
The background noise detection implemented in oceanstream
is based on the remove_noise
function from the echopype library, which in turn is based on the techniques described in the article by De Robertis & Higginbottom (2007), titled 'A post-processing technique to estimate the signal-to-noise ratio and remove echosounder background noise,' published in the ICES Journal of Marine Sciences (64(6): 1282–1291).
The parameters for oceanstream
's apply_remove_background_noise
function, designed to address background noise issues, are as follows:
ds_Sv
: xr.Dataset
ping_num
: int, default=40
range_sample_num
: int, optional
range_sample
dimension to obtain noise estimates. If not provided, it's computed based on a 10-meter vertical bin for each channel, and the minimum value is used.noise_max
: float, default=-125
SNR_threshold
: float, default=3 [dB]
The default values for the parameters in oceanstream
's apply_remove_background_noise
were chosen based on the default values tested in the article for calibrated Simrad EK60 echosounders operating at 18, 38, 120, and 200 kHz
:
Other important considerations highlighted in the article that should be taken into account when adjusting the input parameters for the apply_remove_background_noise
include:
SNR_threshold
be greater than 0.noise_max
represents an upper limit for background noise expected under the operating conditions. It must be determined empirically and will depend on the echosounder. In the examples presented in the article, noise_max
was set at -125 dB at all frequencies, and this value was never exceeded.Raw Dataset Details:
Parameters:
process_parameters = {
"remove_background_noise": {
"ping_num": 40,
"noise_max": -125,
"SNR_threshold": 3,
},
}
Proportion of Valid Samples Before Background Noise Removal:
Proportion of Valid Samples After Background Noise Removal:
Raw Dataset Details:
Parameters:
process_parameters = {
"remove_background_noise": {
"ping_num": 40,
"noise_max": -125,
"SNR_threshold": 0,
},
}
Proportion of Valid Samples Before Background Noise Removal:
Proportion of Valid Samples After Background Noise Removal:
Raw Dataset Details:
Parameters:
process_parameters = {
"remove_background_noise": {
"ping_num": 40,
"range_sample_num": 10,
"noise_max": -125,
"SNR_threshold": 3,
},
}
Proportion of Valid Samples Before Background Noise Removal:
Proportion of Valid Samples After Background Noise Removal:
Raw Dataset Details:
ncei-wcsd-archive
data/raw/Bell_M._Shimada/SH1707/EK60/
Summer2017-D20170620-T011027.raw
Parameters:
process_parameters = {
"remove_background_noise": {
"ping_num": 40,
"noise_max": -125,
"SNR_threshold": 3,
},
}
Proportion of Valid Samples Before Background Noise Removal:
Proportion of Valid Samples After Background Noise Removal:
Raw Dataset Details:
ncei-wcsd-archive
data/raw/Bell_M._Shimada/SH1707/EK60/
Summer2017-D20170620-T011027.raw
Parameters:
process_parameters = {
"remove_background_noise": {
"ping_num": 40,
"noise_max": -125,
"SNR_threshold": 0,
},
}
Proportion of Valid Samples Before Background Noise Removal:
Proportion of Valid Samples After Background Noise Removal:
Raw Dataset Details:
ncei-wcsd-archive
data/raw/Bell_M._Shimada/SH1707/EK60/
Summer2017-D20170620-T011027.raw
Parameters:
process_parameters = {
"ping_num": 40,
"noise_max": -125,
"SNR_threshold": 3,
},
}
Proportion of Valid Samples Before Background Noise Removal:
Proportion of Valid Samples After Background Noise Removal:
Conclusion:
After a thorough analysis, it has become evident that applying background noise removal is more effective after the interpolation step in the processed dataset. When the background noise removal function is applied to an unprocessed dataset, as demonstrated in the example above, approximately 36% of the data is invalidated. This significant loss of data underscores the importance of the sequence in which processing steps are applied.
Adjusting the threshold_SNR
parameter can affect the proportion of invalidated samples. Decreasing the threshold_SNR
leads to a reduction in invalidated samples; however, it is not advisable to set it below 0.
The parameter range_sample_num
, which determines the number of samples along the range_sample
dimension for noise estimation:
In the examples analyzed, different configurations of parameters were applied to a processed dataset. The percentage of invalidated data in these scenarios ranged between 5-17%, which is significantly lower than that observed in the unprocessed dataset. Based on these findings, it is recommended to use the default parameters for background noise removal in future analyses.
When applying background noise removal with the following parameters:
"remove_background_noise": { "ping_num": 40, "range_sample_num": 10, "noise_max": -125, "SNR_threshold": 3, }
We lose around 60-65% of data as noise. This should be investigated to make sure we arent eliminating to much useful data.