Closed emiliom closed 1 year ago
I've examined the current usage of frequency_start
and frequency_end
and the definitions of the corresponding transmit_frequency_start
and transmit_frequency_stop
in the convention. I have some observations and questions.
In the convention (both v1 and v2), the two variables are defined as Mandatory. In echopype, they are only used (present) with EK80, and within EK80 only for BB data and apparently assigned valid values only for a subset of channels. For example, in one EK80 file I've been examining (ek80/ek80_bb_with_calibration/2018115-D20181213-T094600.raw
), the frequency_start/end
variables are assigned valid values only in 2 of the 6 channels, while the other 4 are assigned nan
.
My interpretation is that these variables are simply unnecessary (redundant) for data other than BB data (though I don't understand the case I described above). In other words, for EK60, AZFP and EK80 power data, one could set transmit_frequency_start
= transmit_frequency_stop
= frequency_nominal
.
If that's correct, I recommend always creating these variables. That would be more consistent with the convention, since the variables are mandatory (their description is: "Frequency at the start/end of the transmit pulse."). If we do this, there'll be an additional change needed. Currently the presence of frequency_start
and/or frequency_end
in Beam_group1
is used as a test and decision point for further processing steps; in some cases, the test is for nan
values instead. For example:
https://github.com/OSOceanAcoustics/echopype/blob/ed79cd82332f177e063d1b1341bb9d3f1bb5f734/echopype/calibrate/calibrate_ek.py#L312-L315
If the variables are set to always be present, these checks would need to be updated. If we assign nan
values in cases where the variables are not applicable or used, it should be fairly straightforward to modified these checks. Assigning valid values in all cases (frequency_nominal
instead of nan
) seems more consistent, but the checks would then have to be done either on a different variable or based on transmit_frequency_start != transmit_frequency_stop
.
One last observation: the convention defines these variables as float
type, but echopype sets them as int
type. Both are defined in Hz units.
in one EK80 file I've been examining (
ek80/ek80_bb_with_calibration/2018115-D20181213-T094600.raw
), thefrequency_start/end
variables are assigned valid values only in 2 of the 6 channels, while the other 4 are assignednan
.
Yes, in this case 2 of the channels were set up to transmit BB signals and the other 4 set up to transmit CW signals.
for EK60, AZFP and EK80 power data, one could set
transmit_frequency_start
=transmit_frequency_stop
=frequency_nominal
.If that's correct, I recommend always creating these variables. That would be more consistent with the convention, since the variables are mandatory (their description is: "Frequency at the start/end of the transmit pulse."). If we do this, there'll be an additional change needed. Currently the presence of
frequency_start
and/orfrequency_end
inBeam_group1
is used as a test and decision point for further processing steps; in some cases, the test is fornan
values instead.
Ah this reminds me that this in fact is related to #951, which is related to data format. Under pulse_form
I had:
pulse_form
: right now in the code we use the presence offrequency_start/end
to check which ping/channel is transmitting narrowband or broadband signals; probably more convenient to usepulse_form
, but that could cause backward compatibility issues.
We can go two ways, check for transmit_frequency_start == transmit_frequency_stop
for the CW (True
) and BB (False
) cases. Or use pulse_form
to do the same thing. My preference is to use the latter, since this will be a breaking change of data format and it would be good to get things in good order.
One last observation: the convention defines these variables as
float
type, but echopype sets them asint
type. Both are defined in Hz units.
Oh interesting, I think maybe we could cast them to be float
type. I remember Kavin was resolving something on AZFP related to this that ended up being just the type cast issue (and he had to set things to float
for the computation to go through).
Thanks!! What's the data type or content for pulse_form
? Assuming it holds a set of known strings or simple int codes, it does sound like that'd be the clearest way to go.
It sounds like you're in agreement with creating transmit_frequency_start
and transmit_frequency_stop
in all cases and always populating them with valid, non-nan
values (eg, frequency_nominal
)?
Thanks!! What's the data type or content for
pulse_form
? Assuming it holds a set of known strings or simple int codes, it does sound like that'd be the clearest way to go.
I think it is 1 and 0 -- it'll be good to address this along with #951.
It sounds like you're in agreement with creating
transmit_frequency_start
andtransmit_frequency_stop
in all cases and always populating them with valid, non-nan
values (eg,frequency_nominal
)?
Yes!
I forgot to add one observation & question: It looks like in EK80 frequency_start/end
are ping-invariant, from the examples I've seen and this example parameter datagram:
https://github.com/OSOceanAcoustics/echopype/blob/ed79cd82332f177e063d1b1341bb9d3f1bb5f734/echopype/convert/parse_base.py#L197-L202
But the variables are created with a ping_time
dimension -- which matches the convention (both v1 & v2).
Should we remove the ping_time
dimension? It looks like we should.
No we should NOT remove the ping_time
dimension. This is something can change over time for EK80 broadband data.
Ok, thanks. In that case, we should probably add ping_time
to the variables in EK60 and AZFP, for consistency, right?
Ugh... I don't know.
For AZFP I kept the ping_time
dimension for transmit_duration_nominal
and sample_interval
mostly for the sake of consistency, and also for data from different deployment these may be different (these are configuration).
For transmit_frequency_start
and transmit_frequency_stop
, I feel they are a little like the property of the echosounder and are not configuration, so adding the ping_time
dimension seems confusing.
So, these 4 variables are all ping-invariant. transmit_duration_nominal
and sample_interval
just have an extra possibility of variation compared to transmit_frequency_start
and transmit_frequency_stop
because they can change with deployment.
Also, unlike transmit_duration_nominal
and sample_interval
, transmit_frequency_start
and transmit_frequency_stop
won't have any additional information not already present in frequency_nominal
. They are there truly just for consistency.
It seems like the same could be said for CW channels in EK80. In those cases:
ping_time
dimension already. So, the values to be inserted for CW channels will be replicated across ping_time
.Beam_group2
), currently these variables are not being created. Now that they will, we have to decide whether they have a ping_time
dimension. Having a Beam_group1
with transmit_frequency_start
and transmit_frequency_stop
variables having a ping_time
dimension and a Beam_group2
where they do not seems potentially messy.Yeah, EK80 CW mode is identical with EK60 in this sense, since it just transmits narrowband signals.
transmit_frequency_start
and transmit_frequency_stop
are properties (or one could say limit) of the echosounder itself when it comes to AZFP and EK60. They were not designed to be variable.
What do you recommend for EK80, then, for a Beam_group where all channels are CW?
ping_time
dimension. In that case, CW channels in one group may have a ping_time
dim if there are also BB channels, while in another group they won't.ping_time
dimension and replicate values. In that case, CW channels will be consistent across groups, at the cost of potential confusion to users (plus data bloat).Also, it sounds like you're recommend the same thing for EK60 as for AZFP (no ping_time), right?
I actually think the least confusing would be to not have transmit_frequency_start
and transmit_frequency_stop
in the first place for AZFP, EK60, and EK80 CW data, and only have those present for EK80 BB data.
If we want to strive for having the variables because they are mandatory in the convention, then I would vote for option 1 to omit the ping_time
dimension for AZFP, EK60, and EK80 CW data.
This is pretty messy, but I am not sure if I have better idea that do not make it more confusing.
To follow up from our conversation yesterday: we will include transmit_frequency_start
and transmit_frequency_stop
for all instruments and populate them with values (= frequency_nominal
for EK80 CW, EK60 and AZFP). The ping_time
dimension will be omitted for EK60, AZFP and EK80 where all channels are CW.
Yes, and just so that the logic is recorded here (and we should add that in the docs/ms too) is that the modification we are doing is to always have the mandatory (M) variables but the variable will only have the minimum dimensions required for each instrument/setting type (ie, the variable will always exist, but the dimensions may be different from the convention definition).
@emiliom I think you have a different/more precise way to say the second half, could you edit the above sentence?
Thanks. I'll edit the sentence later. I'll close this issue once I've edited your text.
Yes, and just so that the logic is recorded here (and we should add that in the docs/ms too) is that the modification we are doing is to always have the mandatory (M) variables but the variable will only have the minimum dimensions required for each instrument/setting type (ie, the variable will always exist, but the dimensions may be different from the convention definition).
@emiliom I think you have a different/more precise way to say the second half, could you edit the above sentence?
In echopype, our goal will be to always include mandatory (M) variables. However, for some instrument types, the dimensionality specified in the convention does not apply, or data may not be available at all. Our approach will be to reduce the dimensionality of variables to the minimum that is supported by the instrument type, thus reducing data duplication and bloat that can have a significant impact on data size and computational performance. As an example, the Sonar/Beam_groupX
beam_direction_x/y/z
variables are mandatory and originally defined in the convention as having dimensions of (ping_time, beam)
. But for the primary instruments currently supported by echopype (AZFP, EK60 and EK80), these variables don't vary along those dimensions. We will only assign them a channel
dimension, which is the dimension we introduced to support our gridded Sonar/Beam_groupX
data structure.
Beam_groupX
frequency_start
andfrequency_end
are only used by EK80. Rename them to the equivalent v1 convention names,transmit_frequency_start
andtransmit_frequency_stop