Comments by @baptischmi

[peterdesmet]

Copied and adapted from Slack

thank you and well done! I especially like the addition of the data quality in terms of number of data points used for the main values of flight, dbd, and sd_vvp, and the 'vcp'. i have few minor comments - hope some are helpful:

• [x] To me it seems unnecessary to include both the u and v component, and the combined ground speed and direction. Why not simply having u, v, and w? it it because of back compatibility for other function that need 'ff' or 'dd'?
• [x] What is the angular data gap? is this information not included in the definition of NA or NaN for 'dens' or any other quantity?
• [x] 'radar_wavelength': Do you mean by "unknown" radar wavelength that vol2bird used a default setting, which is 5.3 and 10.6 for S and C-band, respectively?
• [x] 'radar_wavelength' inform on the processing of the data. i think information on rain filter may be also important, or which products fed vol2bird (DBZH, TH, VRADH, CCORH, RHOHV, PHIDP, ZDR). See #35
• [x] instead of having a column 'rcs', why not defining 'dens' as a multiplication of 'dbz' with a fixed rcs of 11 cm^2? same for sd_vvp_thredhold. 'dens' can be easily recalculated if these two must be modified.
• [x] define the value below/above the minimum/maximum allowed, e.g. 'u', 'v', 'dens'.
• [x] i like the proposed order of the variable. I particular, i like having 'sd_vpp' close to 'eta'. it make sense to first have all data related to the speed and direction, finish with sd_vvp, and then display the density/reflectivity values.

[peterdesmet]

@baptischmi what do you mean with:

define the value below/above the minimum/maximum allowed, e.g. 'u', 'v', 'dens'.

A minimum and maximum are defined for those three variables.

[adokter]

• u,v have historically been part of the vol2bird profile data, but I agree there is duplication with ff,dd. Currently only ff,dd are required, and u,v are optional, because the ff,dd pair is more easy to interpret. @peterdesmet any thoughts on whether duplication of data is something we want to avoid at all cost?
• angular data gap refers to field gap of vol2bird output - it's a boolean that states if there are too many angular sectors missing to do a velocity fit - not contained in the other quantities, it helps interpreting NA/NaN values.

  inform on processing of the data

• vpts format in its current form unfortunately doesn't support substantial amounts of metadata, which I agree is a pity and one of the largest limitations of this table format. In the past we have discussed adding a field that contains a reference to a separate table containing metadata values, but our decision was to leave that for a future iteration, as it involves a lot of extra data prep work. @peterdesmet should we track this feature with a separate issue?
• since eta = rcs * dens if sd_vpp>sd_vvp_threshold, there is duplication here again. It is more a matter of convenience to have this value present as a column, and since there is some (simple) algebra involved to obtain this often used quantity, I'm inclined to keep it. rcs as a column I think is useful, for when people want to include seasonal trends of rcs (potentially same for sd_vvp_threshold).

[peterdesmet]

any thoughts on whether duplication of data is something we want to avoid at all cost?

I don't think we have to avoid it at all costs. If many people use ff and dd, it's a welcome convenience. We could opt to make u and v the required terms. We could opt to remove ff and dd (but in that case we should list how to calculate ff and dd).

Created a separate issue for the metadata #35.

[adokter]

I prefer to keep ff and dd, as they are easier to interpret

[baptischmi]

@baptischmi what do you mean with:

define the value below/above the minimum/maximum allowed, e.g. 'u', 'v', 'dens'.

A minimum and maximum are defined for those three variables.

@peterdesmet For instance, speed 'ff' above 100 m/s are already far from being reliable, but which value is set, if for some processing artefacts, you get higher speed values higher than 100 m/s? would you set the value as NA, or as the maximum value allowed (i.e. 100 m/s).

PS: This comment not valid for 'dens' (as stated in my original message), for which the maximum is 'Inf', but rather only for 'u', 'v', 'w', 'ff'.

[bart1]

I just scanned the format definition. As the goal is stated quite general: VPTS is a community developed data exchange format for biological signals detected by weather radars. It seems it should be able to also possibly describe results from other possible identification algorithms. If that is less the goal this comment can be ignored otherwise I feel some of the required field are quite specific to vol2bird. For example sd_vvp and possibly dd and ff. It is not given those metrics are a result of each algorithm.

[adokter]

The goal should be to have it be general, and I think all the required fields will be output of any profiling algorithm, so some measure of speed (ff), direction (dd) and density (dens or dbz or eta) for each of the alitude layers. The only required field that is somewhat debatable as far as requirement is speed variability sd_vvp, but the profiling algorithms I'm aware of all provide it.

[bart1]

The goal should be to have it be general, and I think all the required fields will be output of any profiling algorithm, so some measure of speed (ff), direction (dd) and density (dens or dbz or eta) for each of the alitude layers. The only required field that is somewhat debatable as far as requirement is speed variability sd_vvp, but the profiling algorithms I'm aware of all provide it.

I guess most algorithms can indeed calculate proxies for these numbers, if you know most have something implemented I guess that is good

[peterdesmet]

For instance, speed 'ff' above 100 m/s are already far from being reliable, but which value is set, if for some processing artefacts, you get higher speed values higher than 100 m/s? would you set the value as NA, or as the maximum value allowed (i.e. 100 m/s).

@adokter do you have any advise for what values to use when artefacts result in higher than maximum values?

[adokter]

I would set it as the max allowed (100 m/s) - but the min/max thresholds are chosen such that adjustments like that shouldn't be necessary, or at least very rare. I checked a few years of data for several radars in the US, and I haven't seen ff go above 60 m/s.

[peterdesmet]

Do you mean by "unknown" radar wavelength that vol2bird used a default setting, which is 5.3 and 10.6 for S and C-band, respectively?

@baptischmi: No, I mean that if you do not know the wavelength, but know it is a C-band radar, to then set radar_wavelength values to 5.3. @adokter are wavelength values used by vol2bird when processing? Would they always be known in produced then?

[adokter]

Yes, wavelength is needed to calculate eta from reflectivity dBZ. It reads it from the raw file if available as metadata. If not, it assumes it's C-band with wavelength 5.3 cm

[peterdesmet]

@adokter any way you would rephrase this then (without making it too complicated)?

Wavelength of the radar in cm. If unknown, use 5.3 cm for C-band and 10.6 cm for S-band radar.

I propose:

Wavelength of the radar in cm. 5.3 cm is typically assumed for C-band radar and 10.6 cm for S-band radar.

[bart1]

@peterdesmet I was just thinking should the documentation indicate what columns are typically constant (within radar/ scan). e.g. radar latitude does not really change within radar. It is kind of obvious but other wise you might produce valid files that can not really be handled / processed further at least with bioRad

[peterdesmet]

@bart1 👍 updated in 1c9da40.

Constant for all records from the same radar.

[adokter]

@adokter any way you would rephrase this then (without making it too complicated)?

@peterdesmet slight tweak:

Wavelength of the radar in cm. Most C-band radars operate at approximately 5.3 cm wavelength, and most S-band radars at 10.6 cm

[peterdesmet]

Thanks, updated. And with that we have addressed all comments by @baptischmi listed at the start of this issue.