ICOADS attachments and CDM ``original_values``

[ludwiglierhammer]

This issue is a continuation of #66.

In this table you can see from which ICOADS source original_values for the mapping to the CDM originate.

source	observations-at	observations-dpt	observations-slp	observations-sst	observations-wbt	observations-wd	observations-ws	attatchment	comment
icoads_r3000	(core, AT) | (core, DPT) | (core, SLP) | (core, SST) | (core, WBT) | (core, D) | (core, W)	not available	✔️ seems good
icoads_r3000_NRT	(core, AT) | (core, DPT) | (core, SLP) | (core, SST) | (core, WBT) | (core, D) | (core, W)	not available	✔️ seems good
icoads_r3000_d701	(c99_data, air_temperature) | (core, DPT) ⚠️ not in attachment | (c99_data, baro_pressure) | (c99_data, sea_surface_temperature) | (core, WBT) ⚠️ not in attachment | (core, D); ⚠️ what about one of (c99_data, wind_dir_start), (c99_data, wind_dir_middle), (c99_data, wind_dir_later)? | (core, W); ⚠️ what about one of (c99_data, wind_force_start), (c99_data, wind_force_middle), (c99_data, wind_force_later)?| c99_data	⚠️ data sources are not consistent; some values could be taken from attachment
icoads_r3000_d702	⛔ not defined; original_units are taken from attachment | (core, DPT) | ⛔ not defined; original_units are taken from attachment | (core, SST) | (core, WBT) | (core, WD) | (core, WS)| c99_data | ⛔ attachment does not contain any values; for consistency reasons, take all original_values from core
icoads_r3000_d704	(core, AT); ⚠️ could be taken from (c99_data4, air_temperature) | (core, DPT) ⚠️ not in attachment | (core, SLP);⚠️ could maybe be taken from (c99_data4, barometer) | (core, SST);⚠️ could be taken from (c99_data4, sea_temperature) | (core, WBT);⚠️ could be taken from (c99_data4, wet_bulb_temperature) | (core, D);⚠️ could be taken from (c99_data, wind_direction) | (core, W);⚠️ could be taken from (c99_data4, wind_force) | c99_data4, c99_data5	⚠️ some values could be taken from attachment; but then data sources are not consistent
icoads_r3000_d705-707	(core, AT) | (core, DPT) | (core, SLP) | (core, SST) | (core, WBT) | (core, D) | (core, W) | c99_data without values	✔️ seems good
icoads_r3000_d714	(c99, Airtemp) | (core, DPT); ⚠️ not in attachment | (c99, Pressure) | (c99, SST) | (core, WBT); ⚠️ not in attachment | (c99, Wdir) | (c99, WSp) | c99	⚠️ data sources are not consistent
icoads_r3000_d721	(c99_data, air_temperature) | core, DPT; ⚠️ not in attachment | (c99_data, atm) | (c99_data, sea_temperature) | (core, WBT); ⚠️ not in attachment | (core, D); ⚠️ could be taken from (c99_data, wind_dir) | (core, W); ⚠️ what about one of (c99_data, wind_speed) or (c99_data, wind_force) | c99_data	⚠️ data sources are not consistent ; some values could be taken from attachment
icoads_r3000_d730	(c99_data, AT_outside) | (core, DPT); ⚠️ not in attachment | (core, SLP); ⚠️ could be taken from (c99_data, AP) | (c99_data, SST) | (core, WBT); ⚠️ not in attachment | (core, D); ⚠️ not in attachment | (core, W); ⚠️ not in attachment | (c99_data)	⚠️ data sources are not consistent; some values could be taken from attachment
icoads_r3000_d781	(c99_data, air_temp) | (core, DPT); ⚠️ not in attachment | (c99_data, slp) | (c99_data, sst) | (c99_data, wet_bulb_temperature) | (c99_data, wind_dir) | (c99_data, wind_speed) | c99_data	⚠️ data sources are not consistent

@aanderss, @lizkent, @rcornes: Here are some questions listed regarind this table:

• As you can see, original_value is taken from both core and the attachment for some sources. How should we handle those inconsistencies?
• From where comes the data if not available in the attachment?
• Why are some original_values are taken from core even though data is availabale in the attachment?
• Why is original_value not defined for observations-at and observations-slp in icoads_d702?

I am looking forward to your comments 😃

[lizkent]

Hi,

This would need to be looked at on a case by case basis. Normally the value in the core (or other standard attachment) would be that judged to be the best/most appropriate during the original translation process. Often ICOADS wrote the original data to the c99 attachment and then populated the core and other attachments from that. So mostly I would expect where information was available in both it should be the same in both.

Some data sources have additional data/metadata, or data with additional precision that couldn’t be accommodated by the core section of the format, so in these cases the information can be extracted from c99 (or other attachments, for example the oceanographic attachment has higher resolution sst, and the RV attachment accommodates higher resolution locations).

The dck specific configurations were generated where there was significant information in the c99 free format attachment for that data source that was not available in the core or one of the other standard attachments. This would have been reviewed at the time, but of course it might be sensible to take another look at some point.

So in summary I expect that in most cases where something is available in both the core (or another standard attachment) and c99, then the core/standard value will be chosen. In most cases they should agree. In cases where there was information in both, and c99 was selected, that should have been done after looking at the documentation (e.g. extra precision in c99).

If there are specific cases that don’t seem to fit that picture, then please let me know and I can dig into the documentation.

Thanks, Liz

[ludwiglierhammer]

@lizkent: Thanks for your reply. I'll have a deeper look at it after my vacation 🌊