search

wmo-im / et-acdm

WMO Expert Team on Atmospheric Composition Data Management (ET-ACDM)
https://wmo-im.github.io/et-acdm
MIT License
3 stars 2 forks source link

Agree with each GAW SAG on level 0 data #23

Open joergklausen opened 7 months ago

joergklausen commented 7 months ago

to review the discussion about 'level 0 data' in GitHub (https://github.com/wmo-im/wmds/issues/477) and agree with each GAW domain (SAGs) what would be level 0 data, whether and where they need to be archived.

markusfiebig commented 3 months ago

Here is the definition WDCA and WDCRG are using: https://ebas-submit.nilu.no/templates/Integrating-Nephelometer-Data

This table is in fact valid for all types of data we are handling. There are adaptations possible for data types where the raw data stream has a very large volume, where some pre-compression is allowed.

tomkralidis commented 3 months ago

Definitions for WOUDC: https://guide.woudc.org/en/#chapter-3-standard-data-format

lakkala commented 1 month ago

From May 2024 @wmo-im/wmds/issues/477

"Status update from May 2024: A new proposal is drafted and submitted to the SC-MINT EdBd for follow up and includes a two step approach:

  1. to define Levels of Data for surface based and for satellite based data apart
  2. to merge these definitions later on to be valid for both type of data.

Proposed definitions for Level 0 data: Surface based data: The rawest and unprocessed output from an instrument or instrument transducer in native units (e.g. voltage) (sources WMO-No. 8, volume V & WMO IOM report 131, ‘SPICE’) Satellite based data: Instrument and auxiliary data reconstructed from satellite raw data after removing communications artefacts. [source: GIMO (WMO-No. 8), volume IV] "

atverm commented 1 month ago

The ICOS definition of Level 0 is:

Level 0=Raw data are information or objects directly obtained from human measurements or automated sensors that have not undergone any transformation. They may provide quantitative or qualitative information about physical variables of the environment and may be of various forms, such as images, text files or physical samples. Level 0 data are data in physical units either directly provided by instruments or converted from engineering units (for example, mV, mA, Ω) to physical units.

All ICOS raw data that is collected automatically from instruments is gathered into daily (compressed) archives per instrument that is sent without delay to the repository and minted a PID (based on the AES256 checksum) and linked to the metadata system. All data pipelines build on the full cycle from processing this raw data up to the NRT and final quality controlled data, fully reproducible and transparent.