confirm format of INBO test data

DOV-Vlaanderen / groundwater-logger-validation

Analysis on validation methods for groundwater logger data

MIT License

2 stars 2 forks source link

confirm format of INBO test data #1

Closed fredericpiesschaert closed 5 years ago

fredericpiesschaert commented 5 years ago

@DavorJ a first batch of test files is available at https://github.com/DOV-Vlaanderen/groundwater-logger-validation/tree/master/data/raw/inbo please check if the format is convenient. More details in https://github.com/DOV-Vlaanderen/groundwater-logger-validation/blob/master/data/README.md

DavorJ commented 5 years ago

Format is OK for me. We will have to discuss it though, since csv-files depend on encoding, delimitation, timestamp-format, etc.

DavorJ commented 5 years ago

Encoding: UTF-8 Delimiter: , Decimal: , Quotes: " Timestamp: 5/06/2009 12:00:00 +01:00

df <- data.table::fread(logger_path, dec = ",")
df[, DRME_OCR_UTC_DTE := as.POSIXct(gsub("(.*):", "\\1", DRME_OCR_UTC_DTE),
                                    format = "%d/%m/%Y %H:%M:%S %z", tz = 'UTC')]

Data set content:

Primary key: MEASUREMENT_ID
UTC timestamp: TIMESTAMP_UTC
Value: PRESSURE_VALUE
Units: PRESSURE_UNIT

File convention name: LOCATIONID_SERIALNUMBER.csv