search

wmo-im / wis2box

WIS2 in a box is a reference implementation of a WMO WIS2 Node
https://docs.wis2box.wis.wmo.int
Apache License 2.0
40 stars 16 forks source link

csv2bufr to use data from station metadata when producing BUFR #763

Open maaikelimper opened 2 months ago

maaikelimper commented 2 months ago

In the training material for csv2bufr Dave originally wrote this section: 

!!! bug "Station metadata" 
    The CSV to BUFR conversion tool has been developed as a standalone tool separate from the wis2box.
    As such, all the data to be encoded in BUFR must be present in the CSV file, including the station metadata
    like the location and sensor heights. In future versions the web-application will be updated to load the metadata
    from the wis2box API and to insert the values into the CSV, similar to the process used in FM-12 SYNOP to BUFR.

I think we could implement some code to retrieve missing columns from the station metadata in the csv2bufr process in the wis2box-api plugin: https://github.com/wmo-im/wis2box-api/blob/main/wis2box_api/plugins/process/csv2bufr.py

@david-i-berry could you help me brainstorm how this could be done ?

david-i-berry commented 2 months ago

@maaikelimper we'd need to a join on wigos_station_identifier between https://github.com/wmo-im/wis2box-api/blob/3f2499828df38598a8c39982170a55386680087c/wis2box_api/plugins/process/csv2bufr.py#L163 and https://github.com/wmo-im/wis2box-api/blob/3f2499828df38598a8c39982170a55386680087c/wis2box_api/plugins/process/csv2bufr.py#L167, with the names updated, either in the csv2bufr mapping or in the object resulting from the join.

maaikelimper commented 2 months ago

The input-columns depend on the mapping being used. Each template might have different column-names to refer to the columns like 'latitude' 'longitude' 'altitude' 'barometer_height_above_sealevel' and we need to ensure the csv-string after the join matches the expected input-string for the template ... I think in principle we have all the info we need in the mapping-object from this line onwards: https://github.com/wmo-im/wis2box-api/blob/3f2499828df38598a8c39982170a55386680087c/wis2box_api/plugins/process/csv2bufr.py#L179

Do you have some suggested pseudo-code on how the mappings-object and the input csv could be used to provide a valid csv-string for the mapping template ?