search

cioos-atlantic / atlantic-hurricane-dashboard

1 stars 1 forks source link

Harmonize station data queries upon fetching from WFS #25

Open sjbruce opened 2 days ago

sjbruce commented 2 days ago

Station data queries bring in chunks of data that need to be processed before being useful to the application.

Due to each station dataset being slightly different in terms of variables and frequency, each station data record has a single large text field that holds all the station data for that particular chunk. Chunks can cover several hours of data, this may be only a few records for some stations or many depending on the station sampling frequency. In addition, variable names are not consistent from station to station for many reasons.

After fetching data from WFS but before returning it to the application proper it should be decoded to a JSON object with variable names, standard names, long names, units and a table of data.

Example: 

{
  "table": {
    "columnNames": [<array of field names>],
    "columnTypes": [<array of field data types>],
    "columnUnits": [<array of field units>],
    "columnStdNames": [<array of standard names>],
    "columnLongNames": [<array of long names>],
    "rows": [
      [<array of matching data fields>],
      [<array of matching data fields>],
      ....
     ]
}
JaredMclellan commented 1 day ago

Created new branch and API page for the updated format. Stores the JSON object within the existing object properties in the station_data field. Combines the bin data values together (do we want to keep separate, wasn't sure?) Also wasn't sure if the extra information from properties (min/max time, lat/lon, station name, etc.) was needed but better safe than sorry. Can now also specify station in the request to return only one station's data. By default it returns all station data like before

https://github.com/cioos-atlantic/atlantic-hurricane-dashboard/blob/harmonize_station_data/react/pages/api/erddap_stations.js