As we'll have larger amounts of data with the newer dashboards we could do with being able to retrieve data depending on what the user wants to look at. Having run some tests it looks like writing the data out to a separate file for each ObsData object returned by OpenGHG works quite well. These are standard JSON files that are accompanied by a separate JSON file that provides a lookup table for them and all the metadata required to build the interface for the site.
NOTE: I'll use the term source here to mean an inlet at a site for a specific species
My idea is to:
build the interface from the exported metadata file (containing filenames and metadata)
implement dynamic retrieval of data based on source selection on the dashboard
retrieve some data on first load so an initial plot is created
retrieve the data each time they click on a new source
look at gzip compression, this seems to perform favourably when compared with parquet, reducing the data to around 1/4 of the size of the exported JSON.
As we'll have larger amounts of data with the newer dashboards we could do with being able to retrieve data depending on what the user wants to look at. Having run some tests it looks like writing the data out to a separate file for each
ObsDataobject returned by OpenGHG works quite well. These are standard JSON files that are accompanied by a separate JSON file that provides a lookup table for them and all the metadata required to build the interface for the site.My idea is to: