Data output layout

[truth-quark]

Task: decide on a "layout" for the NetCDF output files. The purpose of this repo is to convert project data to NetCDF files. This needs a common layout. For data (e.g. CAMS....), we have:

• 21 different layers (flags)
• 365 - 366 days

Layout can be one of:

• Each NetCDF has 1 data layer for the entire year (365-366 bands)
• Each NetCDF has 21 data layers for 1 day
• Each NetCDF has 21 data layers for 1 month
• Each NetCDF has 21 data layers for 6 months???
• Each NetCDF has 21 data layers for the entire year (365-366 bands)

TODO: Can these be created from NetCDF tools or should it be done from the project code?

Also, have a look at NetCDF compression to save space: https://gdal.org/drivers/raster/netcdf.html COMPRESS=DEFLATE and ZLEVEL=9 Z level is 1-9, 9 being high compression

[truth-quark]

Start with getting the NetCDF sorted 1st, then deal with metadata etc. Rough task list:

• Check if the scripts in the root dir can do the job
  • NB: contains GeoTIFF functionality, SKIP this!
• Otherwise, experiment with combining data layers using nco/NetCDF commands
  • Do the NetCDF tools allow subsetting & merging variables?
• Try to find a good balance of file size, # variables per file, time duration per file, total # of output files for dataset
• Sort out compression of the data (flag layers should be substantially compressed on account of 1 bit per pixel)
• Is any data cleaning required?
• Is more metadata required? Is the data CF compliant? (later)