stufraser1
commented
6 months ago .bin is not an available file type in data format codelist for describing resource files https://docs.riskdatalibrary.org/en/latest/reference/codelists/#data-formats I vaguely recall discussion of this but cannot find reference to it in an issue. .bin files are a crucial part of describing Oasis LMF files, lets add it if there is not good justification to omit it.
RDL hazard should be used for more than just return period hazard maps / scenario footprints. Central Asia data upload showed some limitations in RDL hazard metadata in describing items such as stochastic catalog, historical track/faults, etc. -- data that is used to develop the return period hazard maps, and should be made available to users.
Similar challenges are posed when considering Oasis hazard files. See below for files and description.
In DDH, a dataset could describe the full set of hazard files, with RDL metadata describing the hazard type, process type, location, intensity measure, etc. Within that dataset each file could be added as a resource. This should suffice, but is there further description in metadata needed/beneficial which states the specific type of data being stored, beyond 'stochastic', 'empirical' calculation method?
Here is an outline of the Oasis hazard module files with references to PiWind, our toy model.
Some examples of csv formats of required Oasis hazard module files are;
events_p.csv – a list of event_ids for the historical or stochastic events. occurrence_lt.csv - defines the occurrence of event_ids within a timeline of years and the calendar dates of those occurrences - used to calculate EP loss curve and AALs. footprint.csv – the main hazard footprint https://github.com/OasisLMF/OasisPiWind/tree/develop/model_data/PiWind
These files are converted to the following special-purpose compressed binary formats which are needed for model execution in the oasis platform; events.bin (list of events) occurrence.bin (list of event occurrences) footprint.bin.z (main hazard footprint) footprint.idx.z (index file of the position of each event in the footprint.bin.z, for fast reading)
The tools to convert the csv files and the file requirements are explained here; https://github.com/OasisLMF/ktools/blob/develop/docs/md/DataConversionComponents.md
The required reference file for the hazard module areaperil_dict.csv - this file defines what each spatial area (represented by areaperil_id) is in the footprint. It could list lat-lon points, lat-lon vertices bounding a model cell, or administrative zones for example.
The format is entirely flexible and decided by each model provider. For example parquet, geotiff and pickle formats have also been used.
Example: https://github.com/OasisLMF/OasisPiWind/tree/develop/keys_data/PiWind
Finally, optional files; Intensity_bin_dict.csv defines the intervals of hazard intensity for the intensity bins in the footprint file. Event_dict.csv gives descriptions of the event_ids. Any other file may be provided as reference data for explaining the model data files
Note that there is no naming convention or dependency on these files so they don’t need to be present in the model files for model execution purposes.
Meta data is specified in json format. Example https://github.com/OasisLMF/OasisPiWind/blob/develop/meta-data/model_settings.json
This file is functionally required for Oasis model execution, to describe the supported peril codes and which events.bin and occurrence.bin files may be used. It doesn’t contain general meta data describing the hazard, such as IMT and units, etc