stufraser1
commented
3 years ago @ldodds best placed to advise
Closed matamadio closed 3 years ago
stufraser1
commented
3 years ago @ldodds best placed to advise
matamadio
commented
3 years ago Please refer to the first post, which I'm updating and commenting, for review. Elements have been collapsed into dropdown for easier reading. Will add example pictures.
This is some element to discuss
This is the comment
pzwsk
commented
3 years ago Agree to discuss that with @ldodds after the explanatory phase - and also in line with MVPs.
In the meantime, useful documentation to read: https://standards.theodi.org/introduction/types-of-open-standards-for-data/ https://blog.ldodds.com/2020/10/14/tip-for-improving-standards-documentation/
Best
matamadio
commented
3 years ago The quickest reference to attributes would be something like this: https://docs.google.com/document/d/1G6dbuB0L7JNTCjrkC7oozMEP8hALfjADCnbEdR17-y8/edit
Proposal (based on today meeting):
matamadio
commented
3 years ago Added file naming convention to docs draft, adapted from latest proposal in discussion document.
To help univocally identify the content of a dataset, the filename must summarise all the key information that allow to distinquish it from the others. The general format, all in lower caps, is:
`[component_code]-{project_name}-[country_iso]-{schema_specifics}-{time}`The name is made of [required] and {optional} attributes. Each component uses the most relevant attribute as schema_specifics, for example:
hzd-[country_iso]-{project_name}-{hazard_type}-{process_type}-{hazard_trigger}-{frequency}-{time} Example: pluvial flood hazard scenario with return period 10 years in 2050 for Afghanistan is named hzd-afg-mhra-fl-fpf-rp10-2050
exp-[country_iso]-{project_name}-{occupancy}-{exposure_model}-{time} Example: residential exposure in Madagascar from Open Street Map 2015 is named exp-mdg-swio_rafi-residential-osm-2015
vln-[country_iso]-{project_name}-{hazard_type}-{occupancy}-{vulnerability_model} Example: flood depth-damage function developed for India by JRC over industrial land cover is named vln-ind-fl-industrial-jrc
lss-[country_iso]-{project_name}-{hazard_type}-{occupancy}-{time} Example: eartquake losses over Madagascar infrastructures over the period 1920-2012 is named lss-mdg-eq-infrastructrure-1920_2012
stufraser1
commented
3 years ago clarification on lss-[country_iso]-{project_name}-{hazard_type}-{occupancy}-{vulnerability_model}-{time} Example: eartquake losses over Madagascar infrastructures over the period 1920-2012 is named lss-mdg-eq-infrastructrure-1920_2012
{vulnerability_model} is not included in the example. Actually I think its not necessary to include {vulnerability_model} in the loss naming convention (unlikely use case that losses are produced with different V models for same hazard under the same project) {project} is missing form the example name though.
matamadio
commented
3 years ago Superceeded by discussion on https://docs.google.com/document/d/1Q6rNo6hky1yTS9DOzpqPWcClLQBLHXUSnj4fKjR6K-Q
Two separate sections under DATA SCHEMAS:
A) User guidelines B) Schema as a DB
A) User guidelines or Schema adoption guidelines
---WIP---
A specific list of attributes (metadata) is defined to describe the content of datasets for each the four components of the schema. Fields are marked with * are considered "mandatory" for the consistency of the whole schema.
0. Shared attributes
To ensure consistency, some attributes are referred using the same code along the whole schema.
The codes are explained below:
Schema component
| **code** | **shortcode** | **name** | |---|:---:|---| | HAZ | H | Hazard | | EXP | E | Exposure | | VUL | V | Vulnerability | | LOS | L | Loss |Data license
| **code** | **name** | **description** | **url** | |---|---|---|---| | **CC0** | Creative Commons CCZero (CC0) | Dedicate to the Public Domain (all rights waived) | https://creativecommons.org/publicdomain/zero/1.0/ | | **CC BY 4.0** | Creative Commons Attribution 4.0 (CC-BY-4.0) | | https://creativecommons.org/licenses/by/4.0/ | | **CC BY-SA 4.0** | Creative Commons Attribution Share-Alike 4.0 (CC-BY-SA-4.0) | | http://creativecommons.org/licenses/by-sa/4.0/ | | **ODbL** | Open Data Commons Open Database License (ODbL) | Attribution-ShareAlike for data(bases) | https://opendatacommons.org/licenses/odbl/summary/ | | **ODC-By** | Open Data Commons Attribution License(ODC-BY) | Attribution for data(bases) | https://opendatacommons.org/licenses/by/summary/ |Country ISO codes
| **code** | **name** | |:---:|---| | ABW | Aruba | | AFG | Afghanistan | | AGO | Angola | | AIA | Anguilla | | ALB | Albania | | AND | Andorra | | ARE | United Arab Emirates | | ARG | Argentina | | ARM | Armenia | | ASM | American Samoa | | ATG | Antigua and Barbuda | | AUS | Australia | | AUT | Austria | | AZE | Azerbaijan | | BDI | Burundi | | BEL | Belgium | | BEN | Benin | | BFA | Burkina Faso | | BGD | Bangladesh | | BGR | Bulgaria | | BHR | Bahrain | | BHS | The Bahamas | | BIH | Bosnia and Herzegovina | | BLM | Saint-Barthélemy | | BLR | Belarus | | BLZ | Belize | | BMU | Bermuda | | BOL | Bolivia | | BRA | Brazil | | BRB | Barbados | | BRN | Brunei | | BTN | Bhutan | | BWA | Botswana | | CAF | Central African Republic | | CAN | Canada | | CHE | Switzerland | | CHL | Chile | | CHN | People's Republic of China | | CIV | Ivory Coast | | CMR | Cameroon | | COD | Democratic Republic of the Congo | | COG | Republic of the Congo | | COK | Cook Islands | | COL | Colombia | | COM | Comoros | | CPV | Cape Verde | | CRI | Costa Rica | | CUB | Cuba | | CUW | Curaçao | | CYM | Cayman Islands | | CYP | Cyprus | | CZE | Czech Republic | | DEU | Germany | | DJI | Djibouti | | DMA | Dominica | | DNK | Denmark | | DOM | Dominican Republic | | DZA | Algeria | | ECU | Ecuador | | EGY | Egypt | | ERI | Eritrea | | ESP | Spain | | EST | Estonia | | ETH | Ethiopia | | FIN | Finland | | FJI | Fiji | | FLK | Falkland Islands | | FRA | France | | FRO | Faroe Islands | | FSM | Federated States of Micronesia | | GAB | Gabon | | GBR | United Kingdom | | GEO | Georgia | | GGY | Guernsey | | GHA | Ghana | | GIB | Gibraltar | | GIN | Guinea | | GMB | The Gambia | | GNB | Guinea-Bissau | | GNQ | Equatorial Guinea | | GRC | Greece | | GRD | Grenada | | GRL | Greenland | | GTM | Guatemala | | GUM | Guam | | GUY | Guyana | | HKG | Hong Kong | | HMD | Heard Island and McDonald Islands | | HND | Honduras | | HRV | Croatia | | HTI | Haiti | | HUN | Hungary | | IDN | Indonesia | | IMN | Isle of Man | | IND | India | | IRL | Ireland | | IRN | Iran | | IRQ | Iraq | | ISL | Iceland | | ISR | Israel | | ITA | Italy | | JAM | Jamaica | | JEY | Jersey | | JOR | Jordan | | JPN | Japan | | KAZ | Kazakhstan | | KEN | Kenya | | KGZ | Kyrgyzstan | | KHM | Cambodia | | KIR | Kiribati | | KNA | Saint Kitts and Nevis | | KOR | South Korea | | KWT | Kuwait | | LAO | Laos | | LBN | Lebanon | | LBR | Liberia | | LBY | Libya | | LCA | Saint Lucia | | LIE | Liechtenstein | | LKA | Sri Lanka | | LSO | Lesotho | | LTU | Lithuania | | LUX | Luxembourg | | LVA | Latvia | | MAC | Macau | | MAF | Saint Martin | | MAR | Morocco | | MCO | Monaco | | MDA | Moldova | | MDG | Madagascar | | MDV | Maldives | | MEX | Mexico | | MHL | Marshall Islands | | MKD | Republic of Macedonia | | MLI | Mali | | MLT | Malta | | MMR | Myanmar | | MNE | Montenegro | | MNG | Mongolia | | MNP | Northern Mariana Islands | | MOZ | Mozambique | | MRT | Mauritania | | MSR | Montserrat | | MUS | Mauritius | | MWI | Malawi | | MYS | Malaysia | | NAM | Namibia | | NCL | New Caledonia | | NER | Niger | | NFK | Norfolk Island | | NGA | Nigeria | | NIC | Nicaragua | | NIU | Niue | | NLD | Netherlands | | NOR | Norway | | NPL | Nepal | | NRU | Nauru | | NZL | New Zealand | | OMN | Oman | | PAK | Pakistan | | PAN | Panama | | PCN | Pitcairn Islands | | PER | Peru | | PHL | Philippines | | PLW | Palau | | PNG | Papua New Guinea | | POL | Poland | | PRI | Puerto Rico | | PRK | North Korea | | PRT | Portugal | | PRY | Paraguay | | PSE | Palestine | | PYF | French Polynesia | | QAT | Qatar | | ROU | Romania | | RUS | Russia | | RWA | Rwanda | | SAU | Saudi Arabia | | SDN | Sudan | | SEN | Senegal | | SGP | Singapore | | SGS | South Georgia and the South Sandwich Islands | | SHN | Saint Helena | | SLB | Solomon Islands | | SLE | Sierra Leone | | SLV | El Salvador | | SMR | San Marino | | SOM | Somalia | | SPM | Saint Pierre and Miquelon | | SRB | Serbia | | SSD | South Sudan | | STP | São Tomé and Príncipe | | SUR | Suriname | | SVK | Slovakia | | SVN | Slovenia | | SWE | Sweden | | SWZ | eSwatini | | SXM | Sint Maarten | | SYC | Seychelles | | SYR | Syria | | TCA | Turks and Caicos Islands | | TCD | Chad | | TGO | Togo | | THA | Thailand | | TJK | Tajikistan | | TKM | Turkmenistan | | TLS | East Timor | | TON | Tonga | | TTO | Trinidad and Tobago | | TUN | Tunisia | | TUR | Turkey | | TUV | Tuvalu | | TZA | Tanzania | | UGA | Uganda | | UKR | Ukraine | | URY | Uruguay | | USA | United States of America | | UZB | Uzbekistan | | VAT | Vatican City | | VCT | Saint Vincent and the Grenadines | | VEN | Venezuela | | VGB | British Virgin Islands | | VIR | United States Virgin Islands | | VNM | Vietnam | | VUT | Vanuatu | | WLF | Wallis and Futuna | | WSM | Samoa | | XXK | Kosovo | | YEM | Yemen | | ZAF | South Africa | | ZMB | Zambia | | ZWE | Zimbabwe |Hazard type
| **code** | **name** | |:---:|---| | **CF** | Coastal Flood | | **CS** | Convective Storm | | **DR** | Drought | | **EQ** | Earthquake | | **ET** | Extreme Temperature | | **FL** | Flood | | **LS** | Landslide | | **MH** | Multi-Hazard | | **TS** | Tsunami | | **VO** | Volcanic | | **WF** | Wildfire | | **WI** | Strong Wind |Hazard process type
| **code** | **hazard\_code** | **name** | |:---:|---|---| | **FCF** | CF | Coastal Flood | | **FSS** | CF | Storm Surge | | **TOR** | CS | Tornado | | **DTA** | DR | Agricultural Drought | | **DTH** | DR | Hydrological Drought | | **DTM** | DR | Meteorological Drought | | **DTS** | DR | Socio-economic Drought | | **Q1R** | EQ | Primary Rupture | | **Q2R** | EQ | Secondary Rupture | | **QGM** | EQ | Ground Motion | | **QLI** | EQ | Liquefaction | | **ECD** | ET | Extreme cold | | **EHT** | ET | Extreme heat | | **FFF** | FL | Fluvial Flood | | **FPF** | FL | Pluvial Flood | | **LAV** | LS | Snow Avalanche | | **LSL** | LS | Landslide (general) | | **TSI** | TS | Tsunami | | **VAF** | VO | Ashfall | | **VBL** | VO | Ballistics | | **VFH** | VO | Proximal hazards | | **VLH** | VO | Lahar | | **VLV** | VO | Lava | | **VPF** | VO | Pyroclastic Flow | | **WFI** | WF | Wildfire | | **ETC** | WI | Extratropical cyclone | | **TCY** | WI | Tropical cyclone |Hazard intensity measure
| **_process\_code_** | **_hazard\_code_** | **_im\_code_** | **_description_** | |:---:|---|---|---| | **QGM** | EQ | PGA:g | Peak ground acceleration in g | | **QGM** | EQ | PGA:m/s2 | Peak ground acceleration in m/s2 | | **QGM** | EQ | PGV:m/s | Peak ground velocity in m/s | | **QGM** | EQ | SA(0.2):g | Spectral acceleration with 0.2s period | | **QGM** | EQ | SA(0.3):g | Spectral acceleration with 0.3s period | | **QGM** | EQ | SA(1.0):g | Spectral acceleration with 1.0s period | | **QGM** | EQ | SA(3.0):g | Spectral acceleration with 3.0s period | | **QGM** | EQ | SA(0.2):m/s2 | Spectral acceleration with 0.2s period | | **QGM** | EQ | SA(0.3):m/s2 | Spectral acceleration with 0.3s period | | **QGM** | EQ | SA(1.0):m/s2 | Spectral acceleration with 1.0s period | | **QGM** | EQ | SA(3.0):m/s2 | Spectral acceleration with 3.0s period | | **QGM** | EQ | Sd(T1):m | Spectral displacement | | **QGM** | EQ | Sv(T1):m/s | Spectral velocity | | **QGM** | EQ | PGDf:m | Permanent ground deformation | | **QGM** | EQ | D\_a5-95:s | Significant duration a5-95 | | **QGM** | EQ | D\_a5-75 :s | Significant duration a5-75 | | **QGM** | EQ | IA:m/s | Arias intensity (Iα) or (IA) or (Ia) | | **QGM** | EQ | Neq:- | Effective number of cycles | | **QGM** | EQ | EMS:- | European macroseismic scale | | **QGM** | EQ | AvgSa:m/s2 | Average spectral acceleration | | **QGM** | EQ | I\_Np:m/s2 | I\_Np by Bojórquez and Iervolino | | **QGM** | EQ | MMI:- | Modified Mercalli Intensity | | **QGM** | EQ | CAV:m/s | Cumulative absolute velocity | | **QGM** | EQ | D\_B:s | Bracketed duration | | **FFF** | FL | d\_fff:m | Flood water depth | | **FPF** | FL | d\_fpf:m | Flood water depth | | **FFF** | FL | v\_fff:m/s | Flood flow velocity | | **FPF** | FL | v\_fpf:m/s | Flood flow velocity | | **TCY** | WI | v\_tcy(3s):km/h | 3-sec at 10m sustained wind speed (kph) | | **ETC** | WI | v\_ect(3s):km/h | 3-sec at 10m sustained wind speed (kph) | | **TCY** | WI | v\_tcy(1m):km/h | 1-min at 10m sustained wind speed (kph) | | **ETC** | WI | v\_ect(1m):km/h | 1-min at 10m sustained wind speed (kph) | | **TCY** | WI | v\_tcy(10m):km/h | 10-min sustained wind speed (kph) | | **ETC** | WI | v\_etc(10m):km/h | 10-min sustained wind speed (kph) | | **TCY** | WI | PGWS\_tcy:km/h | Peak gust wind speed | | **ETC** | WI | PGWS\_ect:km/h | Peak gust wind speed | | **LSL** | LS | d\_lsl:m | Landslide flow depth | | **LSL** | LS | I\_DF:m3/s2 | Debris-flow intensity index | | **LSL** | LS | v\_lsl:m/s2 | Landslide flow velocity | | **LSL** | LS | MFD\_lsl:m | Maximum foundation displacement | | **LSL** | LS | SD\_lsl:m | Landslide displacement | | **LSL** | LS | LSI:- | Landslide susceptibility Index | | **LSL** | LS | haz\_lsl:- | Landslide hazard index | | **TSI** | TS | Rh\_tsi:m | Tsunami wave runup height | | **TSI** | TS | d\_tsi:m | Tsunami inundation depth | | **TSI** | TS | MMF:m4/s2 | Modified momentum flux | | **TSI** | TS | F\_drag:kN | Drag force | | **TSI** | TS | Fr:- | Froude number | | **TSI** | TS | v\_tsi:m/s | Tsunami velocity | | **TSI** | TS | F\_QS:kN | Quasi-steady force | | **TSI** | TS | MF:m3/s2 | Momentum flux | | **TSI** | TS | h\_tsi:m | Tsunami wave height | | **TSI** | TS | Fh\_tsi:m | Tsunami Horizontal Force | | **VAF** | VO | h\_vaf:m | Ash fall thickness | | **VAF** | VO | L\_vaf:kg/m2 | Ash loading | | **FSS** | CF | v\_fss:m/s | Maximum water velocity | | **FSS** | CF | d\_fss:m | Storm surge inundation depth | | **DTA** | DR | CMI:- | Crop Moisture Index | | **DTM** | DR | PDSI:- | Palmer Drought Severity Index | | **DTM** | DR | SPI:- | Standard Precipitation Index |Exposure category
| Category | |---| | Buildings | | Indicators | | Infrastructure | | Crops, livestock and forestry |Exposure occupancy types
| Occupancy | |---| | Residential | | Commercial | | Industrial | | Healthcare | | Education | | Government | | Infrastructure | | Crop | | Livestock | | Forestry | | Mixed |1. Contribution attributes
The data used or produced in a risk analysis project generally fall into one of the four categories described by the RDL schema. Datasets originating from the same project can be grouped by common attributes.
| | **Field name** | **Description** | **Example** | |:---:|---| --- | --- | |* | component | Schema to be used | Exposure | |*| model\_source | Name of source model | OSM | |*| model\_date | Model release date | | | | project | Project under which data has been produced | OpenMapProject | | | purpose | Purpose for what the data has been produced | | | | notes | Details about the dataset | | | | bibliography | Title and authors of documents containing relevant information | | | | version | Version of the dataset | 1.2 | |*| geo\_coverage | ISO code(s) of countries covered | COM,MDG,SYC | |*| license\_code | Type of license | CC0 |
2. Hazard attributes
Example are hazard maps representing the mean or max intensity over an area, or the extent of a phenomena of a given intensity and probability. They can represent a probabilistic scenario, or an empirical event. Most usually these data consist of grid data stored as raster (.tif), but in some cases vector layers are used instead. [ex_hazards1] [ex_hazards2] [ex_hazards3] The RDL hazard schema uses a hyerarchical definition of hazard data structure:
The _**Event set**_ stores one or more scenarios for a specific hazard. The main hazard type is defined by a specific code defined in common tables. Key attributes are: || **Field name** | **Description** | **Example** | |:---:| --- | --- | --- | |*| hazard\_type | Code provided in common tables | _FL_ for Floods | || description | General details about this set of layers | _Simulated flood depth at country scale_ | |*| is\_prob | Probabilistic or deterministic analysis. | _TRUE_ for probabilistic | Each **_Event_** is characterised by three main parameters: the methodology which originated the dataset; the specific occurrance probability, or frequency; and the main and secondary trigger processes which comes before the rapresented hazard in the cascade of events, and are identified as cause, or concause for the manifestation of the represented hazard. || **Field name** | **Description** | **Example** | |:---:|---| --- | --- | |*| calculation\_method | The methodology used for the calculation of this event | _Simulated_ for model simulations; _Observed_ for empirical data; _Inferred_ for models elaborated on empirical data (?) | || frequency | The frequency of occurrence of the present event | _0.01_ meaning probability of once in 100 years | || occurrence\_probability | The occurrance probability | ?? | || occurence\_time\_span | The duration (years) of the period used to specify either the frequency or the occurrence\_probability | _100_ | || trigger\_hazard\_type | Hazard type that triggered the event | _CS_ (Convective Storm) | || trigger\_process\_type | Process type that triggered the event | _TCY_ (Tropical Cyclone) | || description | Provides additional information about this specific event | _The flood has been caused by heavy rainfall during cyclone Karen_ | For each **_Event_**, one more **_Footprint_set_** can be available, where each is one possible realisation of the event, i.e. one footprint could represent minimum, another footprint the average and another one the maximum. In this way, the event uncertainty can be represented explicitly, through the inclusion of multiple footprints per event. Hazard footprints can be assigned an occurrence frequency, and can represent a scenario footprint or return period hazard map. Inclusion of an event start and end time enables description of long-duration events. || **Field name** | **Description** | **Example** | |:---:| --- | --- | --- | |*| process\_type | The typology of hazard process | _FPF_ for Pluvial flood | |*| imt | Intensity measure types | _d_fpf:m_ for water depth | || data\_uncertainty | This attribute describes the typology of uncertainty | _Min to Max range_ | The **_Footprint_** entity contains information on a specific realisation of an **_Event_**. The uncertainty of a particular event is captured either by the construction of many footprints or by a single footprint which contains information about uncertainty. || **Field name** | **Description** | **Example** | |:---:|---| --- | --- | || trigger\_footprint | ? | ? | || uncertainty\_2nd\_moment | ?? | _Minimum_ |
3. Exposure attributes
This component can include a wide variety of data describing structural, infrastructural and environmental asset, population, socio-economic descriptors. Exposure can be stored at multiple scales, more often using vectors, namely polygons (e.g. building footprint), points (e.g. geolocated element) and lines (e.g. transport infrastructures, lifelines), but in same case exposure estimates are aggregated at ADM level or distributed over a raster grid. [ex_exp1] [ex_exp2] [ex_exp3]
|| **Field name** | **Description** | **Example** | |:---:| --- | --- | --- | |*| name | Name of exposure model | | || description | Description of exposure model | | || taxonomy\_source | Name of taxonomy source | | |*| category | Category of exposure model | | || area\_type | Aggregated or per asset | | || area\_unit | Unit of measure of area | | || tag\_names | Additional tag information | | || use | Usage type of the asset (e.g. residential, commercial) | | There are 4 main categories of exposure which include 11 exposure types.
Category
| Category | |---| | Buildings | | Indicators | | Infrastructure | | Crops, livestock and forestry |Type
| Type | |---| | Residential | | Commercial | | Industrial | | Healthcare | | Education | | Government | | Infrastructure | | Crop | | Livestock | | Forestry | | Mixed |4.Vulnerability attributes
The vulnerability schema includes physical fragility and vulnerability relationships in relation to specific hazards or for multi-hazard. The schema distinguishes key information describing the function, including: - function type (i.e fragility, vulnerability, damage-to-loss); - ountries the function was developed for, measured in terms of to geographic relevance. - development approach (empirical, analytical, judgement, hybrid, code-based); - mathematical model used (including exponential, cumulative lognormal/normal); - the intensity measure and asset type the function relates to; - loss parameter / engineering demand parameter values. The schema consists of three base tables (_f\_core_, _f\_specifics_, and _f\_additional_) and five supporting tables (_f\_scoring, damage\_scale, edp\_table, lp\_table, reference\_table_).
5. Loss attributes
Losses can be rapresented in many different way: regular raster grids, points, or polygons. Often, the loss data consist of measures aggregated at the administrative unit level. [ex_loss1][ex_loss2][ex_loss3]
Category
| Category | |---| | Buildings | | Indicators | | Infrastructure | | Crops, livestock and forestry |Loss element
| Loss element | |---| | Structure | | Content | | Business interruption |Loss impact
| Loss impact | |---| | Direct | | Indirect |Frequency
| Frequency | |---| | Return Period | | Probability of exceedance | | Rate of exceedance |Loss type
| Loss type | |---| | Ground-up | | Insured |Metric
| Metric | |---| | Annual Average Loss | | Annual Average Loss Ratio | | Probable Maximum Loss |