Implementation of Impact/Risk assement model for Heat Waves with EMIKAT

maesbri commented 6 years ago

Implementation of the model defined in #21.

p-a-s-c-a-l commented 6 years ago

@bernhardsk @humerh

See also Impact modelling logic example

p-a-s-c-a-l commented 6 years ago

Some ideas for the expected output format of the model, visualisation of tabular data but also the need for an additional aggregation of the model results for easy scenario analysis and comparison are discussed in this issue: https://github.com/clarity-h2020/scenario-analysis/issues/7

p-a-s-c-a-l commented 6 years ago

Related discussions that need feedback from @clarity-h2020/mathematical-models-implementation-team :

p-a-s-c-a-l commented 5 years ago

General workflow and examples based on this comment:

When Impact shall be calculated (IA/RA Step),

1) CSIS Frontend sends request to Impact Model Service (EMKIT + some simple REST Interface) with URI of the current study object. (you need to be logged-in at CSIS to access these example URIs) 2) Impact Model Service needs to authenticate at CSIS and retrieves study object JSON representation. The study object JSON is a 'flat' object, so it contains just links to sub-objects like data-package. Question @therter : Mabe there is possibility to get an object from that API that is already completely resolved with all of its sub objects? 3) Impact Model Service extracts from the study object the properties relevant for the impact model. E.g. field_area and field_data_package. 4) The Data Package object, which follows the Data Package Specification contains meta-data for all HC-LE/EE/VA datasets needed by the impact model service (as sub-objects in field_resources). 5) the datasets should be 'preconfigured' at the level of the Impact Model Service, so the service just needs the unique ids of the respective datasets listed in field_resources. @fgeyer16 Could we use the field uuid to uniquely identify a datasets or does the uuid change for each new revision?

To be discussed: How to notify the CSIS of new Impact Model Results? In general, the Impact Model Service should call the JSON API and PUT meta-data following the Data Resource Specification (JSON Example). But this meta-data about the IA/RA results must not be added to the original Data Package but a copy that is only accessible by the members of the study.

humerh commented 5 years ago

We have to concern the following events:

Creating a new study area: This should trigger a PUT call to emikat ("insertNewScenario") with the fields (name, description) -> returns also the ID of the new created Scenario The description field should at the end contain the description of the study (<2000 chars) PLUS some METADATA, like
CSIS_STUDY_AREA: LL_X, LL_Y, RU_X, RU_Y (in WGS84) CSIS_URL: url to get further info for the study and the data packages
Updating a study: This should trigger a PUSH call to emikat ("updateScenario") with the fields (scenarioId, name, description)

As a consequence of UPDATE all data (mbd) are resynchronized to emikat again.

..... (further definitions will be provided after a already planned session with Patrik Kaleta on Monday)

A SWAGGER-Interface to emikat will be available at the location: https://service.emikat.at/EmiKat/swagger/index.html

patrickkaleta commented 5 years ago

After our weekly developer telco this monday, @bernhardsk @humerh @fgeyer16 and I discussed how Emikat and Drupal could communicate and exchange data. For now we will try the following:

using the SWAGGER-Interface Drupal will notify Emikat when a new Study is created or an existing one is being updated and send necessary information (Study ID, Study area, path to data-package,...)
additional information can be retrieved by Emikat via a REST-view which will contain all necessary fields (since accessing the whole Study object via REST is probably an overkill)
Drupal will have to store the Emikat-given ID (which it will receive as response from Emikat) of the Study as well, so that in the case of changes in a Study, Drupal can tell Emikat which study needs to be updated. (For that we will look into the Business-rules module to see how this could be implemented)

p-a-s-c-a-l commented 5 years ago

Interesting! Would it make sense / be possible to define such a REST View for Map Layers that are relevant in the context of a specific step. Please have a look at this issue @therter is working on: https://github.com/clarity-h2020/map-component/issues/2

Currently we intend to load the whole study JSON + Data Package & Data Resources JSON and parse it in the Map Component. A REST View that extracts just the relevant properties from the (Data Package Resources) would me more convenient.

patrickkaleta commented 5 years ago

@p-a-s-c-a-l Yes, creating such a REST View with only the relevant properties from the data-resource should be possible.

I will first deal with the REST view for Emikat (as sort of a proof of concept) and then have a closer look at the issue you mentioned above.

patrickkaleta commented 5 years ago

I had a look at this over the weekend and tried implementing the following: 1) based on the StudyID a REST view returns the path to another REST view for each resource in the datapackage which is used in the Study 1step_path-to-resource

2) the second REST view returns the relevant fields 2step_resource_analysis-context

Originally I tried to do it all in one step, but the problem is getting the necessary data throughout all the different content types (Study -> Data package -> Resource -> Analysis context). It is possible to display a View inside another View (I did that for the second step where I display the Analysis context fields as a nested View), but as can be seen in the second Screenshot it doesn't appear to be properly formatted JSON.

Would it work for the map compoment, to make multiple REST calls in order to get all necessary fields or does it need to happen in just one REST call?

fgeyer16 commented 5 years ago

@patrickkaleta How did you nest the second view into the first?

therter commented 5 years ago

@patrickkaleta Yes, it is also possible to make multiple rest calls from the map component

patrickkaleta commented 5 years ago

@fgeyer16 in this particular instance I displayed the analysis context as a field with the formatter set to "View" and selected the other REST view. As a view argument I'm passing the Field value, which is the ID of the analysis context.

You can view the settings of this REST view here: https://csis.myclimateservice.eu/admin/structure/views/view/rest_node/edit/rest_export_4

fgeyer16 commented 5 years ago

@patrickkaleta OK This is they way I had done it as well.

bernhardsk commented 5 years ago

Progress update by @bernhardsk (AIT) and @humerh (AIT) in regards to the general task "Implementation of Impact/Risk assement model with EMIKAT #22":

A first version of the Impact Calculation Module has been implemented in AIT EMIKAT. Impact was calculated for Hazard Index Type "heat-wave:consecutive-summer-days" using example data and based on the impact model descriptions provided by PLINIVS. (10 out of 15900 lines are listed:) grafik

A view has been created in EMIKAT to aggregate the results from the impact calculation, which currently looks like this: View (as Table): Damage level of Elements at Risk of each Vulnerability Class for a specific Impact Scenario/Hazard Index (without listing IDs): grafik

The following is also described in https://github.com/clarity-h2020/table-components/issues/4#issuecomment-457285867 but was copied for convenient reading and relevance: However the discussions during the CLARITY Developers/co-creation meeting in Vienna showed that the impact modelling should be changed slightly. The following changes should be carried out: 1) The impact calculation has to focus on 1 specific hazard event at a time (e.g. for heat wave: 6 consecutive days over 30°C max). Therefore it makes no sense to include information about the time period and the emissions scenario (rcp). 2) As input for the hazard event data, the impact calculation should use the Hazard-Local Effect layer/data. 3) The way how the vulnerability functions/tables are taken into account should be changed (due to the changes in bulletpoint "1)").

Next steps will focus on 1) correcting the impact calculation as described above and changing the RA/IA table view accordingly. 2) connecting EMIKAT with the front-end in order to exchange the neccessary data to be shown to users in the front-end RA/IA table. As for the next steps, only impact will be calculated and show. Risk will follow at a later stage. 3) implementing another example scenario (e.g. pluvial flood) using already available or otherwise dummy data.

maesbri commented 5 years ago

@humerh, @bernhardsk Data produced by EMIKAT don't contain proper identifiers (i.e., hazard, elements at risk, vulnerability classes and damage classes identifiers (probably others) which makes difficult to parse and use the information within the CSIS.

I'have found this issue with the damage level probability functions example you provided but I guess it might be happening with other results.

EmiKat_Damage Level Probability Functions_Description_HHu.docx

We must be consistent with the identifiers use across all pieces of information produced/consumed in the whole system.

patrickkaleta commented 5 years ago

@humerh, @bernhardsk , @maesbri I just wanted to post the recent mail exchange regarding https://github.com/clarity-h2020/csis/issues/22#issuecomment-471487338 here in this issue:

Mail from Miguel:

The identifiers can be found either in the Taxonomies elements (for hazards, elements_at_risk, etc.) or in the Enterprise Architect document in github (https://github.com/clarity-h2020/data-package/blob/master/docs/clarity-data-models.eap) . However, for the later, I need to update the ids for elements at risk, damage classes and vulnerability classes (I think) in order to match them with the ones in the taxonomies.

I would also suggest you to have a look at the naples data package example () as the identifiers are also used there (https://github.com/clarity-h2020/data-package/blob/master/examples/dc1-naples/datapackage.json)

I’ll update the pending information during this week as soon as I am done with this other meeting I am currently involved in.

Response from Heinrich:

we have to define, WHERE we can get an identifier, which we can rely.

Emikat has defined Reference tables (or Taxonomies ) similar we found in the Enterprise Schema. For each of the entries we have an own (emikat)-ID. This ID is private for us, but currently the only ID, which is available for us.

The Taxonomies in the Enterprise are changing the names very often.

There are also the Taxonomy Definitions on the Drupal site. These definition are new for us, but we found there a column like “hazard_id”. Is in the future this field FIX? Can we use this column as an alternate ID for our synchronization??

Which column can we use for mapping our definitions???

Contains Drupal the reference definition of taxonomy-Ids, or the Enterprise file??

We have to define this.

Then we can do the remapping and we will return this fixed ID instead of our emikat-ID.

On a similar note, I tested the Entity reports module, which shows the structure of the available content types, paragraphs and also taxonomies. Additionally, it can export the structure as json (example of the structure of content types attached as TXT file, since GitHub doesn't support JSON attachments). Could this be useful for any of you?

p-a-s-c-a-l commented 5 years ago

@humerh Identifiers in Data Packages (in CSIS - "Drupal Site") are unique and immutable. IMHO a Data Package corresponds to a 'project' in EMIKAT. Supporting a new Data Package (e.g. for DCx - Linz @Itsman-AT ) would mean to create a new proejct in EMIKAT and map the CSIS/DataPackage identifiers to your internal identifiers of the HC/EE/etc Datasets/Layers.

Example: Content of the DC1 Naples Reference Data Package related to HC-LE shown in CSIS: HC-LE/DATA. @DanielRodera @maesbri At HC-LE/DATA, the HC-LE Input Layers are shown, but not the actual HC-LE Layers. This is probably related to https://github.com/clarity-h2020/map-component/issues/15

p-a-s-c-a-l commented 5 years ago

@DanielRodera @patrickkaleta @fgeyer16 I wanted to check whether there are resources related to Vulnerability Assessment in the Naples Data Package, but the edit mode shows the identifiers instead of the names of the Resource:

grafik

I think this can be configured in the in the short description view mode but I don know where and how. And of course I don't want to break anything. ;-)

maesbri commented 5 years ago

@patrickkaleta , @humerh

Identifiers can be found either in the drupal taxonomies (i.e., Hazards, EmissionsScenarios, EU-GL and ElementsAtRisk) or in the Enterprise Architect file in github.

I have spent the last hours reviewing both and now they should have exactly the same identifiers. If you use the taxonomies for getting the identifiers, use value contained in the "id" attribute (all taxonomy elements should have this attribute. I have renamed the existing ones (like hazard_id) in order to make it easier.

BTW, please don't rely anymore in the contents of the outdated specification we have in:

data-package-general.md (https://github.com/clarity-h2020/data-package/blob/master/docs/data-package-general.md)
data-package-resource.md (https://github.com/clarity-h2020/data-package/blob/master/docs/data-package-resource.md)

It is too much effort to continuously update the drupal taxonomy, the mark-down tables and the EA file and I don't think it doesn't pay the hassle. My plan is to modify one of the word templates that EA uses to generate the report out of the diagram models in order to generate in a nice way our Data Package specification. I've seen that there is a command-line tool that converts from rtf to asciidoc format which could be the solution to later on update the specification in the .md files in github if we still want to have them there (I'll give it a try).

maesbri commented 5 years ago

@humerh Identifiers in Data Packages (in CSIS - "Drupal Site") are unique and immutable. IMHO a Data Package corresponds to a 'project' in EMIKAT. Supporting a new Data Package (e.g. for DCx - Linz @Itsman-AT ) would mean to create a new proejct in EMIKAT and map the CSIS/DataPackage identifiers to your internal identifiers of the HC/EE/etc Datasets/Layers.

Example: Content of the DC1 Naples Reference Data Package related to HC-LE shown in CSIS: HC-LE/DATA. @DanielRodera @maesbri At HC-LE/DATA, the HC-LE Input Layers are shown, but not the actual HC-LE Layers. This is probably related to clarity-h2020/map-component#15

@p-a-s-c-a-l , @therter No, the problem is that the 4 HC-LE effects raster layers in Meteogrid geoserver are not yet included as resources in the Drupal Data Package (they are currently in the json file https://github.com/clarity-h2020/data-package/blob/master/examples/dc1-naples/datapackage.json). I'll add them now.

BTW, how can be that the map component is listing those 4 layers if they are not in the data package? Did you hardcode them?

BTW2, Why the BBOX study area for naples is pointing to Linz when I open any of the map component?

therter commented 5 years ago

BTW, how can be that the map component is listing those 4 layers if they are not in the data package? Did you hardcode them?

The layers were hard coded and then replaced by the layers from the data package

BTW2, Why the BBOX study area for naples is pointing to Linz when I open any of the map component?

The study area was set to a wrong place. I will set it to naples.

maesbri commented 5 years ago

@p-a-s-c-a-l , @therter

HC-LE resources have been added to the Naples data package (resources identifier ranging from 26 to 29).

p-a-s-c-a-l commented 5 years ago

Model implemented, API available here.

p-a-s-c-a-l commented 5 years ago

Recent changes in model to be implemented.

DenoBeno commented 5 years ago

In theory this is done. Who and how (and when) can check if the output makes sense?

DenoBeno commented 5 years ago

Implemented for heat mortality, I suppose? Close this and open new one for other impacts?

clarity-h2020 / csis

Implementation of Impact/Risk assement model for Heat Waves with EMIKAT #22