Support for Exposure Data [EE API]

[p-a-s-c-a-l]

We need access to the Exposure Data stored in EMIKAT. Currently it supports Population Exposure including ~age group distribution for whole Europe~. There is also finer grained local population exposure data for the city of Naples available which can be served transparently via the same API.

The API could support a parameter $exposureType which contains the id of the respective Taxonomy.

The API will be accessed by the table-state-rest-api and then by the related EE Table Component. Additionally, we need an WMS Endpoint for the EE Map Component.

The respective EE Data Package Resource has to be updated accordingly and example request / response URLs (including WMS URLs for RA/IA maps) have to be added to the Wiki.

[p-a-s-c-a-l]

I seems that "The population are no longer grouped according to age classes due to lack of sufficient data for the age groups.". @humerh can you please clarify this with @stefanon and @alecapolupo ?

[ghost]

Yes, the population will not be grouped in three classes since the data at european level are not available in that format. Copernicus uploaded the population data already spatialized, therefore they should just be converted in the grid format (https://land.copernicus.eu/local/urban-atlas/urban-atlas-2012).

[p-a-s-c-a-l]

the data at european level are not available in that format.

Yes, but Heinrich somehow managed to derive the age groups from other European datasets and to merge it with the Copernicus population data. Right, @humerh?

[ghost]

It is not necessary to apply that procedure because we have just vulnerability function for all the population.

[p-a-s-c-a-l]

Exposure data (Population) calculated from EMIKAT:

Methode: We use the Populationen Data from EUROSTAT (1km*1km) and split this data to the 500m-Grid by using Landcover data from Copernicus. This data are available for all Europe.

REST API: https://service.emikat.at/EmiKatTst/api/scenarios/2846/feature/tab.CLY_EL_POPULATION_FAKE.2016/table/data?rownum=1000

or:

https://service.emikat.at/EmiKatTst/api/scenarios/$emikatId/feature/tab.CLY_EL_POPULATION_FAKE.2016/table/data

Output Example:

{
  "name": "CLY_EL_POPULATION_FAKE",
  "description": "Beschreibung fürCLY_EL_POPULATION_FAKE",
  "columnnames": [
    "GRID_ID",
    "POPULATION_TOTAL",
    "POPULATION_LT14",
    "POPULATION_14TO64",
    "POPULATION_GT64"
  ],
  "rows": [
    {
      "rownum": 0,
      "values": [
        "500mE46730N19800",
        395,
        52,
        254,
        89
      ]
    },
    {
      "rownum": 1,
      "values": [
        "500mE46730N19805",
        394,
        52,
        254,
        88
      ]
    },
    {
      "rownum": 2,
      "values": [
        "500mE46735N19800",
        394,
        52,
        254,
        88
      ]
    },

[p-a-s-c-a-l]

@humerh

Thank you, some questions and remakes:

• CLY_ELPOPULATIONFAKE.2016 should be renamed to CLY_ELPOPULATIONINTERPOLATED.2016 or something like that :-).
• Population Data contains age groups which are not needed any more. We can keep them but they are probably just relevant when dealing with local exposure data.
• We need a WMS Interface for the EE Map Component and ideally a WCS interface for the table-state-rest-api. Alternatively, a tabular format that is not as verbose as the JSON is preferred.

Alternatively we could keep the source data that is needed for visualisation purposes only on (METEOGRID) Geoserver and expose it via WCS/WMS and duplicate the data that is needed for Impact/Adaptation Calculation in EMIKAT. However, I would prefer to keep the data in one central place (EMIKAT).

Here are some samples WCS/WMS URLs:

• WCS: https://clarity.meteogrid.com/geoserver/Exposure/wcs?SERVICE=WCS&VERSION=2.0.1&REQUEST=GetCoverage&COVERAGEID=Exposure:Population_15to65_naples&FORMAT=geotiff&boundingbox=4647500,1947000,4716000,2003000,urn:ogc:def:crs:EPSG::3035
• WMS: https://clarity.meteogrid.com/geoserver/Exposure/wms?service=WMS&version=1.1.0&request=GetMap&layers=Exposure:Population_15to65_naples&bbox=4647500,1947000,4716000,2003000&width=725&height=768&srs=EPSG:3035&format=image/png

Note that these URLs currently point to local Exposure Data obtained from ISTAT (Italian National Institute of Statistics)! The structure and format exposed at the APIs must be the same for the European level exposure data obtained from Copernicus. I think it is more than obvious that we cannot support different data formats for the same data type. We need to be able to switch data packages, e.g. perform the same type of study with European Level Data Package and the Local Data Package.

[humerh]

On your request I changed my Population evaluation and remove the age distribution. I did also a renaming of table.

So my interfaces are now the followings (replace 2846 by $emikatId):

WMS: https://service.emikat.at/geoserver/clarity/wms?service=WMS&version=1.1.0&request=GetMap&layers=clarity:view.2955&bbox=4673500.0,1980000.0,4691500.0,1988000.0&width=768&height=341&srs=EPSG:3035&format=image/gif&CQL_FILTER=SZ_ID=2846

[p-a-s-c-a-l]

Thanks! WMS doesn't seem to work, though:

<?xml version="1.0" encoding="UTF-8" standalone="no"?><!DOCTYPE ServiceExceptionReport SYSTEM "https://service.emikat.at/geoserver/schemas/wms/1.1.1/WMS_exception_1_1_1.dtd"> <ServiceExceptionReport version="1.1.1" >   <ServiceException code="internalError">
      Rendering process failed
3
</ServiceException></ServiceExceptionReport>

Are you going to provide a WCS endpoint, too?

For the Data Package, we need some meta-information about the Exposure Data, especially to properly credit the original data provider (Copernicus). Is the description of the source data already available in our CKAN? If not, where can I find the the original data?

[humerh]

Additional Services available: EMIKAT REST: https://service.emikat.at/EmiKatTst/api/scenarios/2846/feature/tab.CLY_EL_POPULATION_INTERPOLATED.2016/table/data?rownum=1000

or EMIKAT GeoJSON: https://service.emikat.at/EmiKatTst/api/scenarios/2846/feature/tab.CLY_EL_POPULATION_INTERPOLATED.2016/table/geojson?rownum=1000

or CSV: https://service.emikat.at/EmiKatTst/api/scenarios/2846/feature/tab.CLY_EL_POPULATION_INTERPOLATED.2016/table/csv?rownum=1000

or WMS: WMS: https://service.emikat.at/geoserver/clarity/wms?service=WMS&version=1.1.0&request=GetMap&layers=clarity:view.2955&bbox=4673500.0,1980000.0,4691500.0,1988000.0&width=768&height=341&srs=EPSG:3035&format=image/gif&CQL_FILTER=SZ_ID=2846

A WCS endpoint is not available.

[p-a-s-c-a-l]

For the Data Package, we need some meta-information about the Exposure Data, especially to properly credit the original data provider (Copernicus). Is the description of the source data already available in our CKAN? If not, where can I find the the original data?

Is the local Naples Exposure Data in EMIKAT available too, so that we can test with the table rest API?

[humerh]

The local Naples Exposure Date in EMIKAT are too available under the name "tab.CLY_EL_POPULATION.1758"

Example: https://service.emikat.at/EmiKatTst/api/scenarios/2846/feature/tab.CLY_EL_POPULATION.1758/table/geojson?rownum=1000

[humerh]

Used data for calculation:

1) Population European Popuation statistics 2011 GEOSTAT 2011 V2.0.1 European raster of Population in a 1km*1km grid (ETRS89 / LAEA) Source: Eurostat, GEOSTAT 2011 https://ec.europa.eu/eurostat/web/gisco/geodata/reference-data/population-distribution-demography/geostat

2) Land Use This layer has been generated from CORINE Land Cover (CLC) (https://land.copernicus.eu/pan-european/corine-land-cover) 2018. Only the CLC code 11* are extracted: 111.. Artificial surfaces,Urban fabric,Continuous urban fabric 112.. Artificial surfaces,Urban fabric,Discontinuous urban fabric

[humerh]

What is the problem with WMS service? Is it still present?

[p-a-s-c-a-l]

WMS works now. Thanks!

[p-a-s-c-a-l]

EMIKAT REST: https://service.emikat.at/EmiKatTst/api/scenarios/2846/feature/tab.CLY_EL_POPULATION_INTERPOLATED.2016/table/data?rownum=1000

Hm, same question as here: Is it possible to return only those records that are within the area of the current study (2846)? Alternatively, does the API support additional filter parameters, e.g. by bounding box so that we could include the '$studyArea' in the request?

[p-a-s-c-a-l]

Hm, same question as here: Is it possible to return only those records that are within the area of the current study (2846)? Alternatively, does the API support additional filter parameters, e.g. by bounding box so that we could include the '$studyArea' in the request?

[humerh]

Yes, you will get only those records, which are in the area of the study, I have implemented it already. As the content of the records will change due to the discussion about the duration I have not published it.

It will be like this:

https://service.emikat.at/EmiKatTst/api/scenarios/2846/feature/tab.CLY_HAZARD_EVENTS_STUDY.2036/table/data?rownum=1000

[p-a-s-c-a-l]

When I remove the filter CQL_FILTER=SZ_ID=$emikat_id from the Population Exposure WMS URL, I'm, still getting only the exposure for the study area:

I would expect to get exposure data for whole Europe. Why? Because we want to show the visualisation widgets (maps and table) not only in the study views but also in the Data Package and resource views. See https://github.com/clarity-h2020/data-package/issues/26

Therefore it would be helpful to be able to obtain the data for whole Europe via for the REST API, too. E.g. instead of

• https://service.emikat.at/EmiKatTst/api/scenarios/2846/feature/tab.CLY_EL_POPULATION_INTERPOLATED.2016/table/data?rownum=1000

we may use an optional query parameter, e.g.

• https://service.emikat.at/EmiKatTst/api/feature/tab.CLY_EL_POPULATION_INTERPOLATED.2016/table/data?scenario=2846

In any case, while this might work for (static) population data, it will probably not be available for HC-LE data and IA Data, because they are calculated on the fly for each new study, right?

[humerh]

_When I remove the filter CQL_FILTER=SZ_ID=$emikatid from the Population Exposure WMS URL, I'm, still getting only the exposure for the study area:

Currently we have only ONE Study defined and calculated in EmiKat, so the result is the same. If more than one Study exists, this parameter is essential.

... showing data for all the Europe ...

It was a strategic decision to calculate all the dynamic tables only for the area of the study. The recalculation time is therefore in a very good range (seconds or minutes). Recalculation time for Europe needs in many cases hours or days.

The examples of population distribution is currently handled with the same algorithms and is calculated only for the area of the study. Locally I have the information of whole Europe available, so it would be possible to calculate also this layer on a European level. The input layers are also CONSTANTs and so this calculated population layer need not be recalculated for each Study. But I need for my algorithms also this projected view to the study area, so it would be an adiitional task to make this European layer available.

A compromise could be, that we provide only the 1km*1km input layer of population on European level. Is this sufficient?

[p-a-s-c-a-l]

A compromise could be, that we provide only the 1km*1km input layer of population on European level. Is this sufficient?

Since this European population layer is then used for visualisation in CSIS only and not for calculation in EMIKAT, it would be better to follow the same approach as for the Hazard indices layers:

@ghilbrae should make these layers available on METEOGRID Geoserver.

[ghilbrae]

I'm working on this and I hope to have something today or tomorrow. I'd like to know of you @humerh need 2006 and 2011 or only 2011. I'm working on 2011 right now.

[ghilbrae]

@humerh the population for 2011 is available in our geoserver (https://clarity.meteogrid.com/geoserver/). The name of the layer is: europe:EXT_Population_GEOSTAT_1K_2011 You can check it in the layer preview.

Here is a link to the GetMap: https://clarity.meteogrid.com/geoserver/europe/wms?service=WMS&version=1.1.0&request=GetMap&layers=europe%3AEXT_Population_GEOSTAT_1K_2011&bbox=1896628.6179337814%2C1097454.5685083764%2C7058042.777624368%2C6827128.019856293&width=691&height=768&srs=EPSG%3A3035&format=application/openlayers

It may take a while to load but the layer is quite heavy. Let me know if this works for you or if there's something missing or strange.

[humerh]

I can access this layer as WMS service. I would prefer to have a WCS service for this layer, because I need the data and not the picture. A WCS service would be the most efficent access to get the data of the study area.

Currently I continue to use my local copy of data till a WCS service is avalable.

[humerh]

As I need the data (not the map) is it possible to enable also WCS service for this layer?

[p-a-s-c-a-l]

Maybe @LauraMTG can enable the WCS?

[LauraMTG]

@humerh and @p-a-s-c-a-l , It is not possible to access SHAPE data through a WCS service (only for raster files). For this you need to use a WFS service and the procedure is the same.

[LauraMTG]

Hi @humerh and @p-a-s-c-a-l

I have transformed the shapefile into a raster file and it is already in geoserver with the name "geostat_GRID_POPULATION_1K_2011". Now you have the possibility to access the European population layer through the WCS service, if it is useful for you.

[humerh]

Thank you, it looks fine. I will switch to this data source in the next days.

[DenoBeno]

I suppose this is resolved, except that no $exposureType filter exists and/since the data is only available for the population?

[p-a-s-c-a-l]

No Exposure Data seems to be available for this study (Alba Iulia): https://service.emikat.at/EmiKatTst/api/scenarios/3209/feature/tab.CLY_EL_POPULATION_INTERPOLATED.2016/table/data?rownum=1000

[humerh]

Study with EmikatId=3209 was not triggered for calculation. The reason is in my eyes on the drupal side. So you get no data.

[p-a-s-c-a-l]

Hm, but there is an EMIKAT id assigned, hmm

[DenoBeno]

This works now. Guess we can close the issue. Several issues to consider for later

1) the table is not very useful IMO.

• GRID_ID is cryptic for the users, so per-grid data is generally not very useful in the tables. Unless we would have table & map linked on the same page. => maybe we should show average/min/max for the project area in a table? 2) How could we add "future exposure"?