1st Data Package: 02b Hazard Local Effects Layers for pluvial floods

[DenoBeno]

related to: https://github.com/clarity-h2020/data-package/issues/11#issuecomment-535404140_

Create Hazard Local Effects Layers for the flood hazard & impact modelling and make them available in CSIS.

This encompasses the following steps (to me improved!!!):

• [x] ZAMG to provide plain HC Data:
• [ ] METEOGRID to [create HC-LE layers] based on input layers prepared by ATOS(?)
• [ ] METEOGRID to [expose Layers (WCS/WMS) via their GeoServer]()
• [ ] METEOGRID to define some simple styling for Layers (no need to consider thresholds)
• [ ] CIS to [create Data Package Meta-Data in CSIS]()
• [ ] ??? to provide meta-data for [Data Management Plan]()
  Both HC-LE and respective input layers are open data that eventually to deposited in zenodo (if we have calculated the HC-LE data for whole Europe)

Important Notes:

• This tasks depends on Hazard Local Effects Modelling Workflow Implementation, currently HC-LE Data is not yet available!
• The Impact Model shall use HC-LE Data (500m grid), not plain HC Data (12.5km grid)
• There is already MUKLIMO HC data available for Naples. Although this data is meant for the Expert Workflow, we could use it temporarily in the screening workflow as substitute for the HC-LE data which yet has to be calculated.

[negroscuro]

As far as I know the formula for pluvial flood is (from PLINIVS excel):

As well specific layers for this in PLINIVS layers are:

• EU-Hydro corresponds to streams
• USGS - Hydrosheds corresponds to basins
• EU-DEM corresponds to Digital Elevation Model, this is only needed as data to do obtain streams since altitude is needed in start and end point of a single stream.
• SL_2LST I am not sure what is this, I recall Alessandra told me this would not be used to do the calculation so she pointed me that I could forget about it.

So from june those layers are available (Streams and Basins) so I asked Alessandra to crosscheck them to be sure they are correct, and she did, so nothing has been done on this since then (in terms of generation methodology, only load data from different cities regarding streams and basins). Those can be found in Atos Geoserver:

http://services.clarity-h2020.eu:8080/geoserver/clarity/wms?service=WMS&version=1.1.0&request=GetMap&layers=clarity%3Abasins&bbox=1990800.0%2C1013500.0%2C8140000.0%2C5270200.0&width=768&height=531&srs=EPSG%3A3035&format=text%2Fhtml%3B%20subtype%3Dopenlayers

http://services.clarity-h2020.eu:8080/geoserver/clarity/wms?service=WMS&version=1.1.0&request=GetMap&layers=clarity%3Astreams&bbox=1999737.5%2C1015162.5%2C7144137.5%2C5255962.5&width=768&height=633&srs=EPSG%3A3035&format=text%2Fhtml%3B%20subtype%3Dopenlayers

I also understood from her and Luis Torres that most of layers corresponding to land use are also needed for Pluvial Flood formula calculations.

So from that point I am still not 100% sure if all data available is enough to do pluvial flood calculations with such formula. As far as I know yes, but PLINIVS and/or Meteogrid should check formula and data availability to see if anything else is needed.

[p-a-s-c-a-l]

Flood HC Layers are now available in CSIS. See https://github.com/clarity-h2020/local-effects/issues/9#issuecomment-577181444

[negroscuro]

@stefanon From Naples meeting I was told to calculate:

• streams count per cell (done). Regarding this, I have a question. It is just as simple as intersect streams with cells and count coincidences or we shoud have a condition to count streams in order to avoid counting for small pieces of streams in a cell?

• basins altitude per cell (ongoing). Regarding this, I also have a question. The formula I understood is to calculate grid cell mean altitude and then substract minimum altitude of a basin (the one with larger area in the cell) --> que question is, the minimum altitude to substract is the minimum altitude of the basin (which can be outside the cell) or the minimum altitude of the basin area within the cell? (I guess it should be this second option)

[negroscuro]

After a hard day with SQL, I manage to find a query to generate basins data:

update land_use_grid
set basins=x.result
from (
select sq.cell,(ST_SummaryStats(ST_Clip(r.rast, t.cell_geom))).mean-(ST_SummaryStats(ST_Clip(r.rast, ST_Intersection(t.basin_geom,t.cell_geom)))).min as result
from dem_reggio_di_calabria_bilinear r, (
SELECT u.cell,
max(ST_Area(ST_Intersection(b.geom,g.geom))/ST_Area(g.geom)) as percentage
FROM basins b, laea_etrs_500m g, land_use_grid u
WHERE b.city=259 AND b.city=u.city AND u.cell=g.GID AND ST_Area(ST_Intersection(b.geom,g.geom))>0
group by u.cell
ORDER BY u.cell ASC) as sq  
INNER
  JOIN (SELECT u.cell,b.geom as basin_geom, g.geom as cell_geom,
(ST_SummaryStats(ST_Clip(r.rast, g.geom))).mean-(ST_SummaryStats(ST_Clip(r.rast, ST_Intersection(b.geom,g.geom)))).min as result,
max(ST_Area(ST_Intersection(b.geom,g.geom))/ST_Area(g.geom)) as percentage
FROM basins b, dem_reggio_di_calabria_bilinear r, laea_etrs_500m g, land_use_grid u
WHERE b.city=259 AND b.city=u.city AND u.cell=g.GID AND ST_Area(ST_Intersection(b.geom,g.geom))>0
group by u.cell,r.rast,g.geom,b.geom
ORDER BY u.cell ASC) AS t
    ON t.cell=sq.cell
   AND t.percentage=sq.percentage
) as x
where land_use_grid.cell=x.cell;

When I was about to get done with this I realized that:

@stefanon As you may notice in the picture there is a cell with 3 basins intersection where percentages are equal so I do not know which one to take as the majoritary option in order to asign its altitude calculation as the right one to assign... I need another criteria in case of equal majority area intersects the cell...

There could be negative values, I did not spot any but may happen, is that ok?

[NICDDB]

@negroscuro, negative values ​​it is normal that there are, because in the land morphology exist depressed areas. For basins matter the procedure made by us is:

1. assign altitude from DEM for each cell.
2. assign each basin to a cell; did you this? If you have the same parcentage of different basins in same cell, it's not too important where it ends up. I suggest you assign it to the basin with minimum altitude.
3. locate the minimum point (of DEM) for each basin that you have to find in point 2.
4. substract for each cell of the same basin (2.) dem altitude (1.) with minimum point (3.)

@stefanon you confirm too?

[negroscuro]

@negroscuro, negative values ​​it is normal that there are, because in the land morphology exist depressed areas.

Ok!

2. assign each basin to a cell; did you this?

I do not see the point on doing this... what I am doing is to intersect basins and cells, so in can say I have each cell related to a basin, is that enough?

If you have the same parcentage of different basins in same cell, it's not too important where it ends up. I suggest you assign it to the basin with minimum altitude.

Ok

3. locate the minimum point (of DEM) for each basin that you have to find in point 2.

This still not clear to me, I have to locate de basin minimum value or I have to find out which is the basin-cell intersection minimum value?

Thank you!

[stefanon]

This still not clear to me, I have to locate de basin minimum value or the basin-cell intersection minimum value?

Mario the goal is to have for each grid cell the delta height respect to the lowest point of the basin, that is the lowest height of the DEM for the basin: locate the basin minimum height value.

[negroscuro]

This is Linz and its intersecting basins:

I realized I was doing things wrongly since I was only calculating basins altitude considering city area DEM and I have to use DEM for whole area covered by tha basins in order to calculate minimum altitude of a basin. So I have to generate specific DEM for basins as I did for streams which had a similar problem.

[negroscuro]

After some troubles I manage to fix basin altitude calculation for each city cell.

The conclusion is some cells of a city does not have area of intersection with any basin (basin column is null) or DEM raster (altitude colun is null) which makes imposible to calculate this value (minimum column is null) but those are only a few cells of a city.

So take this into account and notice values in ATOS geoserver layer land_use_grid looks like:

The columns are:

• Streams is the stream count for a cell.
• Basin is the corresponding basin ID having the majority of area intersecting with the cell. So it is the one used to calculate the basin altitude.
• Altitude corresponds to mean altitude of a cell
• Built_density is the value obtained as percentage of a cell which represents land use among : "roads" ,"railways" ,"vegetation", "agricultural_areas", "built_open_spaces".
• Minimum is the basin altitude calculation which consist on cell mean altitude minus minimum altitude of the asigned basin for a cell.

@stefanon let me know if you spot any issue or anything I could misunderstood. @humerh just for you to know you can find NULL values and it is something correct from my side, if you have any issue because of this let me know.

Now I will try to regenerate each of 18 current cities pluvial flood data which implies most of those explained columns.

[negroscuro]

I was testing with a small city and it worked but now I moved to a bigger one:

reggio di calabria 2061 cells --> worked kaunas 6884 cells --> failed

This was the error:

BBOX BASINS DEM: 5026200 3385700 5479400 3773500 ...generating DEM for basins of KAUNAS... Input file size is 282250, 216640 0...10...20...30...40...50...60...70...80...90...100 - done. Processing 1/1: /data/KAUNAS/dem/dem_KAUNAS_basins.tif WARNING: The size of each output tile may exceed 1 GB. Use -t to specify a reasonable tile size ERROR: insert_records: Could not allocate memory for INSERT statement ERROR: process_rasters: Could not convert raster tiles into INSERT or COPY statements ERROR: Unable to process rasters

I do not want to think about big capitals with around 30.000-40.000 cells...

This happens because of:

(City boundary in red, basins which overlaps in green)

Some cities overlaps HUGE basins, so basin altitude is calculated by taking the minimum of the basin which can be far away from any city cell. I initially understood this wrongly and I calculated this only by taking the minimum of the basin within each city cell but seems to be wrong. So in the end I am not sure how to solve this, I can only try to generate a less resolution DEM to get basins minimum altitude... what do you think? any other idea?

[stefanon]

@negroscuro Where are you experiencing that error? How's your approach to this calculation?

[negroscuro]

This is an error that happens while generating the DEM tile covering the basins area of the city with GDAL commands. It looks like for that resolution and such a wide area the file is too big and the process consumes too much resources.

The proccess is:

• 1st I find out the coordinates for the bounding box of the basins set overlaping city boundary with SQL
• 2st I generate the raster of DEM for that area with gdal
• 3rd I determine which basins is the one which has the higher overlapping with each city grid cell with SQL
• 4th I calculate the basins altitude by doing cell mean altitude minus basin minimum altitude with SQL (which is minimum value in basin geometries intersecting DEM raster previously calculated)

Now I am trying doing the calculations of cities by using half resolution DEM raster, and seems to be ok:

By the way, is it wrong to find negative basin minimum values? (last column)

[stefanon]

So it's during GDAL processing, what is the resolution you're using now that works? For negative values I suppose the DEM can have depressions under quota 0, can you let me have a look at what raster you're processing? thanks.

[negroscuro]

Yes during GDAL, some kind of resource lacking. Well I reduced resolution to half and seems to be working, now I am trying with heavier/larger cities with more than 30.000 city cells.

The original DEM raster is at ATOS FTP: /clarityftp/europe/eu-dem/eu_dem.zip It is a heavy file which contains 27 TIF's to be unzipped and then I created a tile index shapefile by using a VRT file to access it as a whole map in order to extract specific places. It is located in the same folder as: eu_dem_index.zip in case you need it.

[negroscuro]

Pluvial flood data seems to be correctly generated by last week changes I was working on. Now I am loading the 18 cities with correct pluvial flood data.

[negroscuro]

17 out of 18 cities loaded, but Athina is going to take ages, you can see total time of each city in the picture, notice Helsinki took 34h! This is due to the new UA codes included and the more complex pluvial floods calculations were implemented.

[negroscuro]

I recently updated European data package in CSIS.

I included URLs to check data availbaility for local effect input layers. Remember you can use the map on the following links and click on the desired city boundary to confirm availability since once you click a small table will be shown on the bottom of the web page, under the map component. A city with time value under heat wave or pluvial flood means that hazzard data is available. If nothing is shown on those fields, then city is unavailable.

- Heat wave:

http://services.clarity-h2020.eu:8080/geoserver/clarity/wms?service=WMS&version=1.1.0&request=GetMap&layers=clarity%3Acity&bbox=1969465.0%2C996016.875%2C7168551.0%2C4677889.0&width=768&height=543&srs=EPSG%3A3035&styles=hw_availability&format=text%2Fhtml%3B%20subtype%3Dopenlayers#toggle

(check city boundary color - green is available - red is not available)

- Pluvial flood:

http://services.clarity-h2020.eu:8080/geoserver/clarity/wms?service=WMS&version=1.1.0&request=GetMap&layers=clarity%3Acity&bbox=1969465.0%2C996016.875%2C7168551.0%2C4677889.0&width=768&height=543&srs=EPSG%3A3035&styles=pf_availability&format=text%2Fhtml%3B%20subtype%3Dopenlayers#toggle

(check city boundary color - green is available - red is not available)

- Mortality

http://services.clarity-h2020.eu:8080/geoserver/clarity/wms?service=WMS&version=1.1.0&request=GetMap&layers=clarity%3Amortality&STYLES=mortality&bbox=1995203.0178%2C1014243.9784%2C7079558.6386%2C5270223.5146&width=768&height=642&srs=EPSG%3A3035&format=text%2Fhtml%3B%20subtype%3Dopenlayers#toggle

(check city boundary color - green is available - red is not available)

[negroscuro]

Regarding pluvial floods I was told to include a new parameter into land_use_grid layer which is run_off_average of a cell, it is a heighted average of run_off_coefficient based on percentages of land use within a cell. It is available for all generated cities (which are 19 im total right now)

[p-a-s-c-a-l]

Flood Local Effects HC-LE Resource has been created in CSIS DEV instance and added to European Data Package:

• Service Endpoints
• Data Package Resource
• Resource on Map&emikat_id=3189&datapackage_uuid=2434ce93-93d4-4ca2-8618-a2de768d3f16&write_permissions=1&time_period=20410101-20701231&emissions_scenario=rcp45&event_frequency=Rare&grouping_tag=taxonomy_term--hazards)

Map layers don't show up since I still need to know the the layer name(s) and recalculation is currently not possible. @humerh.

[p-a-s-c-a-l]

Flood Local Effects HC-LE layers now available.