Closed p-a-s-c-a-l closed 4 years ago
It looks fine. We will import data to emikat.
@p-a-s-c-a-l @humerh @stefanon @NICDDB
I just wanted to post an update to the maximum daily precipitation files from my last post.
The format is the same, with only the precipitation values changing. As a reminder, the precipitation values are shown in the last 3 columns and represent the values of maximum daily precipitation (in mm) which correspond to frequent (once per year), occasional (once in 5 years), and rare (once in 20 years) events.
europe_pr_historical_1971-2000.csv.txt europe_pr_rcp26_2011-2040.csv.txt europe_pr_rcp26_2041-2070.csv.txt europe_pr_rcp26_2071-2100.csv.txt europe_pr_rcp45_2011-2040.csv.txt europe_pr_rcp45_2041-2070.csv.txt europe_pr_rcp45_2071-2100.csv.txt europe_pr_rcp85_2011-2040.csv.txt europe_pr_rcp85_2041-2070.csv.txt europe_pr_rcp85_2071-2100.csv.txt
The problems that I fixed were:
Apologies for any additional work that importing this new data is going to cause!!
... Job for next week ...
I think before spending more time on this, the other issues should be addressed with high priority before the 1st webinar on 11/06/2020. Furthermore, it has to be made sure that the implementation of the pf model does not disturb the service, especially not during live demonstration.
Dear @humerh I'm answering your question here, as this is the issue for the flooding :)
concerning the parts of the formulas, like:
select city, max(minimum) max_delta, min(minimum) min_delta, max(minimum), min(minimum) diff_delta, (max(minimum) - min(minimum)) / 50 class_width from landuse_grid_full_200425 group by city
Would it be acceptable to build this min/max aggregation functions for the "STUDY AREA", and not for the CITY? Or is this region for building this min/max defined in another way (like region of river BASIN)?
this aggregation was done on the layer clarity:land_use_grid that I retrieved as WFS from Clarity server. The only attribute in it that discriminates the data for location is this 'City' code number, so I used it, after verifying on map that every code references to what I think is a study area, for doing this aggregation. The area covered by land_use_grid cells for each 'City' code is well bigger than single cities, but I suggest you check the data yourself too.
The aggregation in the "study area" would be the best and also simplest way, because I fetch always the grid cells from the layer "clarity:land_use_grid" exactly for this "study area".
So I will try to implement this formula using this assumption.
@RobAndGo
I finalized the import of the new version of maximum daily precipitation.
This sounds good, but can I ask how do you fetch the cells for the study area from "clarity:land_use_grid"? Am I missing some other data/layer?
The aggregation in the "study area" would be the best and also simplest way, because I fetch always the grid cells from the layer "clarity:land_use_grid" exactly for this "study area". So I will try to implement this formula using this assumption.
so we are talking of very different things. In yellow your study area, in blue the cells with the same City code for Vienna. The aggregation calculations need to be done on this extension of cells for not to leave values out.
Okay, now it is unambiguous! So I need for that an extra fetch for the boundary of the whole city area. :(
I don't know how your work is organized, but I've made a view with the query I shared in the other post, and use it as the pf_propensity layer. Can a similar view be built in the WFS server as a new layer and make the work simpler?
First run of calculation of Preciptation Flooding Propensity for the study area "Urban heat screening Vienna North / AIT | 25"
sorry for closing the issue, this was a mistake.
First run of calculation of Preciptation Flooding Propensity for the study area "Urban heat screening Vienna North / AIT | 25"
@humerh, cutting my map on the study area it seems we're getting the same results, apart from some cells...
the difference should be in the layer class boundaries of the classification used. QGIS uses these rules: for the FIRST CLASS ONLY, it's value >= lower_range AND value <= upper_range for ALL OTHER CLASSES, it's value > lower_range and value <= upper_range
so for the cells you've in red, that I have circled, I read a value of 40 on my map, and as my scale is:
I have 40 in class HIGH, orange color, as HIGH is for values > 30 and <= 40.
Yes, it seems that the classification rules make the difference.
The next steps for calculation of impact are not so clear for me. We have to join these results to the propability of the events and to the costs. Maybe you have an instruction for this.
Dear @RobAndGo
about the data of the daily precipitation files, can I found somewhere a vector grid of cells for the points you have produced (I suppose they are the centroids of the grid cells)?
@p-a-s-c-a-l @humerh @stefanon @NICDDB
I just wanted to post an update to the maximum daily precipitation files from my last post.
The format is the same, with only the precipitation values changing. As a reminder, the precipitation values are shown in the last 3 columns and represent the values of maximum daily precipitation (in mm) which correspond to frequent (once per year), occasional (once in 5 years), and rare (once in 20 years) events.
The next steps for calculation of impact are not so clear for me. We have to join these results to the probability of the events and to the costs. Maybe you have an instruction for this.
I've done the part that considers the rain probability for Naples, and I'm working on extending to the new precipitation data set available, with proper scaling and classification.
Sorry @RobAndGo but I have another question...
The problems that I fixed were: There was too much land cut off from the coasts, e.g. Naples was also affected
I managed to find a way to assign precipitation values from your "europe_pr" data files to the cells of land_use layer to be used in the calculation, but I've found that the problem of land cut off from the coasts still happens, and not only for Naples. These are two sample for Naples and Athens, from europe_pr_rcp45_2041-2070 dataset, RX1 day Occasional (I've not checked all the datasets but happens for several coast places):
any suggestion to solve this? thanks!
Dear @stefanon, Thanks for checking the data and pointing out this issue!! Robert is on vacation this week, which is why he did not respond yet. He will be back next week.
I think it should be possible to use another mask, when cutting out the sea part. If not, it would also be an option to provide data including the sea. This would resolve issues on the coast. But Robert knows best what is easier to provide and how long it will take to calculate it again for the whole of Europe.
Dear @claudiahahn thank you for your answer and info, for me this is not a big problem, I can continue to work anyway and then use the revised data when they'll be available.
Hi @stefanon I have attached here the unmasked model data for the 3 future periods with the 3 rcp scenarios here:
events_20110101-20401231_rcp26_final.csv.txt events_20110101-20401231_rcp45_final.csv.txt events_20110101-20401231_rcp85_final.csv.txt events_20410101-20701231_rcp26_final.csv.txt events_20410101-20701231_rcp45_final.csv.txt events_20410101-20701231_rcp85_final.csv.txt events_20710101-21001231_rcp26_final.csv.txt events_20710101-21001231_rcp45_final.csv.txt events_20710101-21001231_rcp85_final.csv.txt
Just a note, that there may be jumps in values along the coast, as only the data over land has been bias-corrected. It was not possible to bias-correct the data over water, which is why it was masked out in the first place.
Hi @RobAndGo and @humerh
I managed to include the precipitation data in the flooding propensity variable, the thing required a rearrangement of the procedure I've developed before, with also a bug fix, and the results are like this:
PF prop. frequent event RCP45 2040-2071
PF prop. rare event RCP45 2040-2071
Updated Pluvial Flood model that includes the precipitation data.
I’ve completed the evaluation of the proxy variable ‘propensity to flooding’ that was first detailed in this ‘off-topic’ post here: https://github.com/clarity-h2020/emikat/issues/35#issuecomment-620128024
The refinement done allows now to consider the effect of the mm of rain for the events in the study period, and also cleaned-up one bug in the sample code I’ve shared.
The input parameters needed for the PF propensity evaluation are the following:
• city code – the id given for each city extended area for which the • streams – the number of streams that are found in a cell • minimum – this cell attribute specify the difference in altitude between the cell, and the lowest point of the basin of belonging • runoff_avg – the value of the runoff coefficients in the cell weighted on the area of the cell
N.B.: They are all available in the layer landuse_grid
As these parameters have a range of values that are widely spread in different cities and depends in some cases on the position of the study area, I wrote some procedures to produce a classification in the range 1 to 5 of all these parameters, so that a unique map scale can be used for the different cities:
• cl_streams - this is the cell attribute 'streams' classified on values 1 - 5 • cl_delta_elev - this is the attribute 'minimum' classified on values 1 - 5 • cl_runoff - this is the attribute 'runoff_avg' classified on values 1 – 5
Previously I’d erroneously used classes from 0 to 5, this is now changed, also in the prototype code I’ll share hereafter.
The additional parameter added is the rain mm per land-use grid cells, that I’ve obtained from the maximum daily precipitation files, unmasked grid, published here https://github.com/clarity-h2020/emikat/issues/36#issuecomment-648001602
The procedure precipitation_per_cell selects the needed precipitation dataset grid points based on the extents of the grid cells for given city code, and PR dataset, and then attribute to the land grid cells the value of rain mm using a Nearest Neighbour attribution method.
A classification procedure for rain mm per cell is added to the others so that we have:
Streams --> CL_streams Minimum --> CL_delta_elev Runoff_avg --> CL_runoff Rain_thr_mm --> CL_rain
All parameters with values in the range 1..5
For each rain event for the selected RCP and period (frequent, occasional, rare) the PF propensity is calculated as:
*PF_prop = cl_delta_elev (cl_streams + cl_delta_elev + cl_runoff + cl_rain)**
Here you can find the prototype code I've written in PostgreSQL and that can be adapted to use in EMIKAT. The single query previously shared was rewritten to be more readable. I'm using a local copy of the CLARITY layer land_use_grid (named landuse_grid_full_200425), and the precipitation datasets were imported each in a single table, like 'europe_pr_rcp45_2071-2100'.
-- function --
CREATE OR REPLACE FUNCTION clarity.precipitation_per_cell(citycode integer, pr_dataset character varying)
RETURNS TABLE(cell bigint, distance double precision, city bigint, country character varying, "urban area"
character varying, "RCP" character varying, "Period" character varying,
rx1_f integer, rx1_o integer, rx1_r integer, geom geometry)
LANGUAGE plpgsql
AS $function$
--
-- This function assigns the precipitation value to the grid_cells for selected CITY code and precipitation dataset
--
declare
aSQL varchar; -- sql text query storage
begin
aSQL := 'select distinct on (cell) cell, st_distance(st_centroid(lgf.geom), eps.geom) distance,'||
' lgf.city, eps.country, eps."urban area" , eps."RCP" , eps."Period" ,'||
' eps."RX1day_mm Frequent(1.0)" rx1_f, eps."RX1day_mm Occasional(0.2)" rx1_o, eps."RX1day_mm Rare(0.05)" rx1_r, '||
' lgf.geom '||
' from clarity.landuse_grid_full_200425 lgf, '||
' ('||
' select epr.id, st_transform(epr.geom, 3035) geom, epr.country, epr."urban area", '||
' epr."RCP" , epr."Period" , epr."RX1day_mm Frequent(1.0)" , epr."RX1day_mm Occasional(0.2)" , epr."RX1day_mm Rare(0.05)" '||
' from clarity.'|| pr_dataset ||' epr ,'||
' ('||
' select st_buffer(ST_SetSRID(st_concavehull(st_union(lgf.geom), 0.95), 3035), 8500) geom from '||
' clarity.landuse_grid_full_200425 lgf '||
' where lgf.city = '|| citycode ||' ) q'||
' where st_intersects(st_transform(epr.geom, 3035), q.geom)'||
' ) eps '||
' where st_dwithin(st_centroid(lgf.geom), eps.geom, 18000.0) and lgf.city = '||citycode||' '||
' and eps."RX1day_mm Frequent(1.0)" is not null'||
' ORDER BY lgf.cell, st_distance(st_centroid(lgf.geom), eps.geom) ';
-- raise notice '%', aSQL;
return QUERY EXECUTE aSQL;
END;
$function$
;
-- support views --
CREATE OR REPLACE VIEW clarity.v_clthr_elevation
AS SELECT q.city,
q.max_delta,
q.min_delta,
q.diff_delta,
q.class_width,
q.min_delta + q.class_width AS c_vhigh,
q.min_delta + 3::double precision * q.class_width AS c_high,
q.min_delta + 8::double precision * q.class_width AS c_med,
q.min_delta + 16::double precision * q.class_width AS c_low,
q.min_delta + 50::double precision * q.class_width AS c_vlow
FROM ( SELECT landuse_grid_full_200425.city,
max(landuse_grid_full_200425.minimum) AS max_delta,
min(landuse_grid_full_200425.minimum) AS min_delta,
max(landuse_grid_full_200425.minimum) - min(landuse_grid_full_200425.minimum) AS diff_delta,
(max(landuse_grid_full_200425.minimum) - min(landuse_grid_full_200425.minimum)) / 50::double precision AS class_width
FROM clarity.landuse_grid_full_200425
GROUP BY landuse_grid_full_200425.city) q;
CREATE OR REPLACE VIEW clarity.v_clthr_streams
AS SELECT q.city,
q.max_streams,
q.min_streams,
q.diff_streams,
q.class_width,
q.min_streams + 2::double precision * q.class_width AS c_vlow,
q.min_streams + 4::double precision * q.class_width AS c_low,
q.min_streams + 6::double precision * q.class_width AS c_med,
q.min_streams + 8::double precision * q.class_width AS c_high,
q.min_streams + 10::double precision * q.class_width AS c_vhigh
FROM ( SELECT landuse_grid_full_200425.city,
max(landuse_grid_full_200425.streams) AS max_streams,
min(landuse_grid_full_200425.streams) AS min_streams,
max(landuse_grid_full_200425.streams) - min(landuse_grid_full_200425.streams) AS diff_streams,
(max(landuse_grid_full_200425.streams) - min(landuse_grid_full_200425.streams)) / 10::double precision AS class_width
FROM clarity.landuse_grid_full_200425
WHERE landuse_grid_full_200425.streams IS NOT NULL
GROUP BY landuse_grid_full_200425.city) q;
CREATE OR REPLACE VIEW clarity.v_clthr_runoff
AS SELECT q.city,
q.max_roff,
q.min_roff,
q.diff_roff,
q.class_width,
q.min_roff + q.class_width AS c_vlow,
q.min_roff + 3::double precision * q.class_width AS c_low,
q.min_roff + 8::double precision * q.class_width AS c_med,
q.min_roff + 16::double precision * q.class_width AS c_high,
q.min_roff + 50::double precision * q.class_width AS c_vhigh
FROM ( SELECT landuse_grid_full_200425.city,
max(landuse_grid_full_200425.run_off_average) AS max_roff,
min(landuse_grid_full_200425.run_off_average) AS min_roff,
max(landuse_grid_full_200425.run_off_average) - min(landuse_grid_full_200425.run_off_average) AS diff_roff,
(max(landuse_grid_full_200425.run_off_average) - min(landuse_grid_full_200425.run_off_average)) / 50::double precision AS class_width
FROM clarity.landuse_grid_full_200425
GROUP BY landuse_grid_full_200425.city) q;
-- sample query that evaluates all the needed data for a given city code (244 : Naples in the sample) and RPC and time period for precipitation, returning all the landuse parameters, plus the rain per cell, all the said PF parameters classified in the range 1..5 and the PF propensity variables, pf_prop_f, pf_prop_o, pf_prop_r for frequent, occasional, rare events:
SELECT qq.id,
qq.geom,
qq.fid,
qq.cell,
qq.city,
qq.water,
qq.roads,
qq.railways,
qq.trees,
qq.vegetation,
qq.agricultural_areas,
qq.sports,
qq.built_open_spaces,
qq.dense_urban_fabric,
qq.medium_urban_fabric,
qq.low_urban_fabric,
qq.public_military_industrial,
qq.streams,
qq.basin,
qq.altitude,
qq.built_density,
qq.minimum,
qq.run_off_average,
qq.cell_name,
qq.rx1_f,
qq.rx1_o,
qq.rx1_r,
qq.cl_delta_elev,
qq.cl_streams,
qq.cl_runoff,
qq.cl_f_rain,
qq.cl_o_rain,
qq.cl_r_rain,
qq.cl_delta_elev * (qq.cl_streams + qq.cl_delta_elev + qq.cl_runoff + qq.cl_f_rain) AS pf_prop_f,
qq.cl_delta_elev * (qq.cl_streams + qq.cl_delta_elev + qq.cl_runoff + qq.cl_o_rain) AS pf_prop_o,
qq.cl_delta_elev * (qq.cl_streams + qq.cl_delta_elev + qq.cl_runoff + qq.cl_r_rain) AS pf_prop_r
FROM ( SELECT lgf.id,
lgf.geom,
lgf.fid,
lgf.cell,
lgf.city,
lgf.water,
lgf.roads,
lgf.railways,
lgf.trees,
lgf.vegetation,
lgf.agricultural_areas,
lgf.sports,
lgf.built_open_spaces,
lgf.dense_urban_fabric,
lgf.medium_urban_fabric,
lgf.low_urban_fabric,
lgf.public_military_industrial,
lgf.streams,
lgf.basin,
lgf.altitude,
lgf.built_density,
lgf.minimum,
lgf.run_off_average,
lgf.cell_name,
q_rain.rx1_f,
q_rain.rx1_o,
q_rain.rx1_r,
CASE
WHEN lgf.minimum <= vce.c_vhigh THEN 5
WHEN lgf.minimum > vce.c_vhigh AND lgf.minimum <= vce.c_high THEN 4
WHEN lgf.minimum > vce.c_high AND lgf.minimum <= vce.c_med THEN 3
WHEN lgf.minimum > vce.c_med AND lgf.minimum <= vce.c_low THEN 2
WHEN lgf.minimum > vce.c_low AND lgf.minimum <= vce.c_vlow THEN 1
ELSE NULL::integer
END AS cl_delta_elev,
CASE
WHEN lgf.streams <= vcs.c_vlow THEN 1
WHEN lgf.streams > vcs.c_vlow AND lgf.streams <= vcs.c_low THEN 2
WHEN lgf.streams > vcs.c_low AND lgf.streams <= vcs.c_med THEN 3
WHEN lgf.streams > vcs.c_med AND lgf.streams <= vcs.c_high THEN 4
WHEN lgf.streams > vcs.c_high AND lgf.streams <= vcs.c_vhigh THEN 5
ELSE NULL::integer
END AS cl_streams,
CASE
WHEN lgf.run_off_average <= roff.c_vlow THEN 1
WHEN lgf.run_off_average > roff.c_vlow AND lgf.run_off_average <= roff.c_low THEN 2
WHEN lgf.run_off_average > roff.c_low AND lgf.run_off_average <= roff.c_med THEN 3
WHEN lgf.run_off_average > roff.c_med AND lgf.run_off_average <= roff.c_high THEN 4
WHEN lgf.run_off_average > roff.c_high AND lgf.run_off_average <= roff.c_vhigh THEN 5
ELSE NULL::integer
END AS cl_runoff,
CASE
WHEN q_rain.rx1_f::double precision <= 60::double precision THEN 1
WHEN q_rain.rx1_f::double precision > 60::double precision AND q_rain.rx1_f::double precision <= 75::double precision THEN 2
WHEN q_rain.rx1_f::double precision > 75::double precision AND q_rain.rx1_f::double precision <= 90::double precision THEN 3
WHEN q_rain.rx1_f::double precision > 90::double precision AND q_rain.rx1_f::double precision <= 105::double precision THEN 4
WHEN q_rain.rx1_f::double precision > 105::double precision AND q_rain.rx1_f::double precision <= 120::double precision THEN 5
ELSE 5
END AS cl_f_rain,
CASE
WHEN q_rain.rx1_o::double precision <= 60::double precision THEN 1
WHEN q_rain.rx1_o::double precision > 60::double precision AND q_rain.rx1_o::double precision <= 75::double precision THEN 2
WHEN q_rain.rx1_o::double precision > 75::double precision AND q_rain.rx1_o::double precision <= 90::double precision THEN 3
WHEN q_rain.rx1_o::double precision > 90::double precision AND q_rain.rx1_o::double precision <= 105::double precision THEN 4
WHEN q_rain.rx1_o::double precision > 105::double precision AND q_rain.rx1_o::double precision <= 120::double precision THEN 5
ELSE 5
END AS cl_o_rain,
CASE
WHEN q_rain.rx1_r::double precision <= 60::double precision THEN 1
WHEN q_rain.rx1_r::double precision > 60::double precision AND q_rain.rx1_r::double precision <= 75::double precision THEN 2
WHEN q_rain.rx1_r::double precision > 75::double precision AND q_rain.rx1_r::double precision <= 90::double precision THEN 3
WHEN q_rain.rx1_r::double precision > 90::double precision AND q_rain.rx1_r::double precision <= 105::double precision THEN 4
WHEN q_rain.rx1_r::double precision > 105::double precision AND q_rain.rx1_r::double precision <= 120::double precision THEN 5
ELSE 5
END AS cl_r_rain
FROM clarity.landuse_grid_full_200425 lgf,
( SELECT precipitation_per_cell.cell,
precipitation_per_cell.city,
precipitation_per_cell."RCP",
precipitation_per_cell."Period",
precipitation_per_cell.rx1_f,
precipitation_per_cell.rx1_o,
precipitation_per_cell.rx1_r,
precipitation_per_cell.geom
FROM clarity.precipitation_per_cell(244, '"europe_pr_rcp45_2041-2070"'::character varying)
precipitation_per_cell(cell, distance, city, country, "urban area", "RCP", "Period", rx1_f, rx1_o, rx1_r, geom)) q_rain,
clarity.v_clthr_elevation vce,
clarity.v_clthr_streams vcs,
clarity.v_clthr_runoff roff
WHERE lgf.cell = q_rain.cell AND vce.city = 244 AND vcs.city = 244 AND roff.city = 244) qq;
A sample result of the above model is:
PF prop. frequent event RCP45 2040-2071 :
PF prop. occasional event RCP45 2040-2071 :
PF prop. rare event RCP45 2040-2071 :
where rain cell distribution for these events is:
This is the style used for the PF propensity variables (case rare) in SLD format
but the classes limits and colors can be found in this image too:
Now I have finished the implementation of Pluvial flood in version 1.0. It includes the local effects, the impact calculation, the preparation for adaptation options and the costs.
See: https://github.com/clarity-h2020/csis/wiki/Services-endpoints-(used-by-CSIS)
done
I have generated the daily precipitation values for the three precipitation events that we (@claudiahahn @mattia-leone @stefanon @NICDDB) talked about last week, which are necessary for the PLINIVS flood local effect model.
As an example of what the data means, the following excel table shows an example for the Naples grid point, with each sheet (10 in total) representing either the baseline climate or the 9 future time periods/RCP combinations. The tables show the number of precipitation events (top table) in the 30-year period where the daily precipitation is greater than or equal to a threshold value (columns). The rows show the number of consecutive days where the threshold precipitation is reached. The lower table shows the probability of occurrence in 1 year.
precipitation-matrix_naples.xlsx
From this data, three precipitation events (similar to the heat wave events) have been defined:
The daily precipitation which coincides with these three events have been found for all grid points in Europe for the baseline climate and the 9 future time periods/RCP combinations. I have attached the csv-files with extension .txt below.
europe_pr_eobs_1971-2000.csv.txt europe_pr_rcp26_2011-2040.csv.txt europe_pr_rcp26_2041-2070.csv.txt europe_pr_rcp26_2071-2100.csv.txt europe_pr_rcp45_2011-2040.csv.txt europe_pr_rcp45_2041-2070.csv.txt europe_pr_rcp45_2071-2100.csv.txt europe_pr_rcp85_2011-2040.csv.txt europe_pr_rcp85_2041-2070.csv.txt europe_pr_rcp85_2071-2100.csv.txt
As for the heat wave files, the format here is:
The daily precipitation values in the last 3 columns are in units of mm and, in this example, show that:
The idea is that this data can be incorporated into EMIKAT by @humerh in a similar way as the heat wave event data, and subsequently accessed by the PLINIVS flood local effect model.
Originally posted by @RobAndGo in https://github.com/clarity-h2020/emikat/issues/35#issuecomment-626919149