Height correction for (Temperature) local effects

[DenoBeno]

For the full discussion, please see: https://github.com/clarity-h2020/emikat/issues/28

Conclusion: height correction should be implemented for the temperature local effect indices. According to @RobAndGo, the correction should be 1°C per 100m up to 1000m height and 0.65°C above it:

... since this calculation is done for midday conditions with maximum solar radiation, the temperature decrease with height would not be 0.65°C/100m but rather the steeper dry adiabatic lapse rate of 1°C/100m, particularly for elevations below about 1km in height. This represents the depth of a convectively-mixed boundary layer typical on summer days. Above the boundary layer height of 1km, a more sensible temperature decrease would be the rate of 0.65C/100m.

The way to implement this in EMIKAT is by calculating the difference between the average heights of the 500x500 output cells and the average height between 12x12km^2 CORDIS input cells. Thus, we need the following:

• [x] Average heights for the input cells (@claudiahahn , @RobAndGo ?)
• [x] Average heights for the 500x500m^2 output cells (@negroscuro ?)
• [ ] Modify the INPUT temperature for each output cell by delta(H)*(h<1000m ? 1°C/100m : 0.65°C/100m) before doing the local effect calculations.

@mattia-leone , @claudiahahn , please confirm that this is correct. @humerh : please estimate the work needed to implement this once the heights layers are available.

[claudiahahn]

yes, the temperature values Robert provided for the heat wave events are based on the EURO-CORDEX data (~12 x 12km), which use an average height for each 12 x12 km grid cell (H_avg). I have copied the respective orography file (average height for the input cells) to the ftp server (/clarityftp/europe/EURO-CORDEX_orography/SMHI/orog_EUR-11_SMHI_r0i0p0_v1_fx.nc). Using the EU-DEM (25m x 25m resolution) one could calculate an average height for each 500mx500m cell. The temperature correction can then be applied to all 500mx500m cells. Cells that are below the H_avg should be warmer, cells above the H_avg colder.

[negroscuro]

I am dealing with this:

" Average heights for the 500x500m^2 output cells (@negroscuro ?)"

I am assuming there is only need of having mean altitude of a cell as long as it is a cell of an available city in the system, am I right with this?

[RobAndGo]

I believe so. As the local effect can only be calculated where city data is available, the effect of cell-height is thus only relevant for these cases.

[negroscuro]

Thank you @RobAndGo

Mean altitude data logic is already set. I am currently updating cities currently in the system in order to generate cell mean altitudes for those grid cells belonging to any of those cities which are:

Napoli Stockhom Linz Salzburg Linz Wien Barcelona Palermo Reggio di Calabria Bari Bologna Genova Tallinn Kaunas Gdansk Helsinki Jelgava Madrid Athina

Data is in layer 'land_use_grid' under column 'altitude'.

[humerh]

Question to @RobAndGo and @negroscuro

How can I access the height of the Euro-Cordex Grid cell (12kmx12km)? Will there be an Geoservice available? Or should I import this height similar as we did for the "charcteristic events"? I imported many month ago an Excel-File provided by @RobAndGo, like this:

[negroscuro]

There is an issue when loading mean altitude so data is not stable, I am deleting and generating it while finding out what is the problem with it, I am getting erroros:

NOTICE: The input raster and input geometry do not intersect. Returning empty raster NOTICE: Invalid band index (must use 1-based). Returning NULL

It looks like DEM raster and cells are not matching for some reason. The point is I am generating DEM from city boundary and I am selecting grid cells by intersecting with city boundary so it does not make sense to me. I do not know what am I missing but I have to deeply check what is wrong with this so I have to try to visualize generated DEM rasters and grid cells in order to find out what is the problem here.

[RobAndGo]

Hi @humerh. Which heights did you use for the 'characteristic events'? I seem to remember that @claudiahahn may have already uploaded some height information from EURO-CORDEX, possibly for that task which you mentioned.

[claudiahahn]

I have copied the respective file (average height for each 12 x12 km grid cell) to the ftp server (/clarityftp/europe/EURO-CORDEX_orography/SMHI/orog_EUR-11_SMHI_r0i0p0_v1_fx.nc). I am not sure if it was intended to put the file on the wms server (@luis-meteogrid, @ghilbrae, @LauraMTG?). Also, I think, transferring data from MTG to ATOS geoserver is still on-going, right? It might for now be easier if you import the file directly?

[ghilbrae]

You are right, the migration is still not done. If you need us to add it to our geoserver we can do that but it will need to be taken into account by ATOS (@negroscuro ) when the migration is done.

[negroscuro]

You are right, the migration is still not done. If you need us to add it to our geoserver we can do that but it will need to be taken into account by ATOS (@negroscuro ) when the migration is done.

@ghilbrae : If you change anything in Meteogrid Geoserver, please provide an updated copy of the Geoserver folder into the ATOS FTP and let me know about it. Migration is not yet completed since we are having problems with docker image volumes, my colleague @DanielRodera is handling this.

[LauraMTG]

This is more complicated @negroscuro and @DanielRodera

Requests for new publications can be frequent and this leads to changes in several Geoserver directories (not only the one that stores the file, but also the style directory etc). Your current version is no longer updated as I indicated previously when I processed new ZAMG hazard index. We need to discuss a proper way to do the MTG-ATOS geoserver migration before continuing to duplicate efforts. I think it can be discussed during the next developers meeting.

[p-a-s-c-a-l]

@DanielRodera If you need help with the setup of GeoServer as Docker container, you can have a look at our docker-geoserver repo.

Accessing the data volume from dockerized GeoServer is not as obvious as it should be. Some thoughts on that in this issue: https://github.com/clarity-h2020/docker-geonode/issues/4

[negroscuro]

@ghilbrae

This is more complicated @negroscuro and @DanielRodera

Requests for new publications can be frequent and this leads to changes in several Geoserver directories (not only the one that stores the file, but also the style directory etc). Your current version is no longer updated as I indicated previously when I processed new ZAMG hazard index. We need to discuss a proper way to do the MTG-ATOS geoserver migration before continuing to duplicate efforts. I think it can be discussed during the next developers meeting.

I said Geoserver folder because as far as I understood from Laura everything published in Geoserver is within Geoserver folder, that means styles, raster data sources, etc... So the idea is getting that we have all needed data. The idea is to replicate the Geoserver instances as it is running in Meteogrid server, so we need latest version of it.

Until we have anything stable keep working on Meteogrid and add there anything you need.

[negroscuro]

@DanielRodera If you need help with the setup of GeoServer as Docker container, you can have a look at our docker-geoserver repo.

Accessing the data volume from dockerized GeoServer is not as obvious as it should be. Some thoughts on that in this issue: clarity-h2020/docker-geonode#4

@p-a-s-c-a-l Is not a problem about deploying a Geoserver it seems there is a problem with the eteogrid instance, something related with some jar files. Thank you anyway for the support!

[LauraMTG]

Yes, but the latest version is changing several times. Well, I propose the following: I will process Claudia's request I'm going to publish on MTG's geoserver I'm going to update the Geoserver directories on the ATOS FTP.

However, please consider that this process is not feasible for every new layer. To complete the Geoserver migration, count on what you may need from us to avoid this.

[negroscuro]

Forget about updating ATOS FTP for now, it is better for everybody if we ask you once our bug is fixed, this way you will do it just once. By the way, which version of Tomcat are you using?

[LauraMTG]

Thank you @negroscuro , that's good. Tomorrow I'll inform you of Tomcat's version.

and @claudiahahn , your file has already been processed and published on MTG's geoserver (for now)

[negroscuro]

There is an issue when loading mean altitude so data is not stable, I am deleting and generating it while finding out what is the problem with it, I am getting erroros:

NOTICE: The input raster and input geometry do not intersect. Returning empty raster NOTICE: Invalid band index (must use 1-based). Returning NULL

It looks like DEM raster and cells are not matching for some reason. The point is I am generating DEM from city boundary and I am selecting grid cells by intersecting with city boundary so it does not make sense to me. I do not know what am I missing but I have to deeply check what is wrong with this so I have to try to visualize generated DEM rasters and grid cells in order to find out what is the problem here.

It looks like the problem was DEM raster was only generated to match streams area, since DEM use was only with streams. Now it is needed with city grid cells so I manage to generate different area dem to match city grid cells also.

[ghilbrae]

@negroscuro This is the info on our geoserver installation: TOMCAT

server.info=Apache Tomcat/8.5.35
server.number=8.5.35.0

GEOSERVER

GeoServer Version
2.14.0
Git Revision
2ecf6413b0d5d304c7dc23d8d7e9f3cfad29f37c
Build Date
21-Sep-2018 20:18
GeoTools Version
20.0 (rev ffc271f317c04e714ea44a4879dd4601bd723d5e)
GeoWebCache Version
1.14.0 (rev 1.14.x/775e94eb2ae59dd9b74aecc709b67d7b968a4317)

[negroscuro]

There is an issue when loading mean altitude so data is not stable, I am deleting and generating it while finding out what is the problem with it, I am getting erroros:

NOTICE: The input raster and input geometry do not intersect. Returning empty raster NOTICE: Invalid band index (must use 1-based). Returning NULL

It looks like DEM raster and cells are not matching for some reason. The point is I am generating DEM from city boundary and I am selecting grid cells by intersecting with city boundary so it does not make sense to me. I do not know what am I missing but I have to deeply check what is wrong with this so I have to try to visualize generated DEM rasters and grid cells in order to find out what is the problem here.

Finally I fixed the error and mean height per 500m cell is generated for current cities in the system.

I have to let you know that this could be a slow process, it cant take minutes or hours, the slowest of the 18 cities was Stockholm which took 11 hours to generate mean altitude per cell (I am using 25 meters DEM data in order to generate the mean).

Not joking... anyway I am exploring an alternative way to do this using GDAL instead of postgres.

[negroscuro]

This is 25m DEM original data for Stockholm area:

This is Stockholm city boundary defined for UA2012:

For you to see resulting mean altitude, I styled it, so it is 500m resoultion, based on the Europea grid, so, altitude per cell and looks like this:

[negroscuro]

The styling is done with 8 quantile intervals.

Same for Naples, DEM data:

Naples UA boundary;

Mean altitude per cell:

So for me it looks correct, if not, let me know.

[negroscuro]

I have been trying the GDAL way to do this, it takes just seconds! but the result is useless... I cannot match European 500m grid cells with generated pixels with this procedure, indeed it is irregular and I miss data.

ORIGINAL DEM FOR STOCKHOLM SMALL AREA

11 hours POSTGRES calcualtions result

GDAL quick operation result

So I will stick to Postgres method.

[p-a-s-c-a-l]

SELECT g.gid,(ST_SummaryStats(ST_Clip(r.rast, g.geom))).mean FROM land_use_grid lug, laea_etrs_500m g, dem_stockholm r WHERE lug.city=427 AND lug.cell=g.gid

Hm, maybe @therter or @stefanon could have a look at the statement?

[humerh]

I have imported the NetCDF file from @claudiahahn already into emikat.

See: have copied the respective file (average height for each 12 x12 km grid cell) to the ftp server (/clarityftp/europe/EURO-CORDEX_orography/SMHI/orog_EUR-11_SMHI_r0i0p0_v1_fx.nc).

[negroscuro]

SELECT g.gid,(ST_SummaryStats(ST_Clip(r.rast, g.geom))).mean FROM land_use_grid lug, laea_etrs_500m g, dem_stockholm r WHERE lug.city=427 AND lug.cell=g.gid

Hm, maybe @therter or @stefanon could have a look at the statement?

@p-a-s-c-a-l I checked and values are generated correctly but for some reason it takes ages. @DenoBeno I need someone to validate this procedure or decide which other alternative I can use. GDAL is quick but does not match the european grid.

[negroscuro]

age height for each 12 x12 km

@humerh You mean I spent around two weeks for nothing? ouch :( that means change scripts back to not doing this altitude calculations....

[claudiahahn]

@negroscuro, I think Heinrich needs both, the 12x12km average height (H_avg) from the EURO-CORDEX files and the 500x500 m elevation grid.

The temperature correction can then be applied to all 500mx500m cells. Cells that are below the H_avg should be warmer, cells above the H_avg colder.

[p-a-s-c-a-l]

I think Heinrich needs both, the 12x12km average height (H_avg) from the EURO-CORDEX files and the 500x500 m elevation grid.

Reopening. @humerh please clarify asap. This has quite some impact on how we proceed with CSIS Public BETA Release. Thanks.

[humerh]

I need the "12x12km average height (H_avg) from the EURO-CORDEX", which is used also for defining the "characteristic events". This heights are already defined in the NetCDF file provided by @claudiahahn, isn't it? I imported this file into EMIKAT, similar to the import job of the characteristic events.

These heights are the starting points for the height-corrections, because all of the current calculations of temperatures and impact are based on this (event) height. Currently only ONE height is used for each study area.

Additionally I (will) access the "" from the layer "land use grid" for getting the height/altitude of each grid cell.

With this both heights I will make the temperature correction.

I do not understand, what 's the problem?

[negroscuro]

@humerh There is no problem, my fault, I misunderstood a previous message here, so I just want to confirm you use both altitude data sources. Thank you for clarification.

[negroscuro]

I still need someone to validate the altitude I am generating, I am not sure if it worths the time spent to be generated.

I will explain myself in more detail. I have DEM_Stockholm raster file which has 25m resolution.

The current process is to use it and calculate mean altitudes with european grid from postgresql database. This takes 11hours, in this particular city and this is the result. (This is 32000 cells x 1.77 seconds per cell)

If I do the same with GDAL it only takes seconds but it is not matching european grid. So If I reduce resolution with this. It can be used as input of postgresql mean altitude calculation instead of using DEM_Stockholm 25m resolution data. I used bilinear interpolation and half resolution to get this: (This is 32000 cells x 0.48 seconds per cell)

This means a lower resolution input data will take less processing time. What do you think?

[negroscuro]

@humerh and @stefanon Can you confirm me if it would be a problem or not to have the basins altitude per cell, streams count per cell, mean altitude per cell and built density per cell as NULL in those cases there is no data? having null allows me to differenciate if data was generated or not, since sometimes cells does not meet the conditions of having data in order to generate a corresponding value.

It would be ok for you to find null values?

[stefanon]

@negroscuro well, if no data are available or can be generated, the only choice I think is to have null values, and then no data in -> no data out... can you show an example?

[negroscuro]

The thing is avoid any problem in later uses of the data I am generating, but from my side is more confortabe and easy if I set to null all those four columns and then set the value when calculated, the thing is that the value could be 0 so that will be confusing if default value for those columns would also be 0... that is why I want to use null but I do not know if it is a problem for any of you.

[negroscuro]

This is the 11h processing result only by using DEM and postgis operations:

I tried by doing GDAL operation to lower resolution to half original DEM raster, then calculated from postgis and I obtained results in 21minutes instead of 11h... and looks like this, pretty similar but not exactly the same

May I use this shorter method instead?

[humerh]

What is the reason for having NULL values?

• maybe this is water area
• other reason .....

Which replacement value can I use, which strategy should I apply?

• Take Altitude = 0m above sea level
• Take the value of the nearest defined cell?
• Maybe I am not allowed to make height correction for these cells
• All depending results on this grid cell should be also set to NULL.

As a conclusion: My algorithms will be very depended on the background, why the values are missing.

[negroscuro]

@humerh look at this.

As you may notice in the picture the are situations where some of the cells (for stockholm 1022 out of 32509 cells) have no altitude value because there is no DEM intersection to calculate it. For some reason and being different data sources DEM is not overlapping all Urban Atlas land usage.

In the above example you are looking at DEM and Urban Atlas trees. So there is land usage but not DEM in the cell so I cannot calculate altitude but the cell is there and exists so I have to set an altitude value for it. My proposal? to use null since other cells could have an altitude 0 assigned since it is a correct value for it and we can have confusion with this kind of situations.

Or look at this other grup or cells which have water land usage values but no DEM.

[humerh]

Can you show a satellite picture for such areas? What is real situation for such an area??

[negroscuro]

Mmmmh I do not find satelllite imagery WMS services to connect to, do you know any?

[stefanon]

Mmmmh I do not find satelllite imagery WMS services to connect to, do you know any?

This area should be covered by Google / Bing satellite at least. What DEM source are you using?

[negroscuro]

I need a WMS URL to use from QGIS. DEM is the one in ATOS FTP I do not know who provided, it is 25m resolution. I would say it is this from copernicus website: https://land.copernicus.eu/imagery-in-situ/eu-dem/eu-dem-v1.1?tab=download

[stefanon]

I need a WMS URL to use from QGIS.

you can use XYZ Tiles for Google & c. or install Quick Map Services plugin

[stefanon]

I would say it is this from copernicus website

@negroscuro From EU-DEM 1.1 metadata can be read that the dataset is at 25m resolution with vertical accuracy: +/- 7 meters RMSE : so it's not improbable that in very mixed ground / water zones due to the vertical accuracy of the DEM areas that are classified for their landuse are instead shown as water bodies in the DEM.

[negroscuro]

These are water areas where cells do not contain DEM data but there is Urban Atlas data.

This is the tree cell with no DEM data:

In the end my questions are 2:

1- Can I reduce resolution from DEM 25m to half in order to improve time to generate mean altitudes per cell? 2- Can I set to NULL as default values in order to distinguish empty values from 0 values on land_use_grid table/layer columns regarding:

• streams count per cell
• basins altitude per cell
• mean altitude per cell
• built_density

From my side both answers are yes but I do not know if there is any problem from your side.

[stefanon]

1- Can I reduce resolution from DEM 25m to half in order to improve time to generate mean altitudes per cell?

How is the resampling done? An average for reducing the dem resolution shouldn't make a big difference on the result that needs to be calculated on a 500x500m grid

2- Can I set to NULL as default values in order to distinguish empty values from 0 values on land_use_grid table/layer columns regarding:

I agree you set the values to NULL where you've no data for the info you list, starting from DTM height, as a 0 value means something different.

[negroscuro]

1- Can I reduce resolution from DEM 25m to half in order to improve time to generate mean altitudes per cell?

How is the resampling done? An average for reducing the dem resolution shouldn't make a big difference on the result that needs to be calculated on a 500x500m grid

Bilinear. Times got reduced a lot. From 30 minutes to 30 seconds or from 3 hours to 3 minutes.

2- Can I set to NULL as default values in order to distinguish empty values from 0 values on land_use_grid table/layer columns regarding:

I agree you set the values to NULL where you've no data for the info you list, starting from DTM height, as a 0 value means something different.

Nice, lets see if @humerh agrees this. Thank you Stefano.

[negroscuro]

From my side height calculation is done and correct, let me know if you have any issue.

[p-a-s-c-a-l]

Thanks! 👍

[negroscuro]

Now there is an style for cell mean altitudes in layer land use grid:

http://services.clarity-h2020.eu:8080/geoserver/clarity/wms?service=WMS&version=1.1.0&request=GetMap&layers=clarity%3Aland_use_grid&bbox=3082000.0%2C1638500.0%2C5564000.0%2C4282500.0&width=720&height=768&srs=EPSG%3A3035&format=text%2Fhtml%3B%20subtype%3Dopenlayers#toggle