Create Local Effects Input Layers for Heat Waves

[p-a-s-c-a-l]

We are still missing some layers for being able to calculate the Local Effects:

• surface temperature
• direct shortwave radiation
• global shortwave radiation
• diffuse shortwave radiation
• sun's altitude angle above the horizon
• sun's azimunth angle

We need a resource (in the form of gridded data) that indicates the percentages (in each cell) of the emissivity, etc. --> action(METEOGRID+ATOS)

[negroscuro]

Still waiting for PLINIVS to check and validate the layers to know if we can close the issue.

[ghost]

Hi Mario,

I've just revised the input layers. The overlapping problem is solved but now a new errors are appeared: 1) During the substraction operatio you lost some important pieces. I attach some pictures as example;

2) I don't know which UA dataset are you using (I think UA 2006 since the agricultural land is the most extended areas) but you are not considering a portion of information. Example: the roads should be taken just from UA but they don't overlap with UA 2012

and with UA 2006

This cannot depend from the substraction procedure because all the information of that areas are missed.

3) you still have some issue with the vegetation layer I overlap this part with my landuse and I do not have this problem.

4) Since the information taken from Urban Atlas dataset covers a smaller space you should cut the outcome using the mask that you downloaded in UA data package in order to avoid the following effect:

Cheers, Ale

[negroscuro]

I am using all layers from UA2012, is this wrong? Can you specify where are we looking at? because it will help to know if I try any fix to see if wrong geometries change... Anyway I need more clear explanations/more detailed of each issue you spotted, because in the attached images I do not know which areas or layers are missing data. thank you.

[negroscuro]

I just tried to see original data from Napoli ROADS(12210 and 12220), I filtered original URBAN ATLAS 2012 data and I got this:

As you may notice roads in red are from original UA2012 data set.

If I do the same, replacing my genetated railways and using original ones fomr UA2012:

So I am not sure if you were referring to roads and railways or what is the issue here.

[stefanon]

In this particular case (roads) the problem is due to the boundaries of UA2012 dataset: the roads shown are cut on the boundary perimeter, while other data shown are not.

[negroscuro]

I am not sure to understand the problem, original roads data as I shown on previous post depicts data available, those are the roads in UA original data... so I do not see what is lost on generation process. Maybe I still do not understand something?

I think a telco (skype between PLINIVS and ATOS, maybe Heinrich, but he is on vacations for two weeks) is urgently needed about all this if you agree.

[stefanon]

Mario, the UA 2012 dataset is cut on the layer IT003L3_NAPOLI_UA2012_Boundary (that is part of the download). Other WMS layers (here you can see 'trees') have a bigger coverage, so it looks like some pieces of roads are missing. (Plinivs is in it's summer holiday period too).

[negroscuro]

I think there is a misunderstood. Alessandra point 1 and point 4, looks the same to me. I mean Stefano, this discussion is because I did not cut all data using Naples boundaries, because we have data from other sources that brings data of a wider area than the Naples boundaries, so we have more vegetation, but roads are not missing, roads are the UA roads, there are no other roads data anywere. So I can cut vegetation/trees to fit Naples boundaries, but I do not see other fix than that, indeed it is not a fix but forget about data off the boundaries, so may I do that? it was not explained anywhere as part of the process but can be done.

[negroscuro]

Regarding Alessandra's answer point 2.

This is data from UA2006 and compared with data I generated looks like there are agricultural areas missing, but the thing is I am not using UA2006 but 2012, so if I check UA2012 original data all seems to be correct:

For some reason agricultural areas are smaller but everything looks to be there in my calculated layers.

[stefanon]

I think there is a misunderstood.

I think too.

In Alessandra post, point 1, first image, I attempted to outline that the missing parts of roads in the picture were due to out of UA2012 boundaries data, and not to data elaboration operations. As you said the data was not available in UA dataset. And indeed Alessandra at point 4. suggested to clip the processing result with the boundaries of UA data to have a consistent result. About the other points I've not all the data with me so can't be of any help.

Alessandra point 1 and point 4, looks the same to me. I mean Stefano, this discussion is because I did not cut all data using Naples boundaries, because we have data from other sources that brings data of a wider area than the Naples boundaries, so we have more vegetation, but roads are not missing, roads are the UA roads, there are no other roads data anywere. So I can cut vegetation/trees to fit Naples boundaries, but I do not see other fix than that, indeed it is not a fix but forget about data off the boundaries, so may I do that?

[negroscuro]

Another question, to work with city boundaries and be able to cut all data from other data sources apart from UA, I need to know which boundary shapefile is the correct one since each UA2012 download has 2 different boundary shapefiles, see the attached image (there are cityBoundary and boundary vector data and does not look obvious since they cover different areas):

[negroscuro]

This change of using city boundaries implies relevant changes on the scripts, so it would be nice to know which shapefile use as soon as possible. Thank you.

[stefanon]

UA2012 data set contains 2 boundaries files: one with suffix 'Boundary' that is the boundary of the UA2012 Naples data layer, and one with suffix 'CityBoundary' that contains the area of the cities, greater cities and larger urban zones, and that extends outside the area covered by UA2012 Naples, so this can't be the one to choose. Also note that DC1 Metropolitan city of Naples area is smaller and fully covered by the UA2021 Boundary and data coverage.

Another question, to work with city boundaries and be able to cut all data from other data sources apart from UA, I need to know which boundary shapefile is the correct one since each UA2012 download has 2 different boundary shapefiles, see the attached image (there are cityBoundary and boundary vector data and does not look obvious since they cover different areas):

[negroscuro]

thank you Stefano

[negroscuro]

I think I got it, I hope this looks better:

So I will try to regenerate all Napoli data and after that other CS cities. But I think there were other issues I still do not understad.

[negroscuro]

I noticed european grid has some inconsistences compared to UA boundary, notice yellow parts of water, those are outside any european grid cell:

So information in UA exist, for water but in some places where european grid does not have any cell making calculations of my scripts not to bring any information in such places for any land usage...

So this is beyond my competences, I do not know if someone else would like to modify the european grid or in the end if doing that makes sense.

[negroscuro]

Napoli data is generated and online, I need PLINIVS to do another review of data and explanations o any pending issue.

--I spotted an error in vegetation so I am currently regenerating data (takes around 8 hours)--

[negroscuro]

I think I have a solution, (light green is old vegetation and darker is new vegetation intersected with city boundary)

Now I am regenerating data, hope to have it on monday.

[ghost]

Hi Mario, as Stefano already told you, all PLINIVS team, including Stefano and me, is on holidays until beginning of September, therefore I don't have all data with me.

Nevertheless, I will quickly reply to some your questions:

1) "I noticed european grid has some inconsistences compared to UA boundary, notice yellow parts of water, those are outside any european grid cell ...." This is not a problem, because we are not considering the see in our algorithm.

2) I am using all layers from UA2012, is this wrong? It's correct, you have to use all the layers of UA2012

3) "So I can cut vegetation/trees to fit Naples boundaries, but I do not see other fix than that, indeed it is not a fix but forget about data off the boundaries, so may I do that? it was not explained anywhere as part of the process but can be done." I explained it to you during the co-creation meeting in Naples in February, so you should find it in your notes.

4) Alessandra point 1 and point 4, looks the same to me. No, point 1 and 4 are not the same: Point 4 is related to cut of the boundaries (as Stefano explained to you), while point 1 is related to the missed parts. As I told you i didn't know which datasets you were using and I compared your layers with both UA 2006 and UA2012. In particular, I don’t find any discontinuities or lack in geometries in UA2012 dataset. The UA2012 original roads data that I used to develop the model are reported in the following picture (I reported the legend in each figure so you can see that I'm talking about the original data): And consequently, clarity layer (red) is not exactly overlapping UA2012 roads layer (grey): I see the same issue also for other layers, like, i.e., railways (clarity layer - light blue, UA2012 – black) or agricultural lands, ecc. ecc.

At moment I cannot say why your UA2012 data is different. Maybe, UA2012 has been modified during the last year since I developed the model more than 1 year ago and, conseguently, I downloaded UA2012 more than 1 year ago, or, maye, there are different datapackages that can be downloaded. If I have time, I will try to investigate the problem.

[stefanon]

Dear Alessandra, the IT003L3_NAPOLI.zip UA2012 file I downloaded 6th of August from Copernicus contains a layer IT003L3_NAPOLI_UA2012 (that is cut on the boundary layer as already discussed) instead of a IT003L2_NAPOLI_UA2012 we can see in your screen captures: I suppose they are different versions datasets, with the present one having roads and other data cut and showing as in your 'red' 'clarityroads' and not like your grey ones.

[negroscuro]

Hi Mario, as Stefano already told you, all PLINIVS team, including Stefano and me, is on holidays until beginning of September, therefore I don't have all data with me.

Hi, Sorry to annoy you during your holydays, I will wait if there is no one from PLINIVS able to answer my questions, but please understand that after all we listen from PLINIVS blaming Atos at Wien meeting last june I cannot allow not to progress on this untill september. So I have to raise questions to know how to progress on this, for me is weekend as well and I am working... I have to take the most of the time because days are passing and spend quite some time waiting to understand why things are wrong and then be able to try to correct the issues.

Nevertheless, I will quickly reply to some your questions:

1. "I noticed european grid has some inconsistences compared to UA boundary, notice yellow parts of water, those are outside any european grid cell ...." This is not a problem, because we are not considering the see in our algorithm.

ok.

2. I am using all layers from UA2012, is this wrong? It's correct, you have to use all the layers of UA2012

ok.

3. "So I can cut vegetation/trees to fit Naples boundaries, but I do not see other fix than that, indeed it is not a fix but forget about data off the boundaries, so may I do that? it was not explained anywhere as part of the process but can be done." I explained it to you during the co-creation meeting in Naples in February, so you should find it in your notes.

Please this is second time asking you stop blaming me because taking or not taking notes about this, it does not help in any way. Having the knowledge I think is your responsability to provide a clear document including and explaining all those steps, because it is being difficult to gather and follow all process details/methodology, the excel is not enough since this kind of things are not there.

4. Alessandra point 1 and point 4, looks the same to me. No, point 1 and 4 are not the same: Point 4 is related to cut of the boundaries (as Stefano explained to you), while point 1 is related to the missed parts. As I told you i didn't know which datasets you were using and I compared your layers with both UA 2006 and UA2012. In particular, I don’t find any discontinuities or lack in geometries in UA2012 dataset. The UA2012 original roads data that I used to develop the model are reported in the following picture (I reported the legend in each figure so you can see that I'm talking about the original data):

I am afraid images are too small for me to see the legend :(

And consequently, clarity layer (red) is not exactly overlapping UA2012 roads layer (grey):

Looking at those images I see different roads because of the Naples city boundary, that is what they look different, missing roads are off boundaries. Stefano explained this to me in previous posts here.

I see the same issue also for other layers, like, i.e., railways (clarity layer - light blue, UA2012 – black) or agricultural lands, ecc. ecc.

At moment I cannot say why your UA2012 data is different. Maybe, UA2012 has been modified during the last year since I developed the model more than 1 year ago and, conseguently, I downloaded UA2012 more than 1 year ago, or, maye, there are different datapackages that can be downloaded. If I have time, I will try to investigate the problem.

Yes please, check with current available data when you have the chance because maybe it is not an issue but just we are comparing different datasets...

Thank you for the answer Alessandra, really appreciated, enjoy your holidays

[negroscuro]

I tried to locate this but i cannot find it... to check why a part is missing...

Anyway I tried to check all railways and honestly for me is really weird to see diconnected railways on my result, I mean, it does not make sense:

But if I filter original UA2012 data to see railways is what it seems to be there... so nothing was lost on the processing and seems to be the same data:

[negroscuro]

Agricultural lands you pointed:

Are correct, since those parts you think I am missing are off city boundary (grey big polygon)):

So as I understood from Stefano explanation, all data off boundary is useless and have to be removed.

Anyway regarding agricultural areas is IMHO due to be using (you and me) different datasets, since issues of having data off boundary happens only when using data from other sources than UA, since all datasets from UA come clipped to fit the boundary they include/define in their datasets. So I only cut when using STL or ESM data sources, (and agricultural areas are only comming from UA).

[negroscuro]

Naples is already loaded with correction of boundary.

[stefanon]

Dear Mario, let me try to answer some of your posts...

Sorry to annoy you during your holydays, I will wait if there is no one from PLINIVS able to answer my questions, but please understand that after all we listen from PLINIVS blaming Atos at Wien meeting last june I cannot allow not to progress on this untill september. So I have to raise questions to know how to progress on this, for me is weekend as well and I am working... I have to take the most of the time because days are passing and most of the time I am waiting to understand why things are wrong and then be able to try to correct the issues.

From my point of view you're having support and answers also if, as already told, PLINIVS, that is located in an university campus, is in it's (only) holiday period (linked to the University holidays), unlike others that are not reachable during their holidays. About the work delay and strict times etc I think it's not due to PLINIVS, and moreover for me too is weekend and my holiday days too, and I'm writing to you to possibly help you doing your work. About the lack of details in Alessandra document describing the steps to be done, I think it was detailed enough to describe what to do and on what data to the IT partner in charge: if you (keep in mind I identify you with your IT services company) instead tought it wasn't like that, a discussion should have been raised publicly long time ago, to analyze the possible problematic points.

Looking at those images I see different roads because of the Naples city boundary, that is what they look different, missing roads are off boundaries. Stefano explained to me in previous posts here.

I think it's at this point clear that the UA datasets used are different, so direct comparison from Alessandra and your results can't be done.

Anyway I tried to check all railways and honestly for me is really weird to see diconnected railways on my result, I mean, it does not make sense:

the railway example too, if I identified it correctly, differs due to the difference in UA2012 layers. About your comment on the disconnected railways, they represent only the open air segments, the discontinituies are tunnel segments.

Are correct, since those parts you think I am missing are off city boundary (grey big polygon)):

to be precise the 'boundary' we're talking about sourced from Copernicus is an extension of the Metropolitan City of Naples, including some other municipalities of other provinces.

all data off boundary is useless and have to be removed.

they are 'useless' due to the fact that outside this boundary there aren't UA2012 data in this new dataset, and for model calculations all parameters involved are needed.

Naples is already loaded with correction of boundary.

Good. If I can add, I think you should be more confident in your work, you can't have to stop and wait for feedback every now and then: if you've clear the procedure and implemented it, be your own judge, evaluate if results are consistent with the specs and their use and put them out. Only those who don't do, don't make mistakes, just my 2c.

I'll try to follow further issues as far as I can, thanks for your understanding.

[negroscuro]

Asking you to review it helps, since 4 eyes see more than just 2 and the earliler this is done the better for everyone. Ok I will try to load Stockholm and Linz. Thank you for your help Stefano.

[negroscuro]

All 3 CS cities input layers are already loaded into database and avaialble through Atos Geoserver (WFS/WMS). Also Land use grid percentages are created for 3 CS.

NAPLES

LINZ

STOCKHOLM

[negroscuro]

Regarding basisns, I am getting the ones overlapping with the BBOX and I keep them as they are, I am not cutting anything so:

• Should I use boundary instead of bbox? --> I think here is a yes, but just in case...
• Should I keep complete overlaping basins or should I cut the them to fit city boundary?

Regarding streams I am getting the ones overlapping with the BBOX and I keep them as they are, I am not cutting anything, so:

• Should I use boundary instead of bbox? --> I think here is a yes, but just in case...
• Should I cut the streams to fit city boundaries? in that case, when calculating height difference between start and ending point will be calculated with new geometries is this correct?

This is how it looks right now. Streams out of dot-line delimiting Napoli area, and also Basins on turquoise covering a very large area...

[negroscuro]

Regarding input data projection and CSIS map display there is a question regarding data projection since original UA data comes in "ETRS_1989_LAEA", which corresponds to EPSG:3035. The thing is map component in CSIS is using EPSG:4326 to display this data which is the standard projection for world. This implies a reprojection on they fly by Geoserver from EPSG:3035 to EPSG:4326. So my question is, does it worth to change input data generation to EPSG:4326 in order to save some processing and give a better user experience while using CSIS map, avoiding Geoserver to do reprojections?

[stefanon]

I'm not in favour of a full input data reprojection to WGS84. As most of the data CSIS needs to map should be in EPSG:3035, why not to use this CRS in the map component instead? problems with the base (background) layers? anyway for my experience, Geoserver is rather smart in reprojecting datasets, bigger improvements in performance can be obtained with proper indexing, tiles and cache settings (speaking for WMS). This is my view, but for sure there should be someone more authoritative about this matter.

Regarding input data projection and CSIS map display there is a question regarding data projection since original UA data comes in "ETRS_1989_LAEA", which corresponds to EPSG:3035. The thing is map component in CSIS is using EPSG:4326 to display this data which is the standard projection for world. This implies a reprojection on they fly by Geoserver from EPSG:3035 to EPSG:4326. So my question is, does it worth to change input data generation to EPSG:4326 in order to save some processing and give a better user experience while using CSIS map, avoiding Geoserver to do reprojections?

[negroscuro]

Regarding basisns, I am getting the ones overlapping with the BBOX and I keep them as they are, I am not cutting anything so:

• Should I keep original geomtries overlaping basins (even those parts of the geometry off the city boundary) or should I cut the them to fit city boundary and just keep those parts of the geometry within city boundary?

Regarding streams I am getting the ones overlapping with the BBOX and I keep them as they are, I am not cutting anything, so:

• Should I cut the streams to fit city boundaries? in that case, when calculating height difference between start and ending point will be calculated with new geometries is this correct?

This is how it looks right now. Streams out of dot-line delimiting Napoli area, and also Basins on turquoise covering a very large area...

Lets do not forget about this question. It would be nice to know which option is the apropiate one.

[stefanon]

Mario, for this last question I wouldn't cut basins or streams but leave them full for sure for model calculations, but also for visualization on map also if outside city boundaries.

[negroscuro]

Nice, thank you Stefano, then it is correct as it is right now. So it looks like I am done with it.

[negroscuro]

Nice, then it is already working as expected.

Regarding pluvial floods, I did not create any land use percentage. Is it needed? I mean should I calculate grid cell percentage for basins and streams? I do not know if that makes any sense...

[negroscuro]

Can this issue be closed? since input layers are already generated and solar radiation is going to be calculated by Heinrich by using ETA formula already approved by Robert Goler. Or there is anything else missing?

[DenoBeno]

Maybe we should rename this into "input layers for heat" and close it? Not sure what we need for flood.