Where can we currently do the screening?

[DenoBeno]

I tried Vienna and Linz, both failed. Where does the screening currently work?

[humerh]

Currently I have only this local paramters for the calculation of local effects:

[humerh]

I would propose to use a simplified formula to calculate solar radiation:

Eta ( η ) = Sun's altitude angle above the horizon - 21th of June: 90° - latitude + 23°27'

Global radiation = Global_radiation0 cos((latitude-23,45)PI/180 Diffuse radiation = Diffuse_radiation0 cos((latitude-23,45)PI/180 Direct radiation = Direct_radiation0 cos((latitude-23,45)PI/180

Global radiation0 = 1040 Wh/m2 Diffuse_radiation0 = 150 Wh/m2 Direct radiation0 = 890 Wh/m2

I think, in comparision to all other model simplification this is a quite good estimation!

It would simplify our life. Emikat would implement this formula and we need no new data source.

[humerh]

I have to add, that this simplified formula is only valid for June 21th (midsummer) !!!!

[DenoBeno]

I don't really care, but someone has to check the data in the end. Are we delivering telephone numbers or something plausible?

On Sun, 6 Oct 2019, 10:03 Heinrich Humer, notifications@github.com wrote:

I have to add, that this simplified formula is only valid for June 21th (midsummer) !!!!

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/clarity-h2020/emikat/issues/25?email_source=notifications&email_token=AAWTC7RR6QXJLG27XQPAAVTQNGLVNA5CNFSM4I5ZRTO2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAOD4AA#issuecomment-538721792, or mute the thread https://github.com/notifications/unsubscribe-auth/AAWTC7TESWJPYOX4IVIA4H3QNGLVNANCNFSM4I5ZRTOQ .

[patrickkaleta]

@RobAndGo can you please take a look at the proposed solution by Heinrich. If this approximation is good enough for us to use it in our calculations, @negroscuro could stop looking for a way on how to get the actual radiation values and instead focus on other tasks.

[DenoBeno]

At the moment, I would prefer having some response everywhere than just getting empty results.

However, my working assumption is that proposed workaround would deliver less reliable results than the calculation based in actual data. Thus, we need the following information:

1. How much difference is there between the two estimates?
2. How does that compare with high-quality models and with the reality?
3. Can we get the data for all cities or not? If yes, where, how, why don't we already have it.
4. Can we somehow indicate if the calculation was done using data or this approximation?
5. If not, can we have two screening packages with different coverages?
6. Any other ideas?

[humerh]

To verify the approach, which I suggested I did a backward calculation from the retrieved radiation values:

						declination	23,45
		Global	Direkt	Diffuse	latitude	Eta'	cos(latitude-23,45))	Global0	Direkt0	Diffuse0
GR	ATHINA	1.010	871,7	138	37,98	75,47	0,968016409	1043,37074	900,501264	142,559567
IT	NAPOLI	994	856	137	40,83	72,62	0,954344655	1041,55244	896,950588	143,554008
RO	ALBA_IULIA	958	822	136	46,04	67,41	0,923277275	1037,60812	890,306761	147,301362
SE	STOCKHOLM	828,3	696,7	131,4	59,27	54,18	0,810859581	1021,50856	859,211652	162,050253

[claudiahahn]

sorry, Robert is on vacation and I didn't know about this github issue. We'll have a look at Heinrichs suggestion.

Here is what we have suggested to Mario:

"Hi Mario, Maybe you could use data from the Copernicus atmospheric monitoring service: https://atmosphere.copernicus.eu/catalogue#/product/urn:x-wmo:md:int.ecmwf::copernicus:cams:prod:an:surface-solar-irradiation:pid291

Apparently, they have data for Europe and one can contact them to get access. It should be possible to get data for the 21st of June, 12:00 (that is what the local effect calculation needs, right?). One also has to select a year. Here one could maybe use the year 2012, since most of the other datasets are available for 2012?

I hope that helps.

Regards, Claudia and Maja"

For me it looked like you can download a data set for whole Europe if you write an email and ask for it. If you click the download bottom, next to the maps of Europe and Africa is the following text: “Contact us to download a volume of CAMS radiation and CAMS McClear data over Europe or Africa”

If you click on the map for Europe (AGATE) you are being directed to this website: http://www.soda-pro.com/web-services/radiation/cams-mcclear

It looks like they do have data for the whole of Europe in raster format. I think you just need to specify the date and time you are interested in.

[claudiahahn]

from the documentation:

[DenoBeno]

Sounds good.

[RobAndGo]

I would propose to use a simplified formula to calculate solar radiation:

Eta ( η ) = Sun's altitude angle above the horizon - 21th of June: 90° - latitude + 23°27'

Global radiation = Global_radiation0 cos((latitude-23,45)PI/180 Diffuse radiation = Diffuse_radiation0 cos((latitude-23,45)PI/180 Direct radiation = Direct_radiation0 cos((latitude-23,45)PI/180

Global radiation0 = 1040 Wh/m2 Diffuse_radiation0 = 150 Wh/m2 Direct radiation0 = 890 Wh/m2

I think, in comparision to all other model simplification this is a quite good estimation!

It would simplify our life. Emikat would implement this formula and we need no new data source.

I'm a bit late on this, but a couple of comments (hopefully they are relevant!):

1. for European latitudes the equation for Eta is sufficient.
2. If the issue is how to a) calculate the global radiation from the extra-terrestial irradiance and b) split the global radition into the diffuse and direct radiation components (daily values), then there is a paper done from the Netherlands which developed an empirical equation which seems to be ok. The paper can be found here: Spittersetal_1986_SeparatingTheDiffuseAndDirectComponentOfGlobalRadiation.pdf I won't explain it in detail for fear that perhaps this isn't the issue of concern, but the interesting part starts from the lower half of pdf-page 2 with the empirical equation splitting the global radiation into direct and diffuse shown on pdf-page 4. Explanations of what the symbols mean are within the text and also in the appendix on pdf-page 11.

[negroscuro]

Claudia provided this link where to get Solar radiation data: http://www.soda-pro.com/web-services/radiation/cams-mcclear

I was contacting via email to copernicus staff regarding solar radiation data set, so as I see, we are going to rely in ETA formula from Heinrich, so I forget about Copernicus solar radiation dataset, indeed they were asking me for several details I could not answer in order to get the appropiate data.

This was their answer:

Dear Mario,

Thank you your email and for your interest in our HelioClim-3 database and associated SoDa services.

If I well understand your request, you need solar data for European area in raster format for the 2012 21st, June? Please could you provide me the time resolution needed? We can provide 15 min, hourly or daily data? Also could you precise me which irradiation data that you want? We can provide data for horizontal, tilted and/or normal planes?

According your answer, I will be able to propose our best price.

With my best regards,

Mathilde

So I will not reply this as it is seems not needed anymore.

[negroscuro]

Regarding the city avilability in the system I think we can rely on the flags already present in the layer available in our geoserver. There are 2 city layers indeed: -city_bbox -city_boundary

After discussion with PLINIVS it seems bbox one is no longer needed and the one to use is boundary By querying this layer you can get two booelan fields: heat_wave and pluvial_flood Those boolean values saying if a city input layers are generated, depending on the hazard you are interested in.

Is this enough for you to make the CSIS more efficient in showing only available cities?

Currently this is something updated while the loading process is running, so this is being updated.

As you may notice those two highlighted cities seems to be available only for pluvial floods but is a workaround to avoid my script to load them since those two cities gave me too much problems to generate its input layers so it is a wat I used to left them aside and I will come back to them at the end of the loading process since they took too much time, this means there are no pluvial flood data for them, there is nothing for those two cities, but all the rest are real status. This means a city with true on both, is an available city, while a city with both fields as false are NOT available yet.

[DenoBeno]

@negroscuro: where can we get that table?

[DenoBeno]

I would propose to use a simplified formula to calculate solar radiation: Eta ( η ) = Sun's altitude angle above the horizon - 21th of June: 90° - latitude + 23°27' Global radiation = Global_radiation0 cos((latitude-23,45)PI/180 Diffuse radiation = Diffuse_radiation0 cos((latitude-23,45)PI/180 Direct radiation = Direct_radiation0 cos((latitude-23,45)PI/180 Global radiation0 = 1040 Wh/m2 Diffuse_radiation0 = 150 Wh/m2 Direct radiation0 = 890 Wh/m2 I think, in comparision to all other model simplification this is a quite good estimation! It would simplify our life. Emikat would implement this formula and we need no new data source.

I'm a bit late on this, but a couple of comments (hopefully they are relevant!):

1. for European latitudes the equation for Eta is sufficient.
2. If the issue is how to a) calculate the global radiation from the extra-terrestial irradiance and b) split the global radition into the diffuse and direct radiation components (daily values), then there is a paper done from the Netherlands which developed an empirical equation which seems to be ok. The paper can be found here: Spittersetal_1986_SeparatingTheDiffuseAndDirectComponentOfGlobalRadiation.pdf I won't explain it in detail for fear that perhaps this isn't the issue of concern, but the interesting part starts from the lower half of pdf-page 2 with the empirical equation splitting the global radiation into direct and diffuse shown on pdf-page 4. Explanations of what the symbols mean are within the text and also in the appendix on pdf-page 11.

On today's telco, we have agreed to follow this approach. Needs less resources in the DP == less trouble for us.

[negroscuro]

@negroscuro: where can we get that table?

I explained that, it is "city_boundary" layer, available in Atos Geoserver as WMS and WFS. Maybe I was not clear enough, sorry.

[p-a-s-c-a-l]

Hm, this complicates things. The cities taxonomy as it is now is not sufficient. We have to extend it by those boolean flags and depending on the study type (heat wave or pluvial flood screening) dynamically generate the list of selectable cities. Not sure if this is easily possible with Drupal @patrickkaleta ?

ATM we don't need this since we don't support pluvial floods yet. So it's save to re-import the cities taxonomy where heat_wave = true @therter

[p-a-s-c-a-l]

Still open seems the question about solar radiation and mortality rates. Are those information now available for all cities with heat_wave == true? If not, those new flags tell us that HW/PF HC-LE input data is available for the respective cities, but not that Impact Calculation is possible at all.

How ca we consider that when updating the cities taxonomy? I would suggest to 1) download the city_boundary layers, remove all records where heat_wave != true 2) remove all records where no solar radiation / mortality rates information is available. Who can do this? @humerh ? 3) re-import the cities into the current cities taxonomy.

[humerh]

As the radiation will be calculated in EMIKAT from the latitude we automatically have radiation for all locations in the near future. I also implement the EUROSTAT-Interface in an extended way, and I import the last 20 years of data. If the value of the requested year does not exist, the algorithm will use a neighbour year.

For all cities having NO VALUE at all, I will manually insert an "EXPERT ESTIMATION" and comment this value.

So from my site we will have no restriction any more.

[RobAndGo]

Hi @humerh Here is an explanation of how to use the method proposed in the paper from Spitters et al. that I posted here. Separating the diffuse and direct component of global radiation.docx

[negroscuro]

I am not managing mortaility rates or solar radiation. As Heinrich explained they are calculating solar radiation with ETA formula and it seems to be available for any city, but I do not know about mortality rates.

I can only guarantee that heat_wave = true means all land use layers are generated and land use grid percentages also available among Atos Geoserver layers.

Load process is still progressing, currently 102 cities left out of 540.

[negroscuro]

Nice news, cities are progresing fast. Now it is only 27 cities left! apart from LEFKOSIA and LISBOA which need a retry.

[patrickkaleta]

Hm, this complicates things. The cities taxonomy as it is now is not sufficient. We have to extend it by those boolean flags and depending on the study type (heat wave or pluvial flood screening) dynamically generate the list of selectable cities. Not sure if this is easily possible with Drupal @patrickkaleta ?

ATM we don't need this since we don't support pluvial floods yet. So it's save to re-import the cities taxonomy where heat_wave = true @therter

I will look into the possibility of showing only certain cities based on the selected study type.

What is definitely possible is to hide all cities/regions from the selection that are unpublished (Drupal 8.6. introduced a publishing status field for taxonomy terms, however the UI for this won't get updated until D8.8. details). So, for now @therter could use the Bulk edit (I created one for the Cities taxonomy) to unpublish all of them and then only re-publish all the ones he gets with his script. That way it wouldn't be necessary to completely delete and create this taxonomy each time (and then re-fill all field_city_region fields in those Studies which already had a city selected). But I don't know how exactly your script is working and what's easiest for you.

[therter]

completely delete and create this taxonomy each time (and then re-fill all field_city_region fields in those Studies which already had a city selected). But I don't know how exactly your script is working and what's easiest for you.

This is the way, my script is working and this script does already exist. So it's easy for me to execute this script again with new city data.

[negroscuro]

Nice news, cities are progresing fast. Now it is only 27 cities left! apart from LEFKOSIA and LISBOA which need a retry.

It ended, I will start with Lisboa and Lefkosia to see if I have better luck with them this time.

I have been discussing also with Miguel Angel, that next time with a huge load of data like this, we should try with new postgresql12 and postgis3 since it looks like they improved parallel queries resolution with more than 1 CPU usage. Probably it will improve a lot.

[DenoBeno]

@humerh : how does it look like on our side now? Are we ready to do studies in all these cities?

[negroscuro]

Lefkosia is generated, only one remaining is Lisboa (currently trying).

[negroscuro]

Finally Lisboa got loaded, so all 540 cities are ready.

[DenoBeno]

Today I tried to do a study in Barcelona and it didn't work at first. Something really simple didn't get generated in EMIKAT (ask Heinrich for details) and then the whole chain failed.

Heinrich got it working in no time again, but I'm not sure if this is a 1 in a million chance or something that will happen every time now. Please try it out and report if it fails.

On Mon, 28 Oct 2019 at 19:26, Mario Nuñez notifications@github.com wrote:

Finally Lisboa got loaded, all 540 cities are loaded into the system and ready.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/clarity-h2020/emikat/issues/25?email_source=notifications&email_token=AAWTC7QIBRYS2VTPHUSLZBTQQ4VGLA5CNFSM4I5ZRTO2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECN5JSY#issuecomment-547083467, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAWTC7QZJN37ZMDXR5XFGDTQQ4VGLANCNFSM4I5ZRTOQ .

[negroscuro]

After importing data into ATOS input layer local effects database, Heinrich csv file with mortality data has 1000 cities while I have only 545 because those cities are the ones with all 3 datasets available (UA2012, STL and ESM) that is ok. I realized that city codes does not match between CSV mortality and UA2012 I am using, so for me it did things a bit harder, but I manage to seek for matchings by using city names besides they are not exactly the same names, but seems to work.

sample:

So what should I do? if I try to get mortality I will end up having less cities availabl since only 333 of them will have mortality data (because there are only 333 city coincidences between the two data sets). The rest of the mortality data... I do not see the point of storing such information if we do not have the city data for input layers for local effects.

Any idea on what to solve this? because I checked manually for some not matching city names and seems like those cities are not in CSV mortality dataset.

[negroscuro]

Anybody knows what to do in such situation?

[claudiahahn]

Today I tried to do a study in Barcelona and it didn't work at first....Heinrich got it working in no time again, but I'm not sure if this is a 1 in a million chance or something that will happen every time now. Please try it out and report if it fails.

I created a new study (Naples - test, 54) but the local effect was not calculated At least there were no results in the table and I couldn't display Tmrt on the map. Even though the legend for Tmrt was displayed (it has same range as the legends for Linz and Vienna)

[p-a-s-c-a-l]

Yes, there are some issues at the EMIKAT backend atm. Please see also https://github.com/clarity-h2020/emikat/issues/29

[negroscuro]

Matching by not using last 2 digits of the city code as Heinrich pointed I managed to match 460 out of 540 cities.

It looks like:

It is already published in our Geoserver as mortality layer with city boundaries geometry, city id and name from UA, and all the mortality data provided by Heinrich:

[p-a-s-c-a-l]

@therter Can you please assess the impact on the cities taxonomy in CSIS and take the required steps to update it if necessary?

[negroscuro]

Now available cities for CSIS taxonomy are in layer MORTALITY where now 460 cities are set. The only problem is that I found an incoherence in mortality data:

For some reason there is a duplicate, a city without name (null) that for some reason is generating 2 ARNHEM so I think I should delete or ignore any city without a name, is that ok @humerh ?

Indeed there are several, and this can only generate problems IMHO:

For that mortality layer should be generated like:

select c.id,c.name,c.code, m.deaths,m.population, m.rate, c.boundary from mortality m, city c where c.id=m.city and m.name is not null order by c.id ASC;

I think @humerh already commented this in the monday's telco, those 2 last code characters indicate city center and outskirts, so I have to ignore outskirts data. So problem is solved.

[humerh]

If we speek about "Urban areas" we should use the term of the Eurostat definition:

See: https://ec.europa.eu/eurostat/web/cities/background

One file there is the official list of cities and regions: https://ec.europa.eu/eurostat/documents/4422005/4430532/City-FUAs-Greater-cities-list-2017.xls

It is important to understand the systematic of coing of the cities and regions: Character 1-2: Nationality code Character 3-5: Number of the city for one country. Character 6-7: Definition of the boundaries of the city region:

Please look to the definitions given there:

[humerh]

NL009C2 is another version of boundary than NL009C1.

As longer I think about all these problems I recognise, that it is HARDLY possible to get an homogenous view of statistic data for all Europe. Maybe it is better to find only ONE figure for mortality rates for all Europa and ignore all local statistics. :-(

[therter]

Now available cities for CSIS taxonomy are in layer MORTALITY where now 460 cities are set.

This layer are imported now

[negroscuro]

NL009C2 is another version of boundary than NL009C1.

As longer I think about all these problems I recognise, that it is HARDLY possible to get an homogenous view of statistic data for all Europe. Maybe it is better to find only ONE figure for mortality rates for all Europa and ignore all local statistics. :-(

I would say it is working right now, by ignoring subcity districts levels, as you explained, is not it?

The only thing is that as a new requirement for a city to be available in CSIS we went from 545 available cities to 440 because taking into account mortality data, that is all.

• Left upper image: original just from UA (city_boundary in geoserver) 545 cities (yellow ones in image below).
• Right upper image: you can see final available cities (mortality in geosserver) 440 cities (orange ones in image below).

In the end, yellow ones are not available any more. Those are 105 "lost" cities in total.

[p-a-s-c-a-l]

@therter Is the updated list of 440 cities imported in CSIS? Can we close this issue?

[therter]

The last import of the citiy taxonomy were the cities from the mortality layer (see https://github.com/clarity-h2020/docker-drupal/issues/128#issuecomment-556973195). This were 460 cities. At the moment, the mortality layer contains 545 cities. Should I reimport this cities?

[p-a-s-c-a-l]

Before importing anything we have to clarify if this is really the most current layer, because there a now different numbers floating around (440 vs. 460 vs. 540) and 460 still seems to be the correct number.

[negroscuro]

545 cities is the old count, without taking into account mortality. Now, with mortality data, the count downs to 460 cities. You imported 460 and it seems there are repeated cities, I am checking why.

[negroscuro]

Solved, final result is 455 cities. The problem was data from mortality has some records without city name and matching code for the city (only using 5 characters) since they are some kind of outskirts or close municipalities to the 'main' city.

[negroscuro]

... Should I reimport this cities?

@therter yes please do a reimport. Sorry for the inconvenience.

[therter]

@therter yes please do a reimport. Sorry for the inconvenience.

Ok, I will reimport the layer mortality (http://services.clarity-h2020.eu:8080/geoserver/clarity/wfs?request=GetFeature&typeNames=clarity:mortality&outputFormat=SHAPE-ZIP)

[therter]

I did the reimport. So I think, I can close this issue.