New EMIKAT calculation currently not possible

[DenoBeno]

I just tried it, the input data looks OK but the calculation breaks with error and there is no "hazard local data" or "impact".

Population calculation works.

• https://csis.myclimateservice.eu/study/50/step/1843/view/maps

[DenoBeno]

Calculation doesn't work in Naples either. I hope this is a known issue that will be fixed soon.

[p-a-s-c-a-l]

This Napoli Screening study created by @mattia-leone shows the following error:

"There have been 2 errors in the calculation process. Please try to adapt your Study settings or contact the site administrators"

Now, is the screening ATM generally generally broken or is this just a temporary failure?

[p-a-s-c-a-l]

@mattia-leone wrote:

tried Napoli this morning and it worked, now "local effect" tab (and related maps) disappeared...

Reason: somebody changed the study type to Basic Screening: General: https://csis.myclimateservice.eu/study/100/view/context

[NICDDB]

Thank you @p-a-s-c-a-l, i recovered the simulation launched yesterday on Napoli. Today i've tried to lunch a new simulation on Bologna but there is a problem: the calculation process has been done but the map it's not aviable. @humerh can you you can make it possible for us to do simulations on other Italian cities besides Naples, for example Bologna and Palermo. We would need these for tomorrow or Sunday at the latest, because they can represent a great opportunity for the follow up of clarity as @DenoBeno has already wished.

[p-a-s-c-a-l]

Regarding the Bologna Screening study:

• Table data is available for HC-LE (T_MRT, etc.)&emikat_id=3344&datapackage_uuid=2434ce93-93d4-4ca2-8618-a2de768d3f16&write_permissions=1&study_variant=BASELINE&time_period=Baseline&emissions_scenario=Baseline&event_frequency=Frequent) and Exposure&emikat_id=3344&datapackage_uuid=2434ce93-93d4-4ca2-8618-a2de768d3f16&write_permissions=1&study_variant=BASELINE&time_period=Baseline&emissions_scenario=Baseline&event_frequency=Frequent), but not for Impact&emikat_id=3344&datapackage_uuid=2434ce93-93d4-4ca2-8618-a2de768d3f16&write_permissions=1&study_variant=BASELINE&time_period=Baseline&emissions_scenario=Baseline&event_frequency=Frequent).

• The WMS delivers empty images for all parameters (HCE-LE, EE, IA) calculated by EMIKAT.

I would have expected that EE an HC-LE map layers do work. So apart from failing impact calculation probably also a configuration problem at the side of GeoServer? You can check the HC-LE Map here&emikat_id=3344&datapackage_uuid=2434ce93-93d4-4ca2-8618-a2de768d3f16&write_permissions=1&study_variant=BASELINE&time_period=Baseline&emissions_scenario=Baseline&event_frequency=Frequent&grouping_tag=taxonomy_term--hazards).

[NICDDB]

yes probably is a configuration problem. I lunch a new study on Palermo and it work! Thank you so much. Anyway i will try to do a studies on other city and i will keep to update. I have a suggestion when you select a study scenario there is the stuff "Event frequency" which is related to the power of impact (correct me if I'm wrong) but it is not easy to understand.

[NICDDB]

The new studies on Bologna and Genoa have been successful! So I think the problems have been solved. Thank you @p-a-s-c-a-l , @humerh, @DenoBeno.

[p-a-s-c-a-l]

Today I tried a new study for Naples, but it didn't produce any results:

• https://csis.myclimateservice.eu/study/109/step/3471/view/table
• https://csis.myclimateservice.eu/study/109/step/3474/view/table

@humerh Can you please check what's wrong?

[p-a-s-c-a-l]

I changed the study are of study #109 to force a re-calculation but still not results available.

[patrickkaleta]

According to the batch jobs it seems like the re-calculation was completed and there should be results available (8040 results for impact):

@humerh: EmikatID for this particular study is 3351.

[p-a-s-c-a-l]

O.K. Now it's working again:

Thanks! Closing.

[humerh]

Please modify the URL for the reports in the following way:

Current link: https://serrice.emikat.at/EmiKatTst/api/scenarios/3351/feature/view.2974/table/data?rownum=1000

to: https://serrice.emikat.at/EmiKatTst/api/scenarios/3351/feature/view.2974/table/data?rownum=1000&filter=SZ_ID%3D3351 (3351=$emikat-id)

You should do it for the features "view.2974", "view.2975" and "view.2955".

This reduces the amount of rows, which are fetched at evaluation.

I think, this has an connect with the problem.

[DenoBeno]

Today we encountered an issue with this again.

Study no. 112, in Linz:

• the calculation took very long time, showing "step 9" for ages.
• https://csis.myclimateservice.eu/study/112/step/3625/view/table - baseline/rare values are 0
• https://csis.myclimateservice.eu/study/112/step/3625/view/maps - temperature indices aren't shown.

[patrickkaleta]

Today we encountered an issue with this again.

Study no. 112, in Linz:

• the calculation took very long time, showing "step 9" for ages.

This seems to be a new problem. Study 113 is currently also taking a lot of time at the very same step (it's the "Rebuild Table CLY_HW_T_MRT" batch job)

• https://csis.myclimateservice.eu/study/112/step/3625/view/table - baseline/rare values are 0
• https://csis.myclimateservice.eu/study/112/step/3625/view/maps - temperature indices aren't shown.

Those are probably related to the issue @humerh mentioned in his comment yesterday. The filter SZ_ID={$emikatID} needs to be added to the requests made by the map and table components. This has probably yet to be implemented.

[patrickkaleta]

@humerh EmikatID for the Study 112 is 3352

[p-a-s-c-a-l]

Those are probably related to the issue @humerh mentioned in his comment yesterday. The filter SZ_ID={$emikatID} needs to be added to the requests made by the map and table components. This has probably yet to be implemented

Yes, I'm going to update the EMIKAT template resources.

[patrickkaleta]

Regarding the longer than usual waiting times, Heinrich just told me that another Emikat client recently triggered a lot of processes, which got queued before our CSIS requests. Things should be back to normal tomorrow.

Heinrich is thinking about creating an independent queue just for our CSIS requests, so that this wouldn't happen to us again.

[p-a-s-c-a-l]

You should do it for the features "view.2974", "view.2975" and "view.2955".

@humerh AFAIR we've removed the SZ_ID filter parameter from all URLs because it don't work.

I've added parameter again, but now I'm running into the following error for https://service.emikat.at/EmiKatTst/api/scenarios/3287/feature/view.2974/table/data?rownum=10&filter=STUDY_VARIANT%3D%27BASELINE%27&filter=TIME_PERIOD%3D%27Baseline%27&filter=EMISSIONS_SCENARIO%3D%27Baseline%27&filter=EVENT_FREQUENCY%3D%27Rare%27&filter=SZ_ID=3287&column=GRID_ID&column=STUDY_VARIANT&column=TIME_PERIOD&column=EMISSIONS_SCENARIO&column=EVENT_FREQUENCY&column=T_MRT&column=T_UTCI&column=T_A&column=DISCOMFORT_LEVEL:

"Error during data retrieval - EMIKAT-Server.DB.ConnectionProblem: Can't access db: java.sql.SQLSyntaxErrorException: ORA-00904: \"SZ_ID\": ungültiger Bezeichner\n - "

[p-a-s-c-a-l]

I've temporarily reactivated the buggy monitoring service till the new one is implemented. I've deactivated mail notifications and and changed the EMIKAT polling interval to 2 hours.

You can check the status of services at health-check.clarity.cismet.de

I've added the SZ_ID parameter to EMIKAT URLs as requested here, but that parameter seems not supported by HC-LE and Impact resources:

[p-a-s-c-a-l]

This is strange. @humerh is the order of parameters important?

This works:

https://service.emikat.at/EmiKatTst/api/scenarios/3287/feature/view.2974/table/data?rownum=10&filter=STUDY_VARIANT%3D%27BASELINE%27&filter=TIME_PERIOD%3D%27Baseline%27&filter=EMISSIONS_SCENARIO%3D%27Baseline%27&filter=EVENT_FREQUENCY%3D%27Rare%27&column=GRID_ID&column=STUDY_VARIANT&column=TIME_PERIOD&column=EMISSIONS_SCENARIO&column=EVENT_FREQUENCY&column=T_MRT&column=T_UTCI&column=T_A&column=DISCOMFORT_LEVEL

This works too:

https://service.emikat.at/EmiKatTst/api/scenarios/3287/feature/view.2974/table/data?rownum=10&filter=SZ_ID=3287

This does not work:

https://service.emikat.at/EmiKatTst/api/scenarios/3287/feature/view.2974/table/data?rownum=10&filter=STUDY_VARIANT%3D%27BASELINE%27&filter=TIME_PERIOD%3D%27Baseline%27&filter=EMISSIONS_SCENARIO%3D%27Baseline%27&filter=EVENT_FREQUENCY%3D%27Rare%27&column=GRID_ID&column=STUDY_VARIANT&column=TIME_PERIOD&column=EMISSIONS_SCENARIO&filter=SZ_ID=3287&column=EVENT_FREQUENCY&column=T_MRT&column=T_UTCI&column=T_A&column=DISCOMFORT_LEVEL

However, this does:

https://service.emikat.at/EmiKatTst/api/scenarios/3287/feature/view.2974/table/data?rownum=10&filter=SZ_ID=3287&filter=STUDY_VARIANT%3D%27BASELINE%27&filter=TIME_PERIOD%3D%27Baseline%27&filter=EMISSIONS_SCENARIO%3D%27Baseline%27&filter=EVENT_FREQUENCY%3D%27Rare%27&column=GRID_ID&column=STUDY_VARIANT&column=TIME_PERIOD&column=EMISSIONS_SCENARIO&column=EVENT_FREQUENCY&column=T_MRT&column=T_UTCI&column=T_A&column=DISCOMFORT_LEVEL

[p-a-s-c-a-l]

OK. now after changing the order of parameters, it works again.

Seems I got hit again by the principle of least surprise. :-)

[p-a-s-c-a-l]

OK, but there is still this problem with study 113.

[DenoBeno]

The service looks nice.

I do see one possible issue though. Today, I am not able to do any new calculations. What worked yesterday in Linz doesn't work today. There is even no error, it's just that there are no results shown in the end.

Old studies are still shown. New ones finish without error message but nothing is shown in the end. It would be nice if this could also be tested automatically.

[humerh]

The last problems of calculation had all one reason: Till now the bach jobs were elaborated by ONE Batch-Job (for all emikat customers). Last week one of the customers started two big jobs in the background. The consequence was, that the recalculation of Clarity jobs had very long delays as a consequence. To overcome this problem I implemented a second JOB worker. My plan was, that one should be reserved for clarity. Unfortunately the consequences were, that the sequence of evaluation was not given any more. So later jobs started before the inputs were ready.

I had to redesign the batch calculation module in a broader way. Yesterday evening I deployed a new version and I hope, that the recalculation will work correctly now.

[p-a-s-c-a-l]

Currently, no EMIKAT results at all seem to be available. E.g. Please check the following studies:

https://csis.myclimateservice.eu/study/35/step/1524/view/table https://csis.myclimateservice.eu/study/33/step/1513/view/table

I've recalculated Study 35 but still no result. @humerh Can you please check what's wrong?

[patrickkaleta]

Currently, no EMIKAT results at all seem to be available. E.g. Please check the following studies:

https://csis.myclimateservice.eu/study/35/step/1524/view/table https://csis.myclimateservice.eu/study/33/step/1513/view/table

I've recalculated Study 35 but still no result. @humerh Can you please check what's wrong?

Related to this. Results are actually there but are currently not displayed. Probably a permission issue on Emikat-side. Heinrich is investigating this.

[humerh]

I deactivated the Job-Runner for deeper tests. Now the service is available again.

[p-a-s-c-a-l]

There have been 7 problem(s) in a total of 13 batchjobs.

ID: 70369, Name: Rebuild all views..., Status: ERR, Message: ERR No signature of method: static at.ac.ait.emikat.server.service.EmiKatRebuildableTableImpl.refreshTCs() is applicable for argument types: (at.ac.ait.emikat.server.service.SessionContext, java.lang.Integer) values: [at.ac.ait.emikat.server.service.SessionContext@56f653c7, ...] Possible solutions: refreshTC(at.ac.ait.emikat.server.service.SessionContext, long, long)

See https://csis.myclimateservice.eu/maintenance/check-emikat-results

[DenoBeno]

I had a talk with Heinrich after the telco today. He has fixed some issues last week but didn't re-deploy the server afterwards.

After re-deploying the server again, we found out that the calculation worked OK on some places (Vienna, Barcelona) and failed elsewhere (Stockholm) while stating that all is OK.

Few minutes later, Heinrich discovered that the issue is with height information missing e.g. in Stockholm. The last information that I have is that he will implement a check for the availability of the height data. If available, height correction will be calculated, if not the local effect without the height correction will be calculated.

@humerh , anything else to add?

[humerh]

As Denis said I had a problem with the naming of the function in batch processing. I deployed this morning the adapted batch server. The calculation ist now ok.

The second problem (independend of the first one) was, the the heiht correction did not handle the situation, if the height data of some location does not exist till now. I changed the calculation in that way, that I now will deeliver the results WITHOUT height correction.

For me this problem is solved.

[p-a-s-c-a-l]

Thanks. ATM it seems that the Test Study isn't working: There have been 6 problem(s) in a total of 13 batchjobs.

See also https://github.com/clarity-h2020/csis/issues/153#issuecomment-603040305 and https://github.com/clarity-h2020/csis-technical-validation/issues/21

[negroscuro]

Regarding height availability, only those cities have that data:

259 "REGGIO_DI_CALABRIA"
247 "BARI"
246 "GENOVA"
7 "SALZBURG"
245 "PALERMO"
244 "NAPOLI" 298 "KAUNAS"
305 "JELGAVA"
248 "BOLOGNA"
338 "GDANSK"
123 "BARCELONA" --> being generated right now 112 "TALLINN"
122 "MADRID"

Stockholm will be next city after Barcelona if you need it, since I am testing a bug fixing on land categorization with Barcelona land use generation. So if Barcelona generation result as expected and after proceeding with Stockholm I will delete every other city to be reloaded again in order to fix the bug which is already present on the rest of the loaded cities.

[negroscuro]

Due to an error on land categorization I am renewing database contents so cities will be available in some days, I am going one by one generating them.

[NICDDB]

Hi @p-a-s-c-a-l , @DenoBeno, @humerh me @stefanon and @mattia-leone are working on validation and we have a problem: we create a study for Naples and it's work on map, but the table is not complete, there are only 100 rows in the table. We have a lot of urgency to have this data to complete the validation of the models, so we expect a quick response. Thank you all

[negroscuro]

Sorry for the inconvenience, I was not aware you were using it during this morning and with such urgency. Napoli is currently being loaded it should end around 16:30h. Data cannot be loaded if city is already there so I had to delete it first.

[NICDDB]

ok Mario i understood, doesn't matter if it will be uploaded later today.

[negroscuro]

ok Mario i understood, doesn't matter if it will be uploaded later today.

I will let you know as soon as it is available again.

[NICDDB]

ok thank you

[negroscuro]

It seems like is taking longer than expected.

[negroscuro]

It ended, now it is available for you to use it.

[NICDDB]

Than You Mario

Nicola Addabbo

Il Mer 25 Mar 2020, 16:51 Mario Nuñez notifications@github.com ha scritto:

It ended, now it is available for you to use it.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/clarity-h2020/emikat/issues/33#issuecomment-603919771, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOQOEWZA2X7IX7NHCN6Z4ELRJISBPANCNFSM4KVE3DOQ .

[negroscuro]

There is a new style o layer city to check city availability in the system: -red, not available -green, fully available -yellow, not all hazards data available (pluvial flood/heat wave)

http://services.clarity-h2020.eu:8080/geoserver/clarity/wms?service=WMS&version=1.1.0&request=GetMap&layers=clarity%3Acity&STYLES=city_availability&bbox=1969465.0%2C996016.875%2C7168551.0%2C4677889.0&width=768&height=543&srs=EPSG%3A3035&format=text%2Fhtml%3B%20subtype%3Dopenlayers#toggle

[negroscuro]

There is missing mortality population/deaths data for cities in Turkey area:

http://services.clarity-h2020.eu:8080/geoserver/clarity/wms?service=WMS&version=1.1.0&request=GetMap&layers=clarity%3Amortality&STYLES=mortality&bbox=1995203.0178%2C1014243.9784%2C7079558.6386%2C5270223.5146&width=768&height=642&srs=EPSG%3A3035&format=text%2Fhtml%3B%20subtype%3Dopenlayers#toggle

@humerh do you have such data to add it to the system?

[humerh]

Eurostat does not have many data from Turkey, because the EC delivers not as much money as Erdogan wishes :-)

The last data we have are from 2004 and 2012. For such cases I use internally a typical value of 0.01 (1%)) for the mortallity rate:

[humerh]

Sorry, the labels where not readable:

[negroscuro]

So this means there is no need to add any data for them, will you use your default value (1%) ok. Thank you.

[DenoBeno]

EMIKAT calculations have been working for a few days now (for me)

[p-a-s-c-a-l]

Resolved. Any problems will be automatically reported by our Integration Tests in https://github.com/clarity-h2020/csis-technical-validation/issues

[p-a-s-c-a-l]

Again

[p-a-s-c-a-l]

There have been 17 problem(s) in a total of 17 batchjobs.

Calculation was started at: 01.01.1970 01:00:00

ID: 77242, Name: Rebuild all views..., Status: INI, Message: ID: 77240, Name: Rebuild Table CLY_IMPACT_RESULT_HW#1838, Status: INI, Message: ID: 77238, Name: Rebuild Table CLY_HW_T_MRT#1856, Status: INI, Message: ID: 77237, Name: Rebuild Table CLY_HW_FLUXES#1856, Status: INI, Message: ID: 77236, Name: Rebuild Table CLY_HW_GRID_DETAILS_PROJ#1856, Status: INI, Message: ID: 77235, Name: Rebuild Table CLY_HW_GRID_DETAILS#1856, Status: INI, Message: ID: 77233, Name: Rebuild Table CLY_URBAN_ATLAS#1776, Status: INI, Message: ID: 77232, Name: Rebuild Table CLY_AO_LAYER_PARAMS#1796, Status: INI, Message: ID: 77231, Name: Rebuild Table CLY_EUROSTAT_CITIES_MORTALITY#2056, Status: INI, Message: ID: 77230, Name: Rebuild Table CLY_EL_POPULATION_INTERPOLATED#2016, Status: INI, Message: ID: 77229, Name: Rebuild Table CLY_UA_FRACTION#2016, Status: INI, Message: ID: 77227, Name: Rebuild Table CLY_ADAPTATION_OPT_ITEM#1837, Status: INI, Message: ID: 77226, Name: Rebuild Table CLY_ADAPTATION_OPTION#1837, Status: INI, Message: ID: 77225, Name: Rebuild Table CLY_PROJECT#1837, Status: INI, Message: ID: 77224, Name: Rebuild Table CLY_HAZARD_EVENTS_STUDY#2036, Status: INI, Message: ID: 77223, Name: Rebuild Table CLY_GRID_ETRS89_1K#1757, Status: INI, Message: ID: 77222, Name: Rebuild Table CLY_PARAMETER#1976, Status: INI, Message:

I need an adaptation scenario to proceed with the implementation of adaptation map and table.