search

clarity-h2020 / emikat

http://www.emikat.at/?lang=en
1 stars 0 forks source link

Studies not getting the EMIKAT ID assigned #20

Closed DenoBeno closed 5 years ago

DenoBeno commented 5 years ago

probably related to: https://github.com/clarity-h2020/emikat/issues/6, https://github.com/clarity-h2020/emikat/issues/5

We have just noticed that new studies aren't getting the EMIKAT ID assigned anymore. This has probably been the case for a while already. Please check & fix.

DenoBeno commented 5 years ago

See also https://github.com/clarity-h2020/emikat/issues/19

patrickkaleta commented 5 years ago

The problem has to be on Emikat side, since POSTs and PUTs currently don't work even when using the Emikat Swagger interface (so without any CSIS involvement). I already informed Heinrich and he is working on it.

BTW: I configured the CSIS helpers module to log some status messages when Studies are being sent to Emikat. You can find those message here: https://csis.myclimateservice.eu/admin/reports/dblog?type%5B%5D=EmikatHelperFunctions&hostname=

There you can see that it last worked last week on 18.9.2019. Then, until today, there were no other attempts to send Study updates to Emikat.

patrickkaleta commented 5 years ago

@humerh solved the problem. Our POST and PUT requests are being processed properly again by Emikat.

DenoBeno commented 5 years ago

Excellent. I hope it will stay this way.

On Tue, 24 Sep 2019, 17:26 Patrick Kaleta, notifications@github.com wrote:

Closed #20 https://github.com/clarity-h2020/emikat/issues/20.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/clarity-h2020/emikat/issues/20?email_source=notifications&email_token=AAWTC7QF7IE5L2ZKHT4DNRLQLIWTZA5CNFSM4IZ5GFV2YY3PNVWWK3TUL52HS4DFWZEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW5KTDN5WW2ZLOORPWSZGOTZ2URTI#event-2658486477, or mute the thread https://github.com/notifications/unsubscribe-auth/AAWTC7RQL3C5FCPE2NY2RFTQLIWTZANCNFSM4IZ5GFVQ .

p-a-s-c-a-l commented 5 years ago

@humerh It still doesn't seem to work for this study.

patrickkaleta commented 5 years ago

@humerh It still doesn't seem to work for this study.

This study doesn't yet have a data package and study area. That's why Emikat is not informed about this study and therefore there is also no emikatID available. (The csis helpers module checks for those relevant fields and informs Emikat only if they are available/have changed.)

A while ago I discussed with @humerh what would be considered relevant information for Emikat to be able to start the calculations. That is also mentioned in the readme.md of the Emikat repository.

I will add this information about the relevant fields in the readme of the csis helpers module as well.

patrickkaleta commented 5 years ago

Additionally, I can also create another Drupal log message indicating each time a request is not send to Emikat after a Study update. But that could spam our log messages, so I would only use this for a limited time.

p-a-s-c-a-l commented 5 years ago

Ah, ok, my fault. Now there is a study id but still no data from EMIKAT, e.g. Population Exposure: https://service.emikat.at/EmiKatTst/api/scenarios/3209/feature/tab.CLY_EL_POPULATION_INTERPOLATED.2016/table/data?rownum=1000

How long does that take @humerh ?

DenoBeno commented 5 years ago

In my opinion, you should show messages on the top of the page: emikat triggered, EMIKAT not triggered because...

At least for admins and developers...

On Wed, 25 Sep 2019, 13:56 Pascal Dihé, notifications@github.com wrote:

Ah, ok, my fault. Now there is a study id but still no data from EMIKAT, e.g. Population Exposure: https://service.emikat.at/EmiKatTst/api/scenarios/3209/feature/tab.CLY_EL_POPULATION_INTERPOLATED.2016/table/data?rownum=1000

How long does that take @humerh https://github.com/humerh ?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/clarity-h2020/emikat/issues/20?email_source=notifications&email_token=AAWTC7QIXG3WO36CLBFVQ7LQLNGWNA5CNFSM4IZ5GFV2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7RT6BY#issuecomment-534986503, or mute the thread https://github.com/notifications/unsubscribe-auth/AAWTC7XZQF76UPV2Y2ATFBDQLNGWNANCNFSM4IZ5GFVQ .

p-a-s-c-a-l commented 5 years ago

Although the EMIKAT ID 3209 is now assigned to study 33 and EMIKAT ID 3189 to study 35, EMIKAT calculation was not triggered. But the logs show:

grafik

grafik

patrickkaleta commented 5 years ago

In my opinion, you should show messages on the top of the page: emikat triggered, EMIKAT not triggered because... At least for admins and developers...

Probably a good idea for developers and admins. I'll have a look at how to implement something like that.

patrickkaleta commented 5 years ago

Although the EMIKAT ID 3209 is now assigned to study 33 and EMIKAT ID 3189 to study 35, EMIKAT calculation was not triggered. But the logs show:

Emikat is indeed triggered and a batch job is created and queued (Denis and I checked this via an Emikat Client), but it seems that a previous batch job got stucked. Heinrich is not in the office today, but we already called to inform him of this issue. He'll get it fixed soon hopefully.

DenoBeno commented 5 years ago

I think that this is fixed now. In emikat client, all teh batch jobs have been finished...

Trying to trigger the https://csis.myclimateservice.eu/study/33/ again.... Yes, it's working.

grafik

I'm curious to see how long it takes to run it all now...

DenoBeno commented 5 years ago

aproximately 6 minutes.

grafik

For the future reference - if it isn't done in 10-15 minutes, then we have a problem.

DenoBeno commented 5 years ago

Ropening this.

While EMIKAT does finish the calculation batch job, some data is not available for the study #33. E.g., all of these are empty:

image

Unsurprisingly, nothing is shown on csis either.

image

Same for study 35. :-(

DenoBeno commented 5 years ago

Oh, come on. I tried to get a calculation working for linz and nothing happened. Logs say:

image

Nothing in EMIKAT batch-jobs.

The EMIKAT did start calculation when I changed the area again.

image

For linz, I do get the population, but no local effect data. :-(

image image

For the reference: "result = 0" in emikat client means that no data was generated.

image

Batch took 5-6 minutes again.

DenoBeno commented 5 years ago

I tried for Napoli in the hope that there we would have all the data.. Result: a new type of error.

image

Napoli, second try, with different area.. First the server reconnected....

image

Then the UA fraction and population and all other layers built as they should:

image

Which doesn't mean that they are also shown on CSIS. Hm.... OK, this time EMIKAT is producing the data as required, so we also have an issue with the CSIS app...

image

p-a-s-c-a-l commented 5 years ago

Which doesn't mean that they are also shown on CSIS. Hm.... OK, this time EMIKAT is producing the data as required, so we also have an issue with the CSIS app...

The table and thus EMIKAT API do not receive the variables values we have agreed upon, see https://github.com/clarity-h2020/csis-helpers-module/issues/14#issuecomment-535789402.

So probably that's an issue of the taxonomy / paragraph / whatever? We need value not meaning. I tried to fix it in csis-helpers-module, but it didn't work.

DenoBeno commented 5 years ago

Which doesn't mean that they are also shown on CSIS. Hm.... OK, this time EMIKAT is producing the data as required, so we also have an issue with the CSIS app...

The table and thus EMIKAT API do not receive the variables values we have agreed upon, see clarity-h2020/csis-helpers-module#14 (comment).

So probably that's an issue of the taxonomy / paragraph / whatever? We need value not meaning. I tried to fix it in csis-helpers-module, but it didn't work.

Ah, I wasn't aware of that one. OK, this shouldn't be that difficult to fix. If nothing else works, we'll replace the paragraph with node...

patrickkaleta commented 5 years ago

Which doesn't mean that they are also shown on CSIS. Hm.... OK, this time EMIKAT is producing the data as required, so we also have an issue with the CSIS app...

The table and thus EMIKAT API do not receive the variables values we have agreed upon, see clarity-h2020/csis-helpers-module#14 (comment).

So probably that's an issue of the taxonomy / paragraph / whatever? We need value not meaning. I tried to fix it in csis-helpers-module, but it didn't work.

The problem with the meaning and value field should be resolved for now. @p-a-s-c-a-l: Can you confirm that your apps are getting correct values for emission_scenario, time_period,...?

...

Ah, I wasn't aware of that one. OK, this shouldn't be that difficult to fix. If nothing else works, we'll replace the paragraph with node...

@DenoBeno no, this should not be necessary. Extracting fields out of the paragraphs turned out to be quite straightforward, so IMO the current structure is fine and doesn't need to be changed again.

patrickkaleta commented 5 years ago

I have now also improved the status notifications as proposed by Denis. In addition to the log messages in the BE, Admins and developers now see directly in the FE, if Emikat has been triggered or not, after updating a Study (as well as whether or not a Recalculation will be evoked).

Only exception: changing the Study area doesn't show those notifications, since the Study is updated inside the map-component via REST and no page-reload is triggered. In those cases there will be only the log messages in the BE available. status messages

p-a-s-c-a-l commented 5 years ago

Can you confirm that your apps are getting correct values for emission_scenario, time_period,...?

yes, it is.

DenoBeno commented 5 years ago

So probably that's an issue of the taxonomy / paragraph / whatever? We need value not meaning. I tried to fix it in csis-helpers-module, but it didn't work. ...

As explained in https://github.com/clarity-h2020/csis-helpers-module/issues/14#issuecomment-535830997, this is not really a bug, rather a design feature that we'll need in the future. For now, you can ignore it, I have configured the "DP variable meanings" entries in such a way that they deliver the same information as "DP variables".

That's OK as long as we only use the variables with EMIKAT. The moment we start implementing the variables support for other resources, there will be more than one "DP variables" entry possible for each "DP variable meanings" entry. E.g., the "historical" time period will map to "Baseline" for EMIKAT and to "historical" for the heat hazards, see https://github.com/clarity-h2020/data-package/issues/47#issuecomment-533827669 .

DenoBeno commented 5 years ago

Guess we can close this issue?

patrickkaleta commented 5 years ago

Guess we can close this issue?

Yes, we can close this. Emikat IDs are assigned to studies once they have all the relevant fields filled out.