CSIS URI variables in praxis 2 [HC-LE]: Local Effects for Heatwaves

DenoBeno commented 5 years ago

I have implemented a "Local Effects or Heatwaves" resource following the instructions from the https://github.com/clarity-h2020/csis/wiki/Services-endpoints-(used-by-CSIS).

The result is https://csis.myclimateservice.eu/node/1272 and now I wonder if this is aright way to do it.

For a start, this is how URIs look: https://service.emikat.at/EmiKatTst/api/scenarios/${emikat_id}/feature/view.2974/table/data?rownum=1000&filter=PROJECT%3D${study_variant}&filter=PERIOD%3D${time_period}&filter=RCP%3D${emissions_scenario}&filter=FREQUENCE%3D${event_frequency}&filter=SZ_ID%3D$emikat_id

That is, this resource has following variables: ${emikat_id}, ${study_variant}, ${time_period}, ${emissions_scenario}, and ${event_frequency}

And this are the allowed values for the variables:

Study_variant can be either isn't known until the user decides on some adaptation options.

DenoBeno commented 5 years ago

Potential Issues:

Study_variant isn't known until the user decides on some adaptation options. Or we start with "baseline" and "adaptation-01" being the same?
From the wiki, I understand that we also need a "baseline" time period. OK, this can be easily added.

Wiki also claims the need for the baseline RPC, but this makes no sense to me. At baseline time period all RPCs must deliver the same result and there is AFAIK no scenario claiming that the future climate will remain the same. Do we need this?

Question: is this the way to go?

humerh commented 5 years ago

Please remark: I have now adapted the name of the FILTER Column names: STUDY_VARIANT TIME_PERIOD EMISSIONS_SCENARIO EVENT_FREQUENCY

See also discussions in https://github.com/clarity-h2020/data-package/issues/42 and the definition in the wiki page: https://github.com/clarity-h2020/csis/wiki/Services-endpoints-(used-by-CSIS)

p-a-s-c-a-l commented 5 years ago

@humerh: I replaced the template URLs listed under Local Effects or Heatwaves with some variables and got the following errors:

p-a-s-c-a-l commented 5 years ago

Ignoring STUDY_VARIANT and EMISSION_SCENARIO (which is not needed for time period baselkine), I get an empty result for https://service.emikat.at/EmiKatTst/api/scenarios/2846/feature/view.2974/table/data?rownum=1000&filter=TIME_PERIOD%3D%27Baseline%27&filter=EVENT_FREQUENCY%3D%27rare%27&filter=SZ_ID%3D2846

humerh commented 5 years ago

You have to use the correct spelling: https://service.emikat.at/EmiKatTst/api/scenarios/2846/feature/view.2974/table/data?rownum=1000&filter=TIME_PERIOD%3D%27Baseline%27&filter=EVENT_FREQUENCY%3D%27Rare%27&filter=SZ_ID%3D2846

But this was not only your error, I had some errors to correct. I hope, it is now ok!

humerh commented 5 years ago

The correct spelling is also neccessary for WMS call:

https://service.emikat.at/geoserver/clarity/wms?service=WMS&version=1.1.0&request=GetMap&layers=clarity%3Aview.2974&bbox=4672000.0%2C1979500.0%2C4687500.0%2C1988000.0&width=768&height=421&srs=EPSG%3A3035&format=image%2Fgif&CQL_FILTER=STUDY_VARIANT%3D%27BASELINE%27%20and%20TIME_PERIOD=%27Baseline%27%20AND%20EMISSIONS_SCENARIO=%27Baseline%27%20AND%20EVENT_FREQUENCY=%27Rare%27%20AND%20SZ_ID=2846&styles=T_A

p-a-s-c-a-l commented 5 years ago

@humerh OK, great. This works:

simple-table

Could you please rename Ocas~s~ional to Occasional?

humerh commented 5 years ago

Could you please rename Ocassional to Occasional? Done

DenoBeno commented 5 years ago

Hehe, I'm soooo dyslexic. This time I managed to put a correct value in taxonomy (hail spellchecker!) without even noticing the issue....

DenoBeno commented 5 years ago

You have to use the correct spelling: https://service.emikat.at/EmiKatTst/api/scenarios/2846/feature/view.2974/table/data?rownum=1000&filter=TIME_PERIOD%3D%27Baseline%27&filter=EVENT_FREQUENCY%3D%27Rare%27&filter=SZ_ID%3D2846

But this was not only your error, I had some errors to correct. I hope, it is now ok!

This returns empty set atm :-(

By the way: according to this discussion thread, single quotes should not be used in URIs. However, when I try removing them, the call fails completely. replacing %3D with = is OK, replacing %27 with single quotes causes the call to fail completely. It would be good if we could dispose with %27, to make the URIs more readable => less probable to make an error.

DenoBeno commented 5 years ago

The correct spelling is also necessary for WMS call:

https://service.emikat.at/geoserver/clarity/wms?service=WMS&version=1.1.0&request=GetMap&layers=clarity%3Aview.2974&bbox=4672000.0%2C1979500.0%2C4687500.0%2C1988000.0&width=768&height=421&srs=EPSG%3A3035&format=image%2Fgif&CQL_FILTER=STUDY_VARIANT%3D%27BASELINE%27%20and%20TIME_PERIOD=%27Baseline%27%20AND%20EMISSIONS_SCENARIO=%27Baseline%27%20AND%20EVENT_FREQUENCY=%27Rare%27%20AND%20SZ_ID=2846&styles=T_A

Based on this, I have entered this link for the WMS in https://csis.myclimateservice.eu/node/1272

https://service.emikat.at/geoserver/clarity/wms?service=WMS&version=1.1.0&request=GetMap&layers=clarity%3Aview.2974&bbox=4672000.0%2C1979500.0%2C4687500.0%2C1988000.0&width=768&height=421&srs=EPSG%3A3035&format=image%2Fgif&CQL_FILTER=STUDY_VARIANT%3D%27${study_variant}%27%20and%20TIME_PERIOD=%27${time_period}%27%20AND%20EMISSIONS_SCENARIO=%27${emissions_scenario}%27%20AND%20EVENT_FREQUENCY=%27${event_frequency}%27%20AND%20SZ_ID=${emikat_id}&styles=T_A

The problem is that this isn't working now. I'm not going to update other links until I can test them. :-(

DenoBeno commented 5 years ago

Ignoring STUDY_VARIANT and EMISSION_SCENARIO (which is not needed for time period baselkine), I get an empty result for https://service.emikat.at/EmiKatTst/api/scenarios/2846/feature/view.2974/table/data?rownum=1000&filter=TIME_PERIOD%3D%27Baseline%27&filter=EVENT_FREQUENCY%3D%27rare%27&filter=SZ_ID%3D2846

One can indeed ignore some filters, but I would advise against it in production system.

For the time period "baseline", all emission scenarios should indeed return the same, so ignoring this one may be safe, as long as EMIKAT isn't producing any unnecessary data... Ideal solution would be if EMIKAT would ignore this filter for baseline timeframe and always return the same data but I suppose this is overkill...
For the time period baseline, the user could be interested to see what happens when adaptation options are applied and EMIKAT might be actually producing the data. Ignoring this parameter could therefore be dangerous.

It's IMO better to always explicitly set a filter and check if anything is returned than not to set it and risk getting double the amount of the data...

p-a-s-c-a-l commented 5 years ago

Now, the following URLs return an empty results: :-( Yesterday, they used to work.

@humerh Can you please fix this ASAP as I need a dependable API for client development. Thanks.

DenoBeno commented 5 years ago

This works again now. Thanks, Heinrich.

humerh commented 5 years ago

I fixed the problem.

DenoBeno commented 5 years ago

OK, adding entries to the resource:

this works: https://service.emikat.at/EmiKatTst/api/scenarios/2846/feature/view.2974/table/data?rownum=1000&filter=STUDY_VARIANT%3D%27BASELINE%27&filter=TIME_PERIOD%3D%27Baseline%27&filter=EMISSIONS_SCENARIO%3D%27Baseline%27&filter=EVENT_FREQUENCY%3D%27Rare%27&filter=SZ_ID%3D2846 thus: https://service.emikat.at/EmiKatTst/api/scenarios/${emikat_id}/feature/view.2974/table/data?rownum=1000&filter=STUDY_VARIANT%3D%27${study_variant}%27&filter=TIME_PERIOD%3D%27${time_period}%27&filter=EMISSIONS_SCENARIO%3D%27${time_period}%27&filter=EVENT_FREQUENCY%3D%27${event_frequency}%27&filter=SZ_ID%3D${emikat_id}

and same for geojson and csv should be OK. Problem: I can add the csv one to "download path" and I would like to add the others to "service paths (references)", but what should be the type?

For now, I have added all three of them as @resource, but this doesn't make sense. We need more types.

And https://service.emikat.at/EmiKatTst/api/scenarios/2846/feature/view.2974/table/geojson?rownum=1000&filter=STUDY_VARIANT%3D%27BASELINE%27&filter=TIME_PERIOD%3D%27Baseline%27&filter=EMISSIONS_SCENARIO%3D%27Baseline%27&filter=EVENT_FREQUENCY%3D%27Rare%27&filter=SZ_ID%3D2846

is taking forever, I thought that it's broken but it did work in the end.

DenoBeno commented 5 years ago

Status:

https://csis.myclimateservice.eu/node/1272 was updated with correct URIs.
all three service paths (references) for the csv/json/geojson have the same type "@resource". We need a way to differentiate here.
mapview and especially the geojson take too long to render.
with time_period='Baseline', one has to use EMISSIONS_SCENARIO='baseline'
replacing %3D with = in the URI works, but %27 can't be replaced by ´ (or by ` or by '). Ah wait: replacing it with ' does work, just can't be done in MS word.

So this works: https://service.emikat.at/EmiKatTst/api/scenarios/2846/feature/view.2974/table/data?rownum=1000&filter=STUDY_VARIANT='BASELINE'&filter=TIME_PERIOD='20110101-20401231'&filter=EMISSIONS_SCENARIO='rcp26'&filter=EVENT_FREQUENCY='Rare'&filter=SZ_ID=2846

I'll change it all in the https://csis.myclimateservice.eu/node/1272 so that the URIS are easier to read.

DenoBeno commented 5 years ago

The "Baseline" emission scenario has been missing. I have added it now and gave it a good explanation, but this means that our software will need to know that baseline scenario can only be combined with baseline time. Hard-coded in the logic. :-(

https://csis.myclimateservice.eu/taxonomy/term/2256

I have also removed all %XX signs from the links used. E.g. the one used for the map now is:

https://service.emikat.at/geoserver/clarity/wms?

plus:

service=WMS&version=1.1.0&request=GetMap&layers=clarity:view.2974&bbox=4672000.0,1979500.0,4687500.0,1988000.0&width=768&height=421&srs=EPSG:3035&format=image/gif&CQL_FILTER=STUDY_VARIANT='${study_variant}' and TIME_PERIOD='${time_period}' AND EMISSIONS_SCENARIO='${emissions_scenario}' AND EVENT_FREQUENCY='${event_frequency}' AND SZ_ID=${emikat_id}&styles=T_A

(formatting breaks here, so I took it apart) If this works - great, it's easier to read. It does work when I use such URIs in a browser, but browser automatically translates URIs as necessary...

p-a-s-c-a-l commented 5 years ago

There are even more pitfalls: \${emikat_id} in the URL is sometimes replaced to $%7Bemikat_id%7D(by the browser?).

DenoBeno commented 5 years ago

That looks really promising now. I looked at maximal damage and damage probability and it changes with period and scenario as one would expect. :-)

DenoBeno commented 5 years ago

Oh, what a happy day! :-(

Is this affecting us when we use these links from JavaScript programs?

And if yes: how to handle this?

On Fri, 20 Sep 2019, 17:02 Pascal Dihé, notifications@github.com wrote:

There are even more pitfalls: ${emikat_id} in the URL is sometimes replaced to $%7Bemikat_id%7D(by the browser?).

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/clarity-h2020/data-package/issues/45?email_source=notifications&email_token=AAWTC7U254O2VLAZPA5BYXDQKTQZ3A5CNFSM4IXC6P3KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7G7EAQ#issuecomment-533590530, or mute the thread https://github.com/notifications/unsubscribe-auth/AAWTC7WAIE6D66ME2YYY3O3QKTQZ3ANCNFSM4IXC6P3A .

DenoBeno commented 5 years ago

I just tried the test table introduced by @p-a-s-c-a-l in https://github.com/clarity-h2020/data-package/issues/46#issuecomment-533592856 on local effects of heatwaves. It's very easy, just do this:

Log in on the EMIKAT on the browser.
follow the link https://csis.myclimateservice.eu/apps/simple-table-component/build/?emikat_id=2846
change the "view.2975" to "view.2974" in the uri at the top of the table and the result is:

DenoBeno commented 5 years ago

Heinrik has removed the "&filter=SZ_ID=${emikat_id}" part from the table URIs - I'll rewrite the resource now. (Someone already did this, except or the "download path")

Less good news is that we need to define a way to handle three different temperatures that are returned by this view: T_A, T_MRT, T_UTCI

styles=T_A for Apparent temperature
styles=T_TMRT for Mean Radiant Temperature
styles=T_UTCI for UTCI Temperature

In fact, this is also delivering the "discomfort level"...

For now, it is only possible to differentiate between these three cases (and discomfort?) in the maps, so I will add two more resources. But beware: the table data is exactly the same for al three at the moment, this needs fixing.

DenoBeno commented 5 years ago

OK, done. I have also removed the surplus hazards tags from all three, to avoid duplication in the data tab.

Beware: since the tables are the same for all three atm., I haven't duplicated the related yet. This needs to be done once these references can be split.

Oh jee. Map component is not showing this properly. Opening a new issue report: :-(

p-a-s-c-a-l commented 4 years ago

Implemented and working.

clarity-h2020 / data-package

CSIS URI variables in praxis 2 [HC-LE]: Local Effects for Heatwaves #45

Could you please rename Ocas~s~ional to Occasional?