Open claudiahahn opened 4 years ago
Yes, the data has to be put on ATOS GeoServer, therfore I added @DanielRodera
FYI: ATM we are experiencing some issues with the new GeoServer instance, these have to be resolved before we can add any more data to the server.
Are these resources being served on the SMHI server (via WMS or WCS) at the moment?
If yes, we think we should concentrate Atos efforts on the generation of the input layers and keep the SWICCA layers as they are for the moment. Later on, we can add to the Geoserver without a problem.
If not, we would need one layer for each period formatted as a GeoTIFF. Also, Geoserver datastore´s name and the names of the layers to be served.
FYI: ATM we are experiencing some issues with the new GeoServer instance, these have to be resolved before we can add any more data to the server.
Thank you for reporting, we are working on it.
Thank you @p-a-s-c-a-l and @DanielRodera!
If the data is available on the SMHI server, it would be enough to insert the link to the respective file in the resource description in CSIS, right?
@LenaStr what do you think? It seems to be the easiest way to integrate the data right now.
it would be enough to insert the link to the respective file in the resource description in CSIS, right?
Only if the data served via Web Map Service, otherwise it has to imported into ATOS GeoServer. It seems that the origin of the data is this THREDDS Data Server, which has an WMS endpoint. So well' just need to generate the respective getMap
requests pointing to the resources on SWICCA THREDDS.
So what we would need now is for each of the three resources in this CKAN datasets the respective links to the corresponding NetCDF file on swicca.smhi.se/thredds.
The data is now also available in the Copernicus CDS which is a better source for long time preservation than the SMHI server. https://cds.climate.copernicus.eu/cdsapp#!/dataset/sis-water-quantity-swicca?tab=overview
Please include Yeshewa in this discussion as he knows the details for this datasets and which were selected for Clarity. I do not have time to act on this for the moment. (Yesehewa, plesa ask if you need some support on creating the resourses)
/Lena
Från: Pascal Dihé [mailto:notifications@github.com] Skickat: den 3 juni 2020 15:23 Till: clarity-h2020/data-package Kopia: Strömbäck Lena; Mention Ämne: Re: [clarity-h2020/data-package] Integrate SWICCA data in European data package (#68)
it would be enough to insert the link to the respective file in the resource description in CSIS, right?
Only if the data served via Web Map Service, otherwise it has to imported into ATOS GeoServer. It seems that the origin of the data is this THREDDS Data Serverhttps://swicca.smhi.se/thredds/catalog/SWICCA/catalog.html, which has an WMS endpoint. So well' just need the generate the respective getMap requests pointing to the resources on SWICCA THREDDS.
So what we would need now is for each of the the three resources in this CKAN datasetshttps://ckan.myclimateservice.eu/dataset/swicca-temperature-precipitation-hydrological-variables the respective links to the corresponding NetCDF file on swicca.smhi.se/thredds.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/clarity-h2020/data-package/issues/68#issuecomment-638193678, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AB2NFIBKFAYDXVP2ZYKUTJLRUZFEJANCNFSM4NRPXUAQ.
I couldn't find Yeshewa. Did he create a github account? I will write him an email
@yhundecha
that was a mistake, sorry
O.K. now we have three data sources:
Which one is the right one? What data has to be made available and how? As mentioned before, data residing on SMHI Thredds can be accessed via WMS API, so we wouldn't have to import it into ATOS GeoServer in order to put it into the European Data Package and show it on the map.
The data in all the three sources are the same. The first and third do have more data than we would like to have into the European Data Package. The data that need to be put into the data package are those already put in CLARITY CKAN.
O.K. thanks! So we are talking about three datasets:
Where do i find them on SMHI Thredds?
All three are at 0.5 degree grid. The 50km is mentioned as an approximate linear grid size.
So where do I find e.g.
in https://swicca.smhi.se/thredds/catalog/SWICCA/0.5/catalog.html ?
So this means they are not available from THREDDS WMS. In this case, the data has to be converted to GeoTIFF and made available on ATOS GeoServer. Once this is done, I can have a look at the data package and create the template resources.
@DanielRodera When importing the data, it is important that the layer names follow exactly the same naming convention (time period, RCP, etc.) as the existing ZAMG HC layers, otherwise we cannot show them on the CSIS map.
Who will convert the data? @DanielRodera, is it possible for ATOS to do that? or @yhundecha, can you provide the respective geotiff files?
AFAIR the conversion from NetCDF to GeoTIFF was done by @LauraMTG and/or @ghilbrae, so perhaps they can help out here?
I will try to get it converted.
Data is converted, but it looks that I have trouble accessing the GeoServer to put the data on. Any tips on how I can proceed?
I have sent you an email with the access details for the GeoServer.
I have now uploaded the converted GeoTIFF files, together with the original NetCDF files here: /clarityftp/europe/hazard_indices/hydrology/
Sorry, we missed this issue.
@LauraMTG and I, mainly Laura, did the conversion from what was provided by Robert in NetCDF to tif. I think Laura created an script that helped with the conversion and then uploading the layers to our server. This is obviously not set up at the ATOS geoserver. We could try and see if these new layers work with the current script and then upload them from our server to ATOS' or we can try to figure out a way to do it in ATOS' without going through us.
@DanielRodera we should address this problem, maybe on Monday's telco and see what options we have.
So this means they are not available from THREDDS WMS. In this case, the data has to be converted to GeoTIFF and made available on ATOS GeoServer. Once this is done, I can have a look at the data package and create the template resources.
@DanielRodera When importing the data, it is important that the layer names follow exactly the same naming convention (time period, RCP, etc.) as the existing ZAMG HC layers, otherwise we cannot show them on the CSIS map.
The naming of the files was done by Robert originally. What we have to ensure is that nothing is changed by geoserver as it happened last month.
Hello @yhundecha and @RobAndGo
I'm going to process the NetCDF to geotiff conversion for this /clarityftp/europe/hazard_indices/hydrology/ and take the layers to the ATOS server for publication. I'll let you know when it's available.
Dear @LauraMTG: from the entry above it sounds like @yhundecha already has converted the netcdf data to geotiff and put it here: "I have now uploaded the converted GeoTIFF files, together with the original NetCDF files here: /clarityftp/europe/hazard_indices/hydrology/" (from Yeshewa) maybe I have missed something, but I just want to make sure we are not doing things twice
Okay, thanks for the advice. I'd lost that. Regards
But thank you very much for the offer to convert it!! I guess now @DanielRodera has to take over?
Yes, perhaps with help from @ghilbrae and @LauraMTG regarding the styles.
any progress to be reported here @DanielRodera ?
hi all,
First of all sorry about the delay. I didn't get any alert for this issue.
I have uploaded the SWICCA dataset (GeoTiff) in the Geoserver,
Could you please confirm the file names are the correct ones? these names will be the ones published in Geoserver
Thanks! Which file names? There are now 452 layers on GeoServer.
The layers are still not published in Geoserver, I have copied the files (GeoTiffs) from the Atos FTP server to the Geoserver instance.
Before publishing them all, I would like to be sure that the name of the files I have copied from the FTP are the ones that should be published as the layer name. The Geoserver importer plugin (we used this plugin to publish all the MTG layers) will take the name of the file and publish the layer with this name.
The file names can be checked in the Atos FTP: /clarityftp/europe/hazard_indices/hydrology/
Thanks!
Could you please post the list of files. It seems that I don't have access to the sFTP server or the IP has changed (5.79.69.49).
Apart from that, the next question is who will actually create the respective Template Resources and add them the Data Packages?
Hi Pascal,
Indeed, the sFTP server IP has changed, the new one is 95.211.163.27. Your user/pass should be the same as before,
I have attached a file with some file names for SWICCA datasets: SWICCA-names.txt
FLOOD RECURRENCE
cout-yearmax_returnPeriod5_rel_EUR-44_rcp45_IMPACT2C_QM-EOBS_2041-2070_1971-2000_remap0.5.tif cout-yearmax_returnPeriod5_rel_EUR-44_rcp85_IMPACT2C_QM-EOBS_2041-2070_1971-2000_remap0.5.tif cout-yearmax_returnPeriod5_rel_EUR-44_rcp85_IMPACT2C_QM-EOBS_2071-2100_1971-2000_remap0.5.tif cout-yearmax_returnPeriod50_ref_EUR-44_rcp85_IMPACT2C_QM-EOBS_1971-2000_remap0.5.tif cout-yearmax_returnPeriod2_rel_EUR-44_rcp85_IMPACT2C_QM-EOBS_2071-2100_1971-2000_remap0.5.tif cout-yearmax_returnPeriod50_ref_EUR-44_rcp26_IMPACT2C_QM-EOBS_1971-2000_remap0.5.tif cout-yearmax_returnPeriod50_rel_EUR-44_rcp26_IMPACT2C_QM-EOBS_2011-2040_1971-2000_remap0.5.tif cout-yearmax_returnPeriod50_rel_EUR-44_rcp26_IMPACT2C_QM-EOBS_2041-2070_1971-2000_remap0.5.tif
RIVER FLOW
cout-perc99_rel_EUR-44_rcp45_IMPACT2C_QM-EOBS_2071-2100_1971-2000_remap0.5.tif cout-perc99_rel_EUR-44_rcp85_IMPACT2C_QM-EOBS_2041-2070_1971-2000_remap0.5.tif cout-perc99_rel_EUR-44_rcp85_IMPACT2C_QM-EOBS_2071-2100_1971-2000_remap0.5.tif cout-perc90_rel_EUR-44_rcp85_IMPACT2C_QM-EOBS_2071-2100_1971-2000_remap0.5.tif cout-perc95_ref_EUR-44_rcp45_IMPACT2C_QM-EOBS_1971-2000_remap0.5.tif cout-perc95_ref_EUR-44_rcp85_IMPACT2C_QM-EOBS_1971-2000_remap0.5.tif cout-perc95_rel_EUR-44_rcp26_IMPACT2C_QM-EOBS_2011-2040_1971-2000_remap0.5.tif
WATER RUNOFF
out-perc95_rel_EUR-44_rcp85_IMPACT2C_QM-EOBS_2011-2040_1971-2000_remap0.5.tif crun-periodavg_rel_EUR-44_rcp85_IMPACT2C_QM-EOBS_2071-2100_1971-2000_remap0.5.tif crun-periodavg_rel_EUR-44_rcp85_IMPACT2C_QM-EOBS_2041-2070_1971-2000_remap0.5.tif crun-periodavg_rel_EUR-44_rcp45_IMPACT2C_QM-EOBS_2071-2100_1971-2000_remap0.5.tif cout-perc99_ref_EUR-44_rcp45_IMPACT2C_QM-EOBS_1971-2000_remap0.5.tif cout-perc99_ref_EUR-44_rcp26_IMPACT2C_QM-EOBS_1971-2000_remap0.5.tif cout-perc95_rel_EUR-44_rcp45_IMPACT2C_QM-EOBS_2011-2040_1971-2000_remap0.5.tif cout-perc80_rel_EUR-44_rcp85_IMPACT2C_QM-EOBS_2041-2070_1971-2000_remap0.5.tif
Unfortunately, these names are not harmonised with what we have already regarding other European Hazard indices.
Dates should be encoded in the same format as for the other hazard layers, e.g. europe:CI_Tn10prcp2620710101-21001231_ensstd and europe:CI_Tn10prcp4520110101-20401231_ensmean. We could resolve that at the level of variables, however.
Some resources cover two time periods e.g. crun-periodavg_rel_EUR-44_rcp45_IMPACT2CQM-EOBS2071-2100_1971-2000_remap0.5.tif. This is something that we haven't foreseen in the implementation since we were not aware that this can happen. Therefore we cannot resolve that just at the level the resource meta-data (Template Resources). The best thing we could do is to duplicate the file and create two files with distinct time periods, e.g. crun-periodavg_rel_EUR-44_rcp45_IMPACT2CQM-EOBS2071-2100_remap0.5.tif and crun-periodavg_rel_EUR-44_rcp45_IMPACT2CQM-EOBS1971-2000_remap0.5.tif. But I don't know if this makes sense, in particular related to the time coverage 1971-2000 and 2071-2100. So according to the name, this resources seems to cover future and a historical time period for rcp45❓
Related to that and a more severe problem is that we don't support layers covering the historial time period with RCP, e.g. cout-yearmax_returnPeriod10_refEUR-44rcp26_IMPACT2CQM-EOBS1971-2000_remap0.5.nc. This is something we've never foreseen, in our scenario definitions and ATM I don't know how to handle this without changing the implementation.
But perhaps those are just errors in naming the resources @LenaStr @yhundecha ?
I'm not an expert on those SWICCA datasets, but perhaps the datasets with the two timeperiods, e.g. crun-periodavg_rel_EUR-44_rcp45_IMPACT2C_QM-EOBS_2071-2100_1971-2000_remap0.5.tif are values of the period 2071-2100 relative to the reference period of 1971-2000. This could explain the addition of "_rel" after the index name "crun-periodavg".
If this is the case, a simple solution could be to remove the reference time period of 1971-2000 from the name with the assumption that all indices with "_rel" are always taken relative to this reference.
Regarding the inclusion of "rcp" with the reference dataset, I am really not sure what is going on here. Do the datasets: cout-perc99_ref_EUR-44_rcp45_IMPACT2C_QM-EOBS_1971-2000_remap0.5.tif cout-perc99_ref_EUR-44_rcp26_IMPACT2C_QM-EOBS_1971-2000_remap0.5.tif really show differences from one another? From experience with the EURO-CORDEX datasets, the rcp-scenarios are only defined from the year 2010 onwards.
There is a reasson for inclusion of two time periods for the data pertaining to the future time horizons. The values were computed as changes relative to the reference period 1971-2000 and that is why the reference period is always indicated. This can possibly be handled by omitting the reference period as the same reference period is always used.
The reference values for the different rcps should in principle be the same. However, the available gcm-rcm combinations for the different rcps are not the same (which ones are avilable for each rcp can be seen in the metadata included in the data set). Since the values for the future time horizons were computed relative to the reference period, corresponding ensemble mean values of the reference period were computed for each rcp based on the avalabe gcm-rcm combinations for each rcp and this explains the differences in the reference values for the different rcps. But I do not know how we can handle this in the system.
Thanks for the explanation!
There is a reasson for inclusion of two time periods for the data pertaining to the future time horizons. The values were computed as changes relative to the reference period 1971-2000 and that is why the reference period is always indicated. This can possibly be handled by omitting the reference period as the same reference period is always used.
In this case we can simply ignore the reference period part in the filename when creating the map layers as it is just informative but not a template parameter (${time_period}
). So no name changes are needed.
The reference values for the different rcps should in principle be the same. However, the available gcm-rcm combinations for the different rcps are not the same (which ones are avilable for each rcp can be seen in the metadata included in the data set). Since the values for the future time horizons were computed relative to the reference period, corresponding ensemble mean values of the reference period were computed for each rcp based on the avalabe gcm-rcm combinations for each rcp and this explains the differences in the reference values for the different rcps. But I do not know how we can handle this in the system.
O.K. I see. For the reference / historic period (1971-2000), the system currently excepts as placeholder for rcp something like 'baseline' or 'historial' for ${emissions_scenario}
:
So our scenario management and the map component, respectively, support exactly one resource for the reference period. We have to decide which one to use out of the three. I propose to always show the 1971-2000_rc26 combination on the map when the users select the baseline scenario. WDYT?
With the current files we can create templates in the format cout-perc99_rel_EUR-44_${emissions_scenario}_IMPACT2C_QM-EOBS_${time_period}_1971-2000_remap0.5.tif
.
@DanielRodera I think you can go ahead with publishing the data sources as map layers. I will then create a Data Package Resource for 'Flood recurrence (0.5 deg grid)' for testing the integration in CSIS.
Alright! If we have to choose only one reference data, I will go for rcp4.5 as it contains more gcm-rcm combinations than the others and it also contains all combinations that are in the others.
@maesbri, could you point @DanielRodera to this issue? Maybe he did not get an allert for it?
Hi,
I have published the SWICCA dataset as map layers, If you have any doubt please let me know
I've updated the Flood recurrence resource and added a new 'service path (reference)' with the following template URL:
https://geoserver.myclimateservice.eu/geoserver/europe/wms?service=WMS&version=1.1.0&request=GetMap&layers=cout-perc99_rel_EUR-44_${emissions_scenario}_IMPACT2C_QM-EOBS_${time_period}_1971-2000_remap0.5&srs=EPSG:3035&format=image/png
This particular resource can now be used as an example for updating the other SWICCA resources in the European Data Package. Whoever is going to do that has besides creating the respective 'service path (reference)' also to add the correct DP_Variables and DP_ServiceType tags:
@DanielRodera It would help to rename all baseline layers to include the reference period, so that we don't need to duplicate the resource. Example:
cout-perc99_rel_EUR-44_rcp45_IMPACT2C_QM-EOBS_1971-2000_remap0.5
has to be renamed to
cout-perc99_rel_EUR-44_rcp45_IMPACT2C_QM-EOBS_1971-2000_1971-2000_remap0.5
On this map&emikat_id=3209&datapackage_uuid=2434ce93-93d4-4ca2-8618-a2de768d3f16&write_permissions=1&time_period=Baseline&emissions_scenario=Baseline&event_frequency=Frequent&grouping_tag=taxonomy_term--hazards) which is configured to shoe the baseline (historical) scenario, you can see that the Flood recurrence layer is now available:
Of course, nothing is shown on the map since the layers are not yet available on GeoServer.
You can check the request URL in browser network requests (CTRL+SHIFT+i):
Note the we use rcp45 in the layer name as decided previously.
@therter @DanielRodera
I don't know what's wrong here. Perhaps you can find the error.
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE ServiceExceptionReport SYSTEM "http://geoserver.myclimateservice.eu/geoserver/schemas/wms/1.1.1/WMS_exception_1_1_1.dtd">
<ServiceExceptionReport version="1.1.1" >
<ServiceException code="LayerNotDefined" locator="layers">
Could not find layer europe:cout_yearmax_returnPeriod100_rel_EUR_44_rcp85_IMPACT2C_QM_EOBS_2071-2100_1971_2000_remap0_5
</ServiceException>
</ServiceExceptionReport>
Is there there difference? Only the last one is working?!
cout_yearmax_returnPeriod100_rel_EUR_44_rcp85_IMPACT2C_QM_EOBS_2071-2100_1971_2000_remap0_5
cout_yearmax_returnPeriod100_rel_EUR_44_rcp85_IMPACT2C_QM_EOBS_2071-2100_1971_2000_remap0_5
cout_yearmax_returnPeriod100_rel_EUR_44_rcp85_IMPACT2C_QM_EOBS_2071_2100_1971_2000_remap0_5
I have published the SWICCA dataset as map layers.
Thanks!
I've noticed that the layer names are not the same as the filenames on sFTP:
cout_yearmax_returnPeriod100_rel_EUR_44_rcp85_IMPACT2C_QM_EOBS_2071_2100_1971_2000_remap0_5
vs cout-yearmax_returnPeriod100_rel_EUR-44_rcp26_IMPACT2C_QM-EOBS_2071-2100_1971-2000_remap0.5
I'll resolve this at the level of the Data Package Resource.
O.K. as an example, I've changed the Flood recurrence resource and added 4 WMS services paths, one for each available return period. In the maps, this looks like:
This particular resource can now be used as an example for updating the other SWICCA resources in the European Data Package. Whoever is going to do that has besides creating the respective 'service paths (reference)' also to add the correct DP_Variables and DP_ServiceType tags:
River Flow now available on Map&emikat_id=3209&datapackage_uuid=2434ce93-93d4-4ca2-8618-a2de768d3f16&write_permissions=1&time_period=Baseline&emissions_scenario=Baseline&event_frequency=Frequent&grouping_tag=taxonomy_term--hazards)
I have been trying to create the template resource. I tried to get access to where I can create it for water runoff: [(https://csis.myclimateservice.eu/admin/content?title=&type=data_package&status=All&langcode=All)] Unfortunately, it looks that I get stalled. When I try to click on the button 'Edit as simplifiedDataPackage' in front of European wide data package, I get notified that I am not authorized. When I click on the data package, I see the list of all the resouces but with no option to do anything. Any tips?
When I click on the data package, I see the list of all the resouces but with no option to do anything. Any tips?
I think you should do changes to the data package on the dev instance (https://csis-dev.myclimateservice.eu). You should have the required permissions there and during the next synchronisation, the changes will be transferred to the public instance.
So far the SWICCA data is not included in the European data package yet.
@p-a-s-c-a-l , @LauraMTG , @luis-meteogrid, @ghilbrae: What would be the best option to do that?
Lena has provided a link to the data via CKAN (e.g.: https://ckan.myclimateservice.eu/dataset/swicca-temperature-precipitation-hydrological-variables/resource/9543069f-c2fd-48aa-bc90-14d1bb688702)
For the climate indices from ZAMG, it was possible to upload netcdf files to the CLARTY ftp and meteogrid proccessed them and put it on their geoserver. Now the ATOS geoserver is used, right?
Note: the units for the indices provided for the future periods display the "percentage change relative to the reference period (%)"