search

ngds / ckanext-ngds-bku03232018

CKAN extension for NGDS-specific customizations
Other
5 stars 13 forks source link

Geothermal prospector support in NGDS CKAN v2 #580

Closed smrgeoinfo closed 9 years ago

smrgeoinfo commented 9 years ago

see #404, #459 Function works in NGDS CKAN v1. the Button should show up with the WMS distribution links for search results that have WMS distributions @ccaudill here's some notes from Dan Getman e-mail: From: Getman, Dan Dan.Getman@nrel.gov Sent: Tuesday, August 19, 2014 1:52 PM To: Adrian Sonnenschein; ... 1) Just a point of clarification on parameters. The parameters that you should be using, and are for the most part, have not changed and include:

wmsHost: a string (URL escaped) containing the endpoint URL for a WMS; this is the part of the URL that precedes the WMS parameters, basically a WMS request with no parameters part (leave out the question mark). REQUIRED

wmsLayerName: the name of a layer in the WMS to display. ONLY one layer name allowed. REQUIRED

wfsHost: a string (URL escaped) containing the endpoint URL for a WFS that provides feature access to the data displayed by the WMS. The string argument is the part of the URL that precedes the WFS parameters, basically a WFS request with no parameters part (leave out the question mark). OPTIONAL

wfsFeatureTypeName: the name of the wfs feature type exactly as it needs to be specified in the TypeName WFS parameter; the namespace qualifier might or might not be required, I think that might depend on the service implementation. OPTIONAL

You can also use: zoomlevel: an integer to set the desired zoom level mapcenter: a string (URL escaped) containing the coordinates for the center of the map "lon,lat" overlaylayer: Setting this to 0 will turn all layers off by default. Setting this to anything else may be unpredictable.

It is important to note that the layer ids for baselayer parameter have indeed changed and will likely change in the future as well. This parameter has been inadvertently included in some of our example URLs (cutting and pasting error in emailing unfortunately) but was not intended to be used to call GTP from NGDS. This parameter, and many others that you may see loaded into the URL by GTP, are for our use in allowing users to send other users URLs.

2) The URL that you sent has a couple of other issues.

First, setting the baselayer to a now invalid ID is causing no base layer to be loaded. This should explain the gray base map that you observed.

Second, the layer that you are referencing only shows up at zoom level 7 and you have zoom level 1 specified. The result is that the layer actually loads, but is not visible.

You can specify a zoom level of 7 and it will work: https://maps-stage.nrel.gov/geothermal-prospector/#/?zoomlevel=7&wmsHost=ht tp://www.geocommunicator.gov/arcgis/services/PLSS/MapServer/WMSServer&wmsLa yerName=11

Third, the map service that you sent is reporting a non-working legend URL. I'm not sure what the issue is with this, but it will prevent the layer from displaying a legend.

Next Steps:

On my end, we will make sure that the baselayer parameter is handled such that any layer id that is not valid will result in a default base layer being loaded, which should prevent the gray base map issue that you observed.

Also, to address your main concern, I believe it unlikely that the URL parameters which are specified here and in our original emails providing the descriptions will change. If that does occur, we will notify you beforehand to make sure that it will not create a problem on your end.

jessicagood commented 9 years ago

Testing http://uat-ngds.reisys.com/dataset/indiana-borehole-temperatures-wms-wfs-2 (as specified in https://github.com/ngds/ckanext-metadata/issues/14) and it looked like it worked as should. WMS was drawn and features could be queried. Though testing a few other services may have crashed the server hosting the Geothermal Prospector - or it was coincidental that the Geothermal Prospector stopped working at the same time.

JihadMotii-REISys commented 9 years ago

I've done a changing, here https://github.com/ngds/ckanext-metadata/issues/14#issuecomment-68073532. it could be a fix for this (if it wasn't from Geothermal Prospector server)

ccaudill commented 9 years ago

@JihadMotii-REISys I don't see Geothermal Prospector button in new build

JihadMotii-REISys commented 9 years ago

is the new build UAT ? otherwise, can you send me the link to the new build ? Thank you @ccaudill

ccaudill commented 9 years ago

Sorry, it's our internal servers, updated to the latest rpm. 159.87.39.4 159.87.39.5

JihadMotii-REISys commented 9 years ago

I created a dataset in UAT that is up to date with all new code/ configuration, this is the link: http://uat-ngds.reisys.com/dataset/test-dataset

In order to make geothermal button working, your internal servers has to have access to external and also geoserver needs to be set up. (if you're behind reverse proxy) ...

ccaudill commented 9 years ago

I don't understand why this opens in GP with the data that it does, and not well log data that is specified. Perhaps that doesn't matter for this test. Also, the button just vanished recently, so there shouldn't be any work with getting it configured again. You say that external servers need to be accessed and GeoServer needs to be set up, but that's already done in any installation, so I'm confused about why you mention that.

The button is there, I'm just not completely sure that it's working properly.

ykhadilkar commented 9 years ago

Hey @ccaudill

Just want to make sure you followed below steps to set up geoserver after initial installation

  1. After all the steps please update two config files. Update "ckan.hostname" variable with correct URL(without http://) in file /etc/ckan/production.ini
  2. update proxyBaseUrl in file /var/lib/tomcat6/webapps/geoserver/data/global.xml, replacing 127.0.0.1 with correct URL. Restart server after update.

Also make sure internal servers can access https://maps-stage.nrel.gov/ url. And as far as I understand Geothermal Prospector application at https://maps-stage.nrel.gov/ contacts back to CKAN node to get the data to be displayed on map. And that is why we suspect Geothermal Prospector is not able to display data from internal server.

Also I'm little confused about button visibility ... Are you saying button now shows up but when clicked shows up wrong map?

ccaudill commented 9 years ago

You're saying that after running the update rpm at https://github.com/ngds/install-and-run#updating-ngds there's other things that need to be done?

ccaudill commented 9 years ago

The button isn't there at all. Should I be looking in another place?

geo-pros-button

ykhadilkar commented 9 years ago

No ... not after update only after initial installation. If you did after initial installation then don't worry about it.

ykhadilkar commented 9 years ago

I suspect its because server not accessible from outside. Like I said above.. "as far as I understand Geothermal Prospector application at https://maps-stage.nrel.gov/ contacts back to CKAN node to get the data to be displayed on map. And that is why we suspect Geothermal Prospector is not able to display data from internal server." Is it possible to open this server to internet just make sure that is the problem?

ccaudill commented 9 years ago

We followed the instructions here to install: https://github.com/ngds/install-and-run#installation If more steps are required than are listed here, please edit that readme to include them.

ccaudill commented 9 years ago

The server is open to the internet. You should be able to hit it.

ykhadilkar commented 9 years ago

@ccaudill Can you please email be below 2 files or give me SSH access to the server?

ykhadilkar commented 9 years ago

@ccaudill in production.ini ckan.hostname was incorrect (It was default 127.0.0.1) So changed it to correct IP address. But there is something wrong with datapusher / datastore. I am not able to push the CSV data to datastore at all. @FuhuXia any idea? Also in your dataset http://159.87.39.5/dataset/vermont-thermal-conductivity99cc3 I don't see any CSV file at all ... My understanding was that the thing works only with CSV files? @JihadMotii-REISys any comments?

ccaudill commented 9 years ago

@ykhadilkar-rei You can only publish services through this CKAN UI (GeoServer) using a CSV. However, harvested tier 3 data and services will still be perfectly valid, though published through a different process (usually with ArcMap) which does not require a CSV file.

ccaudill commented 9 years ago

Can we get an update on what happened to the Geothermal Prospector button?? This is CRITICAL and should take precedence over other things.

dano-reisys commented 9 years ago

@ccaudill We are looking into it, you should hear from us soon.

dano-reisys commented 9 years ago

@ccaudill @smrazgs I was able to add the well log csv file to the datastore: http://159.87.39.5/dataset/geo-pros-datapusher1 by changing the value of ckan.site_url to 127.0.0.1

Seems like there is some kind of NAT problem going on between the local ip address and public ip address

@159.87.39.5 wget http://159.87.39.5 yields a "Connection time out" wget 127.0.0.1 works fine

Was this instance set up in Amazon?

ccaudill commented 9 years ago

No, it was set up on a local Windows server. I'm not sure how this relates the the Geo Pros button missing... Can we have that button be back this week? @dan-olaru-reisys

dano-reisys commented 9 years ago

So I did some investigation and it looks like @ykhadilkar-rei was right. The the csv data was not getting pushed to the datastore.. i believe the reason for that is that the public IP address is not getting resolved on the server. Datapusher calls the datastore api through 159.87.39.5 but it doesn't get resolved to 127.0.0.1 therefore we get a connection timeout error in datapusher.error.log

there is more info on how the datapusher works here: https://github.com/ckan/datapusher/issues/18

The Geothermal Prospector button doesn't show up because there is nothing in the datastore, so the OGC publishes invalid WMS/WFS.

dano-reisys commented 9 years ago

Assigning a domain name(final.geothermaldata.org) to 159.87.39.5 seems to have fixed the problem: http://final.geothermaldata.org/dataset/geo-pros-datapusher3 1- I had to change the value of ckan.hostname to final.geothermaldata.org in /etc/ckan/production.ini 2- I had to change the value of proxyBaseUrl to final.geothermaldata.org in /var/lib/tomcat6/webapps/geoserver/data/global.xml

I will add some notes to the troubleshooting section of https://github.com/ngds/install-and-run about this (datapusher & datastore & ckan relationship)

ccaudill commented 9 years ago

YES!!!!!! Thank you for helping us troubleshoot that! I am so glad to see that GeothermalProspector button again. Thanks much for making notes for others to troubleshoot. So, why do you think I'm not seeing it elsewhere, like http://final.geothermaldata.org/dataset/washington-well-headerse4379 where it should be an option?

kvuppala commented 9 years ago

:+1:

dano-reisys commented 9 years ago

No problem @ccaudill . Looks like the WMS/WFS is broken @ http://final.geothermaldata.org/dataset/washington-well-headerse4379/resource/bc687067-8fe1-4c52-a5d6-a888c625b87d

it references references some amazon instance...

http://ec2-50-18-49-187.us-west-1.compute.amazonaws.com/arcgis/services/Non-Cached/DNR_WellHeaders/MapServer/WMSServer?request=GetCapabilities&service=WMS

ccaudill commented 9 years ago

That was a bad example ... that link did not work. But this one does, no button: http://final.geothermaldata.org/dataset/alaska-thermal-springs9f940

dano-reisys commented 9 years ago

same problem i think... but the link is valid, i'll have to take a look at it.

ccaudill commented 9 years ago

The service works fine. http://services.azgs.az.gov/ArcGIS/services/aasggeothermal/AKThermalSprings1_8/MapServer/WMSServer?request=GetCapabilities&service=WMS

dano-reisys commented 9 years ago

Should be fixed in instance: http://uat-ngds.reisys.com/dataset/test-wms/resource/5fc4c88d-4a12-4407-bbdf-0c3a029caa45

it will require a code push

ccaudill commented 9 years ago

What fix are we looking at there @dan-olaru-reisys ?

dano-reisys commented 9 years ago

@ccaudill The WMS preview should work now, looking into why the geo-prospector button isn't showing up atm

dano-reisys commented 9 years ago

@ccaudill I was able to get the geo-prospector button to show up for the other datasets, but i am getting a WFS error on maps-stage.nrel.gov (looks like the WFS has no wfsFeatureTypeName... or am I missing something?)

here is the link to the dataset on uat-ngds: http://uat-ngds.reisys.com/dataset/alaska-thermal-springs2

the equivalent of: http://final.geothermaldata.org/dataset/alaska-thermal-springs9f940

On the other hand, the tire 3 dataset does provide wfsFeatureTypeName for WFS)

http://uat-ngds.reisys.com/dataset/colon-tag-test

ccaudill commented 9 years ago

Perhaps that layer for DEWellLogs you created does open in GeoProsp without error, but you still can't access the WFS features "Data for this layer is not available for download." and the WMS isn't showing on the map in GeosProsp either.

I think programmatically, you will have to add the wfsFeatureTypeName to the URL that NGDS CKAN is providing to NREL. It will not come like that in the harvested data for any resource. As indicated in the the first comment of this issue, the URL must be constructed to include "wfsFeatureTypeName: the name of the wfs feature type exactly as it needs to be specified in the TypeName WFS parameter".

smrgeoinfo commented 9 years ago

The WMS is working as it should; Since there's no parameters for the WFS in the prospector URL, it gives that alert message about can't query the data.

See list of parameters for prospector URL call, https://github.com/ngds/ckanext-ngds/issues/404#issue-29764334. need to include wfsHost and wfsFeatureTypeName. Need to see if there is a WFS and WMS distribution

-<gmd:MD_DigitalTransferOptions id="">
   <gmd:onLine>
      <gmd:CI_OnlineResource>   
          <!-- elements left out in this example code -->
            <gmd:description>
            <gco:CharacterString> parameters:{typeName:"ThermalSpring"}                            
             </gco:CharacterString>
           </gmd:description>

wfsHost: a string (URL escaped) containing the endpoint URL for a WFS that provides feature access to the data displayed by the WMS. The string argument is the part of the URL that precedes the WFS parameters, basically a WFS request with no parameters part (leave out the question mark). OPTIONAL

wfsFeatureTypeName: the name of the wfs feature type exactly as it needs to be specified in the TypeName WFS parameter; the namespace qualifier might or might not be required, I think that might depend on the service implementation. OPTIONAL

see bug fix at https://github.com/ngds/ckanext-ngds/commit/f269001f870e986f6d8a797266f2e899e54b8964

smrgeoinfo commented 9 years ago

NOTE https://github.com/ngds/metadata-issues/issues/47. Our metadata isn't following the profile correctly (the issues never end), so to detect if there is a WMS or WFS distribution, probably have to look for 'service=WMS' or 'service=WFS' in the gmd:linkage/gmd:URL:

<gmd:onLine>
   <gmd:CI_OnlineResource>
      <gmd:linkage>
         <gmd:URL>
http://services.azgs.az.gov/ArcGIS/services/aasggeothermal/AKThermalSprings1_8/MapServer/WMSServer?request=GetCapabilities&service=WMS
         </gmd:URL>
dano-reisys commented 9 years ago

@smrazgs So do we have two issues here?

  1. http://uat-ngds.reisys.com/dataset/colon-tag-test

Geothermal prospector button url: https://maps-stage.nrel.gov/geothermal-prospector/#/?baselayer=6&zoomlevel=3&wmsHost=http://uat-ngds.reisys.com/geoserver-srv/DEWellLog/wms?SERVICE=WMS&&wmsLayerName=WellLog&wfsHost=http://uat-ngds.reisys.com/geoserver-srv/DEWellLog/wfs&wfsFeatureTypeName=DEWellLog:WellLog

@ccaudill wrote:

Perhaps that layer for DEWellLogs you created does open in GeoProsp without error, but you still can't > access the WFS features "Data for this layer is not available for download." and the WMS isn't showing > on the map in GeosProsp either.

*2.

@smrazgs wrote:

NOTE ngds/metadata-issues#47. Our metadata isn't following the profile correctly (the issues never end), so to detect if there is a WMS or WFS distribution, probably have to look for 'service=WMS' or 'service=WFS' in the gmd:linkage/gmd:URL:

Are you saying that this dataset should not have a geothermal prospector button? Or perhaps we need to fix package_to_iso.xml so that the iso metadata is generated correctly (with a valid geothermal prospector button?)

ccaudill commented 9 years ago

Neither of those things. You just need to look here in the metadata for the wfsTypeName: - gmd:onLine gmd:CI_OnlineResource 

        gmd:description
        gco:CharacterString parameters:{typeName:"ThermalSpring"}

And make that wfsTypeName be added programmatically to the URL that CKAN NGDS provides to GeoProsp.

dano-reisys commented 9 years ago

@ccaudill what about issue "1." from above? I talked to @JihadMotii-REISys and it seems like this is the first time he has seen this issue. Did this work before? Seems like the geothermal prospector link is valid and the preview for WFS is fine on uat-ngds.

any ideas?

dano-reisys commented 9 years ago

@ccaudill @smrazgs

this appears to be the valid geothermal prospector link for the dataset at this url: http://uat-ngds.reisys.com/dataset/alaska-thermal-springs2

Geothermalprospector link: https://maps-stage.nrel.gov/geothermal-prospector/#/?baselayer=6&zoomlevel=3&wmsHost=http://services.azgs.az.gov/arcgis/services/aasggeothermal/AKThermalSprings1_8/MapServer/WmsServer?&wmsLayerName=ThermalSpring&wfsHost=http://services.azgs.az.gov/arcgis/services/aasggeothermal/AKThermalSprings1_8/MapServer/WmsServer&wfsFeatureTypeName=ThermalSpring

the wfs issue is still there...

ThermalSpring Data for this layer is not available for download.

ccaudill commented 9 years ago

From that link, the AK WMS is showing up exactly as it should in GeosProsp.

smrgeoinfo commented 9 years ago

prospector link seems to be working perfectly at this point. The data are actually for Arkansas, not Delaware. The Metadata for this record is correct, it includes the correct gmd:protocol values in the metadata. I don't know for sure that prospector is supposed to enable download of data from a WFS...

the metadata for the Alaska service http://uat-ngds.reisys.com/dataset/alaska-thermal-springs2 is incorrect (missing the gmd:protocol parameter), so I'm wondering why the prospector button comes up? see comment above (starts "NOTE ngds/metadata-issues#47.") for workaround for metadata problem.

ccaudill commented 9 years ago

FYI - No GeoProsp button in new install. Today this server was wiped and reinstalled. http://test.geothermaldata.org/dataset/vermont-thermal-conductivity geopro

dano-reisys commented 9 years ago

That should be the normal behavior. The fix is in the code, which is not in the v307 rpm

smrgeoinfo commented 9 years ago

can we get it in the rpm soon?

dano-reisys commented 9 years ago

@smrazgs We'll are planning on building a new rpm on 03/31 @FuhuXia

dano-reisys commented 9 years ago

@smrazgs @ccaudill Looks like the reason why the wfs part of the geothermal prospector link is not there is because harvested datasets like: http://test.geothermaldata.org/dataset/vermont-thermal-conductivity lack the parent_source filed in the wms resource. I am trying to modify the logic, so that it doesn't use the parent_source for such datasets.

questions:

From looking at the code it seems like there only one wms per dataset, but there might be multiple wfs resources because its trying to loop over a list of wfs resources

smrgeoinfo commented 9 years ago

@dan-olaru-reisys -- looks like the problem is that action.py geothermal_prospector_url is not finding the WFS connection information in the CKAN package resource. Lets try and focus the issues on specific problems, and close them when we resolve the problem. I'm closing this thread and starting a new more specific thread. If there are other unresolved issues in this thread, please start a new, specific thread.