search

ngds / ckanext-ngds-bku03232018

CKAN extension for NGDS-specific customizations
Other
5 stars 13 forks source link

Geoserver proxy responsible for fetching WMS URL does not work #41

Closed viveksunder closed 10 years ago

viveksunder commented 11 years ago

Adding a WMS to the map does not work. Looking at the server console gives us this beautiful stack trace.

Error - <type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'strip' URL: http://localhost:5000/api/action/geoserver_proxy_layer_name File '/Users/kaffeine/Projects/ngds/new_pyenv/lib/python2.7/site-packages/weberror/errormiddleware.py', line 162 in call app_iter = self.application(environ, sr_checker) File '/Users/kaffeine/Projects/ngds/new_pyenv/lib/python2.7/site-packages/webob/dec.py', line 147 in call resp = self.call_func(req, _args, _self.kwargs) File '/Users/kaffeine/Projects/ngds/new_pyenv/lib/python2.7/site-packages/webob/dec.py', line 208 in call_func return self.func(req, _args, _kwargs) File '/Users/kaffeine/Projects/ngds/new_pyenv/lib/python2.7/site-packages/fanstatic/publisher.py', line 234 in call return request.get_response(self.app) File '/Users/kaffeine/Projects/ngds/new_pyenv/lib/python2.7/site-packages/webob/request.py', line 1053 in get_response application, catch_exc_info=False) File '/Users/kaffeine/Projects/ngds/new_pyenv/lib/python2.7/site-packages/webob/request.py', line 1022 in call_application app_iter = application(self.environ, start_response) File '/Users/kaffeine/Projects/ngds/new_pyenv/lib/python2.7/site-packages/webob/dec.py', line 147 in call resp = self.call_func(req, _args, _self.kwargs) File '/Users/kaffeine/Projects/ngds/new_pyenv/lib/python2.7/site-packages/webob/dec.py', line 208 in call_func return self.func(req, _args, _kwargs) File '/Users/kaffeine/Projects/ngds/new_pyenv/lib/python2.7/site-packages/fanstatic/injector.py', line 54 in call response = request.get_response(self.app) File '/Users/kaffeine/Projects/ngds/new_pyenv/lib/python2.7/site-packages/webob/request.py', line 1053 in get_response application, catch_exc_info=False) File '/Users/kaffeine/Projects/ngds/new_pyenv/lib/python2.7/site-packages/webob/request.py', line 1022 in call_application app_iter = application(self.environ, start_response) File '/Users/kaffeine/Projects/ngds/new_pyenv/lib/python2.7/site-packages/beaker/middleware.py', line 73 in call return self.app(environ, start_response) File '/Users/kaffeine/Projects/ngds/new_pyenv/lib/python2.7/site-packages/beaker/middleware.py', line 155 in call return self.wrap_app(environ, session_start_response) File '/Users/kaffeine/Projects/ngds/new_pyenv/lib/python2.7/site-packages/routes/middleware.py', line 131 in call response = self.app(environ, start_response) File '/Users/kaffeine/Projects/ngds/new_pyenv/lib/python2.7/site-packages/pylons/wsgiapp.py', line 125 in call response = self.dispatch(controller, environ, start_response) File '/Users/kaffeine/Projects/ngds/new_pyenv/lib/python2.7/site-packages/pylons/wsgiapp.py', line 324 in dispatch return controller(environ, start_response) File '/Users/kaffeine/Projects/ngds/new_pyenv/src/ckan/ckan/controllers/api.py', line 70 in call return base.BaseController.call(self, environ, start_response) File '/Users/kaffeine/Projects/ngds/new_pyenv/src/ckan/ckan/lib/base.py', line 289 in call res = WSGIController.call(self, environ, start_response) File '/Users/kaffeine/Projects/ngds/new_pyenv/lib/python2.7/site-packages/pylons/controllers/core.py', line 221 in call response = self._dispatch_call() File '/Users/kaffeine/Projects/ngds/new_pyenv/lib/python2.7/site-packages/pylons/controllers/core.py', line 172 in _dispatch_call response = self._inspect_call(func) File '/Users/kaffeine/Projects/ngds/new_pyenv/lib/python2.7/site-packages/pylons/controllers/core.py', line 107 in _inspect_call result = self._perform_call(func, args) File '/Users/kaffeine/Projects/ngds/new_pyenv/lib/python2.7/site-packages/pylons/controllers/core.py', line 60 in _perform_call return func(args) File '/Users/kaffeine/Projects/ngds/new_pyenv/src/ckan/ckan/controllers/api.py', line 184 in action result = function(context, request_data) File '/Users/kaffeine/Projects/ngds/new_pyenv/src/ckan/ckan/logic/init.py', line 324 in wrapped return _action(context, data_dict, kw) File '/Users/kaffeine/Projects/ngds/new_pyenv/src/ckanext-ngds/ckanext/ngds/geoserver/logic/action.py', line 121 in GETLayerNameWMS thisWMS = WebMapService(thisURL, version) File '/Users/kaffeine/Projects/ngds/new_pyenv/lib/python2.7/site-packages/owslib/wms.py', line 86 in init self._buildMetadata(parse_remote_metadata) File '/Users/kaffeine/Projects/ngds/new_pyenv/lib/python2.7/site-packages/owslib/wms.py', line 126 in _buildMetadata gather_layers(caps, None) File '/Users/kaffeine/Projects/ngds/new_pyenv/lib/python2.7/site-packages/owslib/wms.py', line 120 in gather_layers cm = ContentMetadata(elem, parent=parent_metadata, index=index+1, parse_remote_metadata=parse_remote_metadata) File '/Users/kaffeine/Projects/ngds/new_pyenv/lib/python2.7/site-packages/owslib/wms.py', line 326 in init self.title = testXMLValue(elem.find('Title')).strip() AttributeError: 'NoneType' object has no attribute 'strip'

rclark commented 11 years ago

The proxy is responsible for finding WMS layer names -- it ought not to be handling tile requests.

viveksunder commented 11 years ago

Correction : It does not handle tile requests. It also needs an upgrade of owslib to work. ckanext-spatial now installs a later version of owslib, but this installation is a bit broken because of difficulties building updates to lxml. The way to do this is -

pip install owslib --upgrade STATIC_DEPS=true pip install lxml --upgrade

Its should then work.

viveksunder commented 11 years ago

This does not work on the amazon machine. The WMS getCapabilities request produces invalid XML because of an exception that is generated somewhere at the end of the document. I believe that once this issue has been addressed and we get it to generate valid XML, adding a WMS to the map should work just fine.

The output looks like this -

EPSG:61306413 EPSG:61316405 EPSG:61326405 EPSG:61336405 javax.xml.transform.TransformerException: java.lang.NullPointerException: Unable to transform referenced envelope, crs has not yet been provided. java.lang.NullPointerException: Unable to transform referenced envelope, crs has not yet been provided. Unable to transform referenced envelope, crs has not yet been provided.
rclark commented 11 years ago

Is that a WMS built by our system or externally? That's a server error - we just have to fail gracefully. 

Ryan

On Sun, Nov 3, 2013 at 11:13 AM, viveksunder notifications@github.com wrote:

This does not work on the amazon machine. The WMS getCapabilities request produces invalid XML because of an exception that is generated somewhere at the end of the document. I believe that once this issue has been addressed and we get it to generate valid XML, adding a WMS to the map should work just fine. The output looks like this -

EPSG:61306413 EPSG:61316405 EPSG:61326405 EPSG:61336405 javax.xml.transform.TransformerException: java.lang.NullPointerException: Unable to transform referenced envelope, crs has not yet been provided. java.lang.NullPointerException: Unable to transform referenced envelope, crs has not yet been provided. Unable to transform referenced envelope, crs has not yet been provided. ## Reply to this email directly or view it on GitHub: https://github.com/ngds/ckanext-ngds/issues/41#issuecomment-27650189
viveksunder commented 11 years ago

This is XML generated by geoserver (http://:8080/geoserver/wms?request=GetCapabilities), when we make a 'getCapabilities' request to it. Our code does not directly handle the output of this request, owslib's WebMapService constructor does. I think we have to figure out why a call to getCapabilities produces invalid XML.

rclark commented 11 years ago

I think that happens because there's a layer in that Geoserver that has not been defined properly. It's a Geoserver internal error - you could look at geoservers logs.

On Sun, Nov 3, 2013 at 2:26 PM, viveksunder notifications@github.com wrote:

This is XML generated by geoserver (http://:8080/geoserver/wms?request=GetCapabilities), when we make a 'getCapabilities' request to it. Our code does not directly handle the output of this request, owslib's WebMapService constructor does. I think we have to figure out why a call to getCapabilities produces invalid XML.

Reply to this email directly or view it on GitHub: https://github.com/ngds/ckanext-ngds/issues/41#issuecomment-27654834

smrgeoinfo commented 11 years ago

I’ve seen that error before, and I think it was when we were testing the SMU geoserver. Maybe Raj has some idea where it comes from…

Stephen M Richard Arizona Geological Survey 416 W. Congress #100 Tucson, AZ AZGS: 520-770-3500 Office: 520-209-4127 FAX: 520-770-3505

From: Ryan Clark [mailto:notifications@github.com] Sent: Sunday, November 03, 2013 2:30 PM To: ngds/ckanext-ngds Subject: Re: [ckanext-ngds] Geoserver proxy responsible for fetching WMS tiles does not work (#41)

I think that happens because there's a layer in that Geoserver that has not been defined properly. It's a Geoserver internal error - you could look at geoservers logs.

On Sun, Nov 3, 2013 at 2:26 PM, viveksunder notifications@github.com wrote:

This is XML generated by geoserver (http://:8080/geoserver/wms?request=GetCapabilities), when we make a 'getCapabilities' request to it. Our code does not directly handle the output of this request, owslib's WebMapService constructor does. I think we have to figure out why a call to getCapabilities produces invalid XML.

Reply to this email directly or view it on GitHub: https://github.com/ngds/ckanext-ngds/issues/41#issuecomment-27654834

— Reply to this email directly or view it on GitHubhttps://github.com/ngds/ckanext-ngds/issues/41#issuecomment-27654934.

viveksunder commented 11 years ago

Thanks @smrAzGS . I'll check with Raj.

viveksunder commented 11 years ago

I had the same issue with my local geoserver and I deleted all the published layers on it except for just one which I just published and was able to preview just fine. That seemed to remove the offending entry. I'll probably do the same exercise on the amazon machine's geoserver.

viveksunder commented 11 years ago

On the amazon machine this wasn't as straightforward. I deleted all the layers in the ngds workspace, and still had the same problem. I then deleted all the other workspaces on geoserver, except for ngds. That resolved it.

Any ideas on what our strategy should be to deal with this issue? It seems to me that one offending layer can cause this set of features to not work. @rclark @asonnenschein

rclark commented 11 years ago

The only things that we can do;

On Mon, Nov 4, 2013 at 10:34 AM, viveksunder notifications@github.com wrote:

On the amazon machine this wasn't as straightforward. I deleted all the layers in the ngds workspace, and still had the same problem. I then deleted all the other workspaces on geoserver, except for ngds. That resolved it.

Any ideas on what our strategy should be to deal with this issue? It seems to me that one offending layer can cause this set of features to not work. @rclark @asonnenschein

Reply to this email directly or view it on GitHub: https://github.com/ngds/ckanext-ngds/issues/41#issuecomment-27705027

viveksunder commented 11 years ago

Point 2 is unfortunately not in our hands. owslib does this bit.

rclark commented 11 years ago

Then we have to handle the owslib exception without the user ending up with a 500 or any unexpected result. It won't work, but it has to fail gracefully.

viveksunder commented 11 years ago

I like @rwschwanke's idea in https://github.com/ngds/ckanext-ngds/issues/53. Since this is something that needs active action by an admin, it is perhaps a good idea to include it in such a 'liveness monitor'.

asonnenschein commented 11 years ago

@rclark This is the same error @smrAzGS reported during GRC. Have we figured out the root of this problem yet? Seems as though having one invalid layer in Geoserver can destroy usability of the entire program. If this is the case then I don't think failing gracefully is an option. My solution would be that we figure out what is actually causing this problem and then write some functions which handle the exception. Otherwise, the only way I've ever gotten around this problem is by reinstalling Geoserver (which I don't think we want to tell people they have to do).

rclark commented 11 years ago

We can't fix Geoserver. Way out of scope. We can report bugs.

When I say "fail gracefully" I do not mean "make it work as intended". I mean that this is an example of a failure our software must deal with without throwing out users an indecipherable exception. 

If the WMS has an invalid capabilities document, then out software cannot view that data. A message should indicate to the user who tried to view the WMS that it is bunk, broken, and we can't do anything about it.

On Mon, Nov 4, 2013 at 10:33 PM, Adrian Sonnenschein notifications@github.com wrote:

@rclark This is the same error @smrAzGS reported during GRC. Have we figured out the root of this problem yet? Seems as though having one invalid layer in Geoserver can destroy usability of the entire program. If this is the case then I don't think failing gracefully is an option. My solution would be that we figure out what is actually causing this problem and then write some functions which handle the exception. Otherwise, the only way I've ever gotten around this problem is by reinstalling Geoserver (which I don't think we want to tell people they have to do).

Reply to this email directly or view it on GitHub: https://github.com/ngds/ckanext-ngds/issues/41#issuecomment-27748859

viveksunder commented 11 years ago

I'd suggest incorporating what @rclark is suggesting into @rwschwanke's idea of a liveness monitor. I'm thinking that the backend checks the output of this getCapabilities request, and if it is malformed, puts an alert out to the UI (to every user that is navigating the site, or perhaps just the map page) on the top of the screen - ckan alert style something to the effect of - "We are having problems. Contact the site administrator at - blah@blah.com".

rclark commented 11 years ago

KISS. If the user tries to add a WFS/WMS to the map and the function can't parse the capabilities, notify the user then, perhaps flag the resource in its metadata as down and use that to inform future UI decisions. I'm not into writing another sub system to continually monitor the up/down-ness of all services linked in the system. Not in scope.

On Tue, Nov 5, 2013 at 7:48 AM, viveksunder notifications@github.comwrote:

I'd suggest incorporating what @rclark https://github.com/rclark is suggesting into @rwschwanke https://github.com/rwschwanke's idea of a liveness monitor. I'm thinking that the backend checks the output of this getCapabilities request, and if it is malformed, puts an alert out to the UI (to every user that is navigating the site, or perhaps just the map page) on the top of the screen - ckan alert style something to the effect of - "We are having problems. Contact the site administrator at - blah@blah.com".

— Reply to this email directly or view it on GitHubhttps://github.com/ngds/ckanext-ngds/issues/41#issuecomment-27777949 .

viveksunder commented 11 years ago

I'm not suggesting we write a subsystem either. I think we can simply check if the getCapabilities request is garbled, and put an alert out. I was simply referring to the idea of monitoring things like this that might potentially malfunction without any indication to the user that something is wrong (aside from the 404).

rclark commented 11 years ago

How do we implement monitoring if it is not another dedicated sub-system? I think we're all pretty much talking about the same thing: if a capabilities request is garbled, give the user something other than an HTTP error code.

On Tue, Nov 5, 2013 at 9:00 AM, viveksunder notifications@github.comwrote:

I'm not suggesting we write a subsystem either. I think we can simply check if the getCapabilities request is garbled, and put an alert out. I was simply referring to the idea of monitoring things like this that might potentially malfunction without any indication to the user that something is wrong (aside from the 404).

— Reply to this email directly or view it on GitHubhttps://github.com/ngds/ckanext-ngds/issues/41#issuecomment-27785122 .

viveksunder commented 11 years ago

It also takes far too long to get back to the user if something went wrong. I'm counting over a minute here.

rclark commented 11 years ago

Is it an HTTP timeout you're waiting for? Can you tweak an owslib setting perhaps?

On Wed, Nov 6, 2013 at 2:30 PM, viveksunder notifications@github.comwrote:

It also takes far too long to get back to the user if something went wrong. I'm counting over a minute here.

— Reply to this email directly or view it on GitHubhttps://github.com/ngds/ckanext-ngds/issues/41#issuecomment-27915335 .

viveksunder commented 11 years ago

Perhaps. I'll look into that. Hopefully there's something in there that lets us set a time out value or something.

rwschwanke commented 11 years ago

@rclark are we just waiting for this fix to propagate, or does something more need to be done?

rclark commented 11 years ago

@rwschwanke are we still seeing symptoms?

asonnenschein commented 11 years ago

Is this something I should be addressing with the OGC service code I've been working on?

rwschwanke commented 11 years ago

I am not yet able to observe anything other than user interfaces. I would suggest you do nothing on this until Vivek's changes merge. Then this will come up as a regression test item, if we don't see other symptoms.

rwschwanke commented 10 years ago

@asonnenschein Please create a test case that I can run from the user interface. I don't know the system well enough yet to figure it out.

asonnenschein commented 10 years ago

@rwschwanke Here's a solid WMS service: http://services.azgs.az.gov/ArcGIS/services/aasggeothermal/TXActiveFaults/MapServer/WMSServer

Steps to test from UI:

Contribute > Individual Upload > Data-service: Enter that URL, specify WMS for the type and for layer name you can either enter 'ActiveFault' or you can enter a random string of characters (but you cannot leave it blank). Entering 'ActiveFault' will ensure that the layer 'ActiveFault' is selected from the WMS service, entering a false layer name will test if the failure function works. The failure function gets triggered if there is no user-submitted layer name (like on harvest) or if the user-submitted layer name is not found in the getCapabilities document and so it just returns the first layer in the list of layers. Regardless, for this specific service, there is only one layer so no matter what it will return the ActiveFault layer.

Additional Metadata > Dataset Details Page > Preview: The WMS layer should show up on the map, with the map automatically zoomed to a 'best fit' zoom level for the bounds of the dataset.

Closing this issue.