Server Error message - metadata update

[stufraser1]

When saving updated metadata, and 'Internal Server Error' message appears, though the changes appear to have been saved. This occurred repeatedly when saving changes to Data Quality scores in metadata - no other use cases tested.

[fvanderbiest]

I got a stack trace, it's an SSL certificate issue during communication with geoserver:

01/08/2017 14:24:50SSLHandshakeError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)
01/08/2017 14:36:05Internal Server Error: /layers/hazard:global_reclassify/metadata
01/08/2017 14:36:05Traceback (most recent call last):
01/08/2017 14:36:05  File "/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 132, in get_response
01/08/2017 14:36:05    response = wrapped_callback(request, *callback_args, **callback_kwargs)
01/08/2017 14:36:05  File "/usr/local/lib/python2.7/site-packages/django/contrib/auth/decorators.py", line 22, in _wrapped_view
01/08/2017 14:36:05    return view_func(request, *args, **kwargs)
01/08/2017 14:36:05  File "/usr/src/app/geonode/layers/views.py", line 490, in layer_metadata
01/08/2017 14:36:05    the_layer = layer_form.save()
01/08/2017 14:36:05  File "/usr/local/lib/python2.7/site-packages/django/forms/models.py", line 459, in save
01/08/2017 14:36:05    construct=False)
01/08/2017 14:36:05  File "/usr/local/lib/python2.7/site-packages/django/forms/models.py", line 105, in save_instance
01/08/2017 14:36:05    instance.save()
01/08/2017 14:36:05  File "/usr/local/lib/python2.7/site-packages/polymorphic/models.py", line 90, in save
01/08/2017 14:36:05    return super(PolymorphicModel, self).save(*args, **kwargs)
01/08/2017 14:36:05  File "/usr/local/lib/python2.7/site-packages/django/db/models/base.py", line 734, in save
01/08/2017 14:36:05    force_update=force_update, update_fields=update_fields)
01/08/2017 14:36:05  File "/usr/local/lib/python2.7/site-packages/django/db/models/base.py", line 771, in save_base
01/08/2017 14:36:05    update_fields=update_fields, raw=raw, using=using)
01/08/2017 14:36:05  File "/usr/local/lib/python2.7/site-packages/django/dispatch/dispatcher.py", line 189, in send
01/08/2017 14:36:05    response = receiver(signal=self, sender=sender, **named)
01/08/2017 14:36:05  File "/usr/src/app/geonode/geoserver/signals.py", line 459, in geoserver_post_save
01/08/2017 14:36:05    set_styles(instance, gs_catalog)
01/08/2017 14:36:05  File "/usr/src/app/geonode/geoserver/helpers.py", line 749, in set_styles
01/08/2017 14:36:05    default_style = gs_layer.default_style
01/08/2017 14:36:05  File "/usr/local/lib/python2.7/site-packages/geoserver/layer.py", line 127, in _get_default_style
01/08/2017 14:36:05    return self._resolve_style(element) if element is not None else None
01/08/2017 14:36:05  File "/usr/local/lib/python2.7/site-packages/geoserver/layer.py", line 135, in _resolve_style
01/08/2017 14:36:05    return self.catalog.get_style_by_url(style_workspace_url)
01/08/2017 14:36:05  File "/usr/local/lib/python2.7/site-packages/geoserver/catalog.py", line 797, in get_style_by_url
01/08/2017 14:36:05    dom = self.get_xml(style_workspace_url)
01/08/2017 14:36:05  File "/usr/local/lib/python2.7/site-packages/geoserver/catalog.py", line 211, in get_xml
01/08/2017 14:36:05    response, content = self.http.request(rest_url)
01/08/2017 14:36:05  File "/usr/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1609, in request
01/08/2017 14:36:05    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
01/08/2017 14:36:05  File "/usr/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1351, in _request
01/08/2017 14:36:05    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
01/08/2017 14:36:05  File "/usr/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1272, in _conn_request
01/08/2017 14:36:05    conn.connect()
01/08/2017 14:36:05  File "/usr/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1059, in connect
01/08/2017 14:36:05    raise SSLHandshakeError(e)

[fvanderbiest]

Also ...

02/08/2017 08:47:33 Invalid HTTP_HOST header: ''. The domain name provided is not valid according to RFC 1034/1035.
02/08/2017 09:21:49 Internal Server Error: /layers/hazard:global_reclassify/metadata

[fvanderbiest]

As stated in the ssl doc I've added the site certificate to httplib2's cacerts.txt:

root@0e9cc17d341c:/usr/src/app# cat /tmp/cert.txt >> /usr/local/lib/python2.7/site-packages/httplib2/cacerts.txt

but this did not help.

Still trying to figure out how to fix this.

[fvanderbiest]

My guess is that something changed with the certificate generation, and httplib2 0.9.2 (which is geonode's default version) is not able to cope with that anymore.

Here's what happens on the server with httplib2 0.9.2:

root@0e9cc17d341c:/usr/src/app# python
Python 2.7.13 (default, Jul 24 2017, 20:13:18) 
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import httplib2
>>> http = httplib2.Http(cache=None, timeout=30)
>>> response, content = http.request('https://int.geonode-gfdrrlab.org/geoserver/rest/styles/global_reclassify.xml', 'GET')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1609, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/usr/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1351, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/usr/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1272, in _conn_request
    conn.connect()
  File "/usr/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1059, in connect
    raise SSLHandshakeError(e)
httplib2.SSLHandshakeError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)

While on my computer with latest httplib2 release (0.10.3):

(env) fvanderbiest@granier:/tmp/env$ python
Python 2.7.12 (default, Nov 19 2016, 06:48:10) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import httplib2
>>> http = httplib2.Http(cache=None, timeout=30)
>>> response, content = http.request('https://int.geonode-gfdrrlab.org/geoserver/rest/styles/global_reclassify.xml', 'GET')
>>> print content
<style>
  <name>global_reclassify</name>
  <format>sld</format>
  <languageVersion>
    <version>1.0.0</version>
  </languageVersion>
  <filename>global_reclassify.sld</filename>
</style>

[fvanderbiest]

Fixed with https://github.com/camptocamp/labs_geonode/pull/8, deployed to int. Will push this to prod ASAP with https://github.com/camptocamp/terraform-geonode/commit/3cfad2bf01c2e8d0bcff52a1922619ce71c7b1b1

[fvanderbiest]

This is now fixed.

[fvanderbiest]

Reported upstream with https://github.com/GeoNode/geonode/issues/3194