Closed fvanderbiest closed 6 years ago
Geoserver logs on 02/23/2018
r-geonode-geoserver-geoserver-1-3e820cf1-8020c895a895.log.gz
r-geonode-geoserver-geoserver-1-9d486781-61ececac0414.log.gz
Reverting this change camptocamp/gfdrr-geonode-geoserver-datadir@2a5ad03
No effect on geoserver stability.
Several errors in the logs regarding unavailable layers:
2018-02-27T09:40:21.075456+00:00: 27 Feb 09:40:21 ERROR [geoserver.ows] -
2018-02-27T09:40:21.075799+00:00: org.geoserver.platform.ServiceException: Could not find layer hazard:pga2475v2
2018-02-27T09:40:21.076118+00:00: at org.geoserver.wms.map.GetMapKvpRequestReader.parseLayers(GetMapKvpRequestReader.java:1341)
2018-02-27T09:40:21.076433+00:00: at org.geoserver.wms.map.GetMapKvpRequestReader.read(GetMapKvpRequestReader.java:240)
--
2018-02-27T09:40:21.094813+00:00: 27 Feb 09:40:21 ERROR [geoserver.ows] -
2018-02-27T09:40:21.094979+00:00: org.geoserver.platform.ServiceException: Could not find layer hazard:pga475v2
2018-02-27T09:40:21.095144+00:00: at org.geoserver.wms.map.GetMapKvpRequestReader.parseLayers(GetMapKvpRequestReader.java:1341)
2018-02-27T09:40:21.095299+00:00: at org.geoserver.wms.map.GetMapKvpRequestReader.read(GetMapKvpRequestReader.java:240)
--
2018-02-27T09:40:21.136029+00:00: 27 Feb 09:40:21 ERROR [geoserver.ows] -
2018-02-27T09:40:21.136310+00:00: org.geoserver.platform.ServiceException: Could not find layer hazard:pga975v2
2018-02-27T09:40:21.136580+00:00: at org.geoserver.wms.map.GetMapKvpRequestReader.parseLayers(GetMapKvpRequestReader.java:1341)
2018-02-27T09:40:21.136843+00:00: at org.geoserver.wms.map.GetMapKvpRequestReader.read(GetMapKvpRequestReader.java:240)
--
2018-02-27T09:40:21.359109+00:00: 27 Feb 09:40:21 ERROR [geoserver.ows] -
2018-02-27T09:40:21.359533+00:00: org.geoserver.platform.ServiceException: Could not find layer hazard:pga100v2
2018-02-27T09:40:21.359940+00:00: at org.geoserver.wms.map.GetMapKvpRequestReader.parseLayers(GetMapKvpRequestReader.java:1341)
2018-02-27T09:40:21.360303+00:00: at org.geoserver.wms.map.GetMapKvpRequestReader.read(GetMapKvpRequestReader.java:240)
...
2018-02-27T10:31:52.802630+00:00: Caused by: org.geoserver.platform.ServiceException: hazard:adm2_fu_v3_4bit layer does not exist.
2018-02-27T06:25:15.010196+00:00: Caused by: org.geoserver.platform.ServiceException: hazard:undefined layer does not exist.
2018-02-27T08:05:05.162353+00:00: Caused by: org.geoserver.platform.ServiceException: hazard:adm2_fu_v3_4bit layer does not exist.
Also "Error connecting to the GeoNode server for authentication purposes":
2018-02-27T09:11:56.128518+00:00: 27 Feb 09:11:56 WARN [geonode.security] - Error connecting to the GeoNode server for authentication purposes
2018-02-27T09:11:56.128936+00:00: org.springframework.security.authentication.AuthenticationServiceException: Communication with GeoNode failed (UsernamePasswordAuthenticationToken)
2018-02-27T09:11:56.129161+00:00: at org.geonode.security.GeoNodeAuthenticationProvider.authenticate(GeoNodeAuthenticationProvider.java:62)
2018-02-27T09:11:56.129359+00:00: at org.geoserver.security.GeoServerAuthenticationProvider.authenticate(GeoServerAuthenticationProvider.java:58)
2018-02-27T09:11:56.129551+00:00: at org.springframework.security.authentication.ProviderManager.authenticate(ProviderManager.java:167)
2018-02-27T09:11:56.129753+00:00: at org.geoserver.security.GeoServerSecurityManager$1.authenticate(GeoServerSecurityManager.java:323)
2018-02-27T09:11:56.129936+00:00: at org.geonode.security.GeoNodeAnonymousProcessingFilter.doFilter(GeoNodeAnonymousProcessingFilter.java:60)
2018-02-27T09:11:56.130114+00:00: at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
2018-02-27T09:11:56.130321+00:00: at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:69)
--
2018-02-27T09:11:56.139811+00:00: Caused by: java.net.ConnectException: Connection refused (Connection refused)
2018-02-27T09:11:56.139973+00:00: at java.net.PlainSocketImpl.socketConnect(Native Method)
2018-02-27T09:11:56.140136+00:00: at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
2018-02-27T09:11:56.140297+00:00: at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
2018-02-27T09:11:56.140456+00:00: at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
2018-02-27T09:11:56.140626+00:00: at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
2018-02-27T09:11:56.140810+00:00: at java.net.Socket.connect(Socket.java:589)
2018-02-27T09:11:56.140985+00:00: at sun.reflect.GeneratedMethodAccessor250.invoke(Unknown Source)
2018-02-27T09:11:56.141158+00:00: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2018-02-27T09:11:56.141328+00:00: at java.lang.reflect.Method.invoke(Method.java:498)
2018-02-27T09:11:56.141502+00:00: at org.apache.commons.httpclient.protocol.ReflectionSocketFactory.createSocket(ReflectionSocketFactory.java:140)
2018-02-27T09:11:56.141672+00:00: at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:125)
2018-02-27T09:11:56.141843+00:00: at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
2018-02-27T09:11:56.142015+00:00: at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
2018-02-27T09:11:56.142183+00:00: at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
2018-02-27T09:11:56.142404+00:00: at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
2018-02-27T09:11:56.142590+00:00: at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
2018-02-27T09:11:56.142775+00:00: at org.geonode.security.HTTPClient.sendGET(HTTPClient.java:89)
2018-02-27T09:11:56.142964+00:00: at org.geonode.security.DefaultSecurityClient.authenticate(DefaultSecurityClient.java:185)
2018-02-27T09:11:56.143148+00:00: at org.geonode.security.DefaultSecurityClient.authenticateAnonymous(DefaultSecurityClient.java:161)
2018-02-27T09:11:56.143330+00:00: at org.geonode.security.GeoNodeAuthenticationProvider.authenticate(GeoNodeAuthenticationProvider.java:49)
2018-02-27T09:11:56.143506+00:00: ... 60 more
sftp@ADMIN-gfdrr-prod-app:/mnt/geoserver_datadir/security/auth/geonodeAuthProvider$ cat config.xml
<org.geonode.security.GeoNodeAuthProviderConfig>
<id>52857278:13c7ffd66a8:-7fe9</id>
<name>geonodeAuthProvider</name>
<className>org.geonode.security.GeoNodeAuthenticationProvider</className>
<baseUrl>http://django:8000/</baseUrl>
</org.geonode.security.GeoNodeAuthProviderConfig>
https://github.com/camptocamp/terraform-geonode/blob/f3374326e3f4db02977ccc057a95e9e59b643b47/rancher-environments/gfdrr-geonode-prod/stacks.tf#L74 correctly maps geonode/django
service onto django
hostname and geoserver container sees django app:
root@24f2f4717255:/usr/local/tomcat# ping django
PING django.rancher.internal (10.42.183.160): 56 data bytes
64 bytes from 10.42.183.160: icmp_seq=0 ttl=64 time=0.074 ms
64 bytes from 10.42.183.160: icmp_seq=1 ttl=64 time=0.065 ms
2018-02-23T22:06:40.344920+00:00: 23 Feb 22:06:40 WARN [yaml.CustomBeanWrapper] - MetaData: unknown field 'footer' with value '{items=[org.mapfish.print.config.layout.ColumnsBlock@1ca175af], height=30}'
2018-02-23T22:06:40.345102+00:00: 23 Feb 22:06:40 WARN [yaml.CustomBeanWrapper] - MainPage: Error while writing '{items=[org.mapfish.print.config.layout.ColumnsBlock@1ca175af], height=30}' to footer: java.lang.IllegalArgumentException: argument type mismatch
2018-02-23T22:06:43.677652+00:00: 23 Feb 22:06:43 ERROR [geowebcache.GeoWebCacheDispatcher] - For input string: "{z}" http://www.geonode-gfdrrlab.org/geoserver/gwc/service/gmaps
2018-02-23T22:06:43.678014+00:00: java.lang.NumberFormatException: For input string: "{z}"
2018-02-23T22:06:43.678387+00:00: at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
2018-02-23T22:06:43.678749+00:00: at java.lang.Integer.parseInt(Integer.java:580)
2018-02-23T22:06:43.679103+00:00: at java.lang.Integer.parseInt(Integer.java:615)
2018-02-23T22:06:43.679507+00:00: at org.geowebcache.service.gmaps.GMapsConverter.getConveyor(GMapsConverter.java:77)
2018-02-23T10:14:50.656802+00:00: 23 Feb 10:14:50 ERROR [geoserver.rest] - No such layer group eq-data_1380
2018-02-23T10:14:50.686856+00:00: 23 Feb 10:14:50 ERROR [geoserver.rest] -
2018-02-23T10:14:50.687135+00:00: org.geoserver.rest.RestletException
2018-02-23T10:14:50.687394+00:00: at org.geoserver.catalog.rest.LayerGroupFinder.findTarget(LayerGroupFinder.java:49)
2018-02-23T10:14:50.687645+00:00: at org.restlet.Finder.handle(Finder.java:268)
2018-02-23T09:10:22.705298+00:00: 23 Feb 09:10:22 ERROR [geoserver.ows] -
2018-02-23T09:10:22.705725+00:00: org.geoserver.platform.ServiceException: No such operation wms 1.3.0 GetCapabilities?access_token=607e95d8187911e8848802a677f5a383
2018-02-23T09:10:22.706028+00:00: at org.geoserver.ows.Dispatcher.dispatch(Dispatcher.java:639)
2018-02-23T09:10:22.706348+00:00: at org.geoserver.ows.Dispatcher.handleRequestInternal(Dispatcher.java:258)
2018-02-23T09:10:22.706639+00:00: at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:147)
2018-02-23T09:10:22.706905+00:00: at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:50)
The front LB was the culprit for some weird reason. It should now be fixed.
planning a LB upgrade with https://github.com/camptocamp/terraform-geonode/pull/19/files
We're trying to understand what happened that day.