search

GFDRR / geonode

GFDRR Lab GeoNode
https://www.geonode-gfdrrlab.org
GNU General Public License v3.0
2 stars 3 forks source link

GeoNode GeoServer service flickering since 02/23/2018 9am #77

Closed fvanderbiest closed 6 years ago

fvanderbiest commented 6 years ago

capture du 2018-02-27 09-43-35 We're trying to understand what happened that day.

fvanderbiest commented 6 years ago

Geoserver logs on 02/23/2018

r-geonode-geoserver-geoserver-1-3e820cf1-8020c895a895.log.gz

r-geonode-geoserver-geoserver-1-9d486781-61ececac0414.log.gz

fvanderbiest commented 6 years ago

Reverting this change https://github.com/camptocamp/gfdrr-geonode-geoserver-datadir/commit/2a5ad03708d5ec53e4a82225591b38794f03a734

fvanderbiest commented 6 years ago

Reverting this change camptocamp/gfdrr-geonode-geoserver-datadir@2a5ad03

No effect on geoserver stability.

fvanderbiest commented 6 years ago

Several errors in the logs regarding unavailable layers:

2018-02-27T09:40:21.075456+00:00: 27 Feb 09:40:21 ERROR [geoserver.ows] -  
2018-02-27T09:40:21.075799+00:00: org.geoserver.platform.ServiceException: Could not find layer hazard:pga2475v2 
2018-02-27T09:40:21.076118+00:00:   at org.geoserver.wms.map.GetMapKvpRequestReader.parseLayers(GetMapKvpRequestReader.java:1341) 
2018-02-27T09:40:21.076433+00:00:   at org.geoserver.wms.map.GetMapKvpRequestReader.read(GetMapKvpRequestReader.java:240) 
--
2018-02-27T09:40:21.094813+00:00: 27 Feb 09:40:21 ERROR [geoserver.ows] -  
2018-02-27T09:40:21.094979+00:00: org.geoserver.platform.ServiceException: Could not find layer hazard:pga475v2 
2018-02-27T09:40:21.095144+00:00:   at org.geoserver.wms.map.GetMapKvpRequestReader.parseLayers(GetMapKvpRequestReader.java:1341) 
2018-02-27T09:40:21.095299+00:00:   at org.geoserver.wms.map.GetMapKvpRequestReader.read(GetMapKvpRequestReader.java:240) 
--
2018-02-27T09:40:21.136029+00:00: 27 Feb 09:40:21 ERROR [geoserver.ows] -  
2018-02-27T09:40:21.136310+00:00: org.geoserver.platform.ServiceException: Could not find layer hazard:pga975v2 
2018-02-27T09:40:21.136580+00:00:   at org.geoserver.wms.map.GetMapKvpRequestReader.parseLayers(GetMapKvpRequestReader.java:1341) 
2018-02-27T09:40:21.136843+00:00:   at org.geoserver.wms.map.GetMapKvpRequestReader.read(GetMapKvpRequestReader.java:240) 
--
2018-02-27T09:40:21.359109+00:00: 27 Feb 09:40:21 ERROR [geoserver.ows] -  
2018-02-27T09:40:21.359533+00:00: org.geoserver.platform.ServiceException: Could not find layer hazard:pga100v2 
2018-02-27T09:40:21.359940+00:00:   at org.geoserver.wms.map.GetMapKvpRequestReader.parseLayers(GetMapKvpRequestReader.java:1341) 
2018-02-27T09:40:21.360303+00:00:   at org.geoserver.wms.map.GetMapKvpRequestReader.read(GetMapKvpRequestReader.java:240) 

...
2018-02-27T10:31:52.802630+00:00: Caused by: org.geoserver.platform.ServiceException: hazard:adm2_fu_v3_4bit layer does not exist. 

2018-02-27T06:25:15.010196+00:00: Caused by: org.geoserver.platform.ServiceException: hazard:undefined layer does not exist. 

2018-02-27T08:05:05.162353+00:00: Caused by: org.geoserver.platform.ServiceException: hazard:adm2_fu_v3_4bit layer does not exist.
fvanderbiest commented 6 years ago

Also "Error connecting to the GeoNode server for authentication purposes":

2018-02-27T09:11:56.128518+00:00: 27 Feb 09:11:56 WARN [geonode.security] - Error connecting to the GeoNode server for authentication purposes 
2018-02-27T09:11:56.128936+00:00: org.springframework.security.authentication.AuthenticationServiceException: Communication with GeoNode failed (UsernamePasswordAuthenticationToken) 
2018-02-27T09:11:56.129161+00:00:   at org.geonode.security.GeoNodeAuthenticationProvider.authenticate(GeoNodeAuthenticationProvider.java:62) 
2018-02-27T09:11:56.129359+00:00:   at org.geoserver.security.GeoServerAuthenticationProvider.authenticate(GeoServerAuthenticationProvider.java:58) 
2018-02-27T09:11:56.129551+00:00:   at org.springframework.security.authentication.ProviderManager.authenticate(ProviderManager.java:167) 
2018-02-27T09:11:56.129753+00:00:   at org.geoserver.security.GeoServerSecurityManager$1.authenticate(GeoServerSecurityManager.java:323) 
2018-02-27T09:11:56.129936+00:00:   at org.geonode.security.GeoNodeAnonymousProcessingFilter.doFilter(GeoNodeAnonymousProcessingFilter.java:60) 
2018-02-27T09:11:56.130114+00:00:   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) 
2018-02-27T09:11:56.130321+00:00:   at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:69) 
--
2018-02-27T09:11:56.139811+00:00: Caused by: java.net.ConnectException: Connection refused (Connection refused) 
2018-02-27T09:11:56.139973+00:00:   at java.net.PlainSocketImpl.socketConnect(Native Method) 
2018-02-27T09:11:56.140136+00:00:   at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) 
2018-02-27T09:11:56.140297+00:00:   at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) 
2018-02-27T09:11:56.140456+00:00:   at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) 
2018-02-27T09:11:56.140626+00:00:   at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) 
2018-02-27T09:11:56.140810+00:00:   at java.net.Socket.connect(Socket.java:589) 
2018-02-27T09:11:56.140985+00:00:   at sun.reflect.GeneratedMethodAccessor250.invoke(Unknown Source) 
2018-02-27T09:11:56.141158+00:00:   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
2018-02-27T09:11:56.141328+00:00:   at java.lang.reflect.Method.invoke(Method.java:498) 
2018-02-27T09:11:56.141502+00:00:   at org.apache.commons.httpclient.protocol.ReflectionSocketFactory.createSocket(ReflectionSocketFactory.java:140) 
2018-02-27T09:11:56.141672+00:00:   at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:125) 
2018-02-27T09:11:56.141843+00:00:   at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) 
2018-02-27T09:11:56.142015+00:00:   at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) 
2018-02-27T09:11:56.142183+00:00:   at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) 
2018-02-27T09:11:56.142404+00:00:   at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) 
2018-02-27T09:11:56.142590+00:00:   at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) 
2018-02-27T09:11:56.142775+00:00:   at org.geonode.security.HTTPClient.sendGET(HTTPClient.java:89) 
2018-02-27T09:11:56.142964+00:00:   at org.geonode.security.DefaultSecurityClient.authenticate(DefaultSecurityClient.java:185) 
2018-02-27T09:11:56.143148+00:00:   at org.geonode.security.DefaultSecurityClient.authenticateAnonymous(DefaultSecurityClient.java:161) 
2018-02-27T09:11:56.143330+00:00:   at org.geonode.security.GeoNodeAuthenticationProvider.authenticate(GeoNodeAuthenticationProvider.java:49) 
2018-02-27T09:11:56.143506+00:00:   ... 60 more
fvanderbiest commented 6 years ago 
sftp@ADMIN-gfdrr-prod-app:/mnt/geoserver_datadir/security/auth/geonodeAuthProvider$ cat config.xml 
<org.geonode.security.GeoNodeAuthProviderConfig>
  <id>52857278:13c7ffd66a8:-7fe9</id>
  <name>geonodeAuthProvider</name>
  <className>org.geonode.security.GeoNodeAuthenticationProvider</className>
  <baseUrl>http://django:8000/</baseUrl>
</org.geonode.security.GeoNodeAuthProviderConfig>
fvanderbiest commented 6 years ago

https://github.com/camptocamp/terraform-geonode/blob/f3374326e3f4db02977ccc057a95e9e59b643b47/rancher-environments/gfdrr-geonode-prod/stacks.tf#L74 correctly maps geonode/django service onto django hostname and geoserver container sees django app:

root@24f2f4717255:/usr/local/tomcat# ping django                                                                                                                                             
PING django.rancher.internal (10.42.183.160): 56 data bytes
64 bytes from 10.42.183.160: icmp_seq=0 ttl=64 time=0.074 ms
64 bytes from 10.42.183.160: icmp_seq=1 ttl=64 time=0.065 ms
fvanderbiest commented 6 years ago 
2018-02-23T22:06:40.344920+00:00: 23 Feb 22:06:40 WARN [yaml.CustomBeanWrapper] - MetaData: unknown field 'footer' with value '{items=[org.mapfish.print.config.layout.ColumnsBlock@1ca175af], height=30}' 
2018-02-23T22:06:40.345102+00:00: 23 Feb 22:06:40 WARN [yaml.CustomBeanWrapper] - MainPage: Error while writing '{items=[org.mapfish.print.config.layout.ColumnsBlock@1ca175af], height=30}' to footer: java.lang.IllegalArgumentException: argument type mismatch 
2018-02-23T22:06:43.677652+00:00: 23 Feb 22:06:43 ERROR [geowebcache.GeoWebCacheDispatcher] - For input string: "{z}" http://www.geonode-gfdrrlab.org/geoserver/gwc/service/gmaps 
2018-02-23T22:06:43.678014+00:00: java.lang.NumberFormatException: For input string: "{z}" 
2018-02-23T22:06:43.678387+00:00:   at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) 
2018-02-23T22:06:43.678749+00:00:   at java.lang.Integer.parseInt(Integer.java:580) 
2018-02-23T22:06:43.679103+00:00:   at java.lang.Integer.parseInt(Integer.java:615) 
2018-02-23T22:06:43.679507+00:00:   at org.geowebcache.service.gmaps.GMapsConverter.getConveyor(GMapsConverter.java:77) 
2018-02-23T10:14:50.656802+00:00: 23 Feb 10:14:50 ERROR [geoserver.rest] - No such layer group eq-data_1380 
2018-02-23T10:14:50.686856+00:00: 23 Feb 10:14:50 ERROR [geoserver.rest] -  
2018-02-23T10:14:50.687135+00:00: org.geoserver.rest.RestletException 
2018-02-23T10:14:50.687394+00:00:   at org.geoserver.catalog.rest.LayerGroupFinder.findTarget(LayerGroupFinder.java:49) 
2018-02-23T10:14:50.687645+00:00:   at org.restlet.Finder.handle(Finder.java:268) 
2018-02-23T09:10:22.705298+00:00: 23 Feb 09:10:22 ERROR [geoserver.ows] -  
2018-02-23T09:10:22.705725+00:00: org.geoserver.platform.ServiceException: No such operation wms 1.3.0 GetCapabilities?access_token=607e95d8187911e8848802a677f5a383 
2018-02-23T09:10:22.706028+00:00:   at org.geoserver.ows.Dispatcher.dispatch(Dispatcher.java:639) 
2018-02-23T09:10:22.706348+00:00:   at org.geoserver.ows.Dispatcher.handleRequestInternal(Dispatcher.java:258) 
2018-02-23T09:10:22.706639+00:00:   at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:147) 
2018-02-23T09:10:22.706905+00:00:   at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:50)
fvanderbiest commented 6 years ago

The front LB was the culprit for some weird reason. It should now be fixed.

fvanderbiest commented 6 years ago

planning a LB upgrade with https://github.com/camptocamp/terraform-geonode/pull/19/files