kartoza / docker-geoserver

A docker recipe for GeoServer
GNU General Public License v2.0
620 stars 421 forks source link

Caused by: java.io.UncheckedIOException: java.io.IOException: Stale file handle #389

Closed VP2405 closed 2 years ago

VP2405 commented 2 years ago

Hi, I'm using kartoza/geoserver:2.19.0 on GKE, with a deployment with 2 replicas of geoserver. The replicas are backed by a NFS persistent volume. Geoserver is served in https behind a proxy (the Proxy Base URL in Global configuration is correct, in https), randomly we get the following error:

17 Mar 15:34:55 ERROR [geowebcache.GeoWebCacheDispatcher] - Request failed
org.geowebcache.GeoWebCacheException: Problem communicating with GeoServer
    at org.geoserver.gwc.layer.GeoServerTileLayer.getMetatilingReponse(GeoServerTileLayer.java:657)
    at org.geoserver.gwc.layer.GeoServerTileLayer.getTile(GeoServerTileLayer.java:592)
    at org.geowebcache.util.ResponseUtils.writeTile(ResponseUtils.java:100)
    at org.geowebcache.GeoWebCacheDispatcher.handleServiceRequest(GeoWebCacheDispatcher.java:408)
    at org.geowebcache.GeoWebCacheDispatcher.handleRequestInternal(GeoWebCacheDispatcher.java:268)
    at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:177)
    at org.geoserver.gwc.dispatch.GwcServiceProxy.dispatch(GwcServiceProxy.java:80)
    at jdk.internal.reflect.GeneratedMethodAccessor243.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.geoserver.ows.Dispatcher.execute(Dispatcher.java:876)
    at org.geoserver.ows.Dispatcher.handleRequestInternal(Dispatcher.java:265)
    at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:177)
    at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:52)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:626)
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:733)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.apache.catalina.filters.CorsFilter.handleSimpleCORS(CorsFilter.java:259)
    at org.apache.catalina.filters.CorsFilter.doFilter(CorsFilter.java:163)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.geoserver.filters.ThreadLocalsCleanupFilter.doFilter(ThreadLocalsCleanupFilter.java:26)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.geoserver.filters.SpringDelegatingFilter$Chain.doFilter(SpringDelegatingFilter.java:69)
    at org.geoserver.monitor.MonitorFilter.doFilter(MonitorFilter.java:142)
    at org.geoserver.filters.SpringDelegatingFilter$Chain.doFilter(SpringDelegatingFilter.java:66)
    at org.geoserver.flow.controller.IpBlacklistFilter.doFilter(IpBlacklistFilter.java:89)
    at org.geoserver.filters.SpringDelegatingFilter$Chain.doFilter(SpringDelegatingFilter.java:66)
    at org.geoserver.wms.animate.AnimatorFilter.doFilter(AnimatorFilter.java:70)
    at org.geoserver.filters.SpringDelegatingFilter$Chain.doFilter(SpringDelegatingFilter.java:66)
    at org.geoserver.filters.SpringDelegatingFilter.doFilter(SpringDelegatingFilter.java:41)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.geoserver.platform.AdvancedDispatchFilter.doFilter(AdvancedDispatchFilter.java:37)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:320)
    at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:70)
    at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
    at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
    at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:74)
    at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:91)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
    at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:70)
    at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119)
    at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:74)
    at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:91)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
    at org.geoserver.security.filter.GeoServerAnonymousAuthenticationFilter.doFilter(GeoServerAnonymousAuthenticationFilter.java:51)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
    at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:70)
    at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:158)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:74)
    at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:91)
    at org.geoserver.security.filter.GeoServerBasicAuthenticationFilter.doFilter(GeoServerBasicAuthenticationFilter.java:81)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
    at org.geoserver.security.filter.GeoServerPreAuthenticationFilter.doFilter(GeoServerPreAuthenticationFilter.java:71)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
    at org.geoserver.security.filter.GeoServerRoleFilter.doFilter(GeoServerRoleFilter.java:54)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
    at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:70)
    at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
    at org.geoserver.security.filter.GeoServerSecurityContextPersistenceFilter$1.doFilter(GeoServerSecurityContextPersistenceFilter.java:52)
    at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:74)
    at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:91)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
    at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:215)
    at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178)
    at org.geoserver.security.GeoServerSecurityFilterChainProxy.doFilter(GeoServerSecurityFilterChainProxy.java:142)
    at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358)
    at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.geoserver.filters.LoggingFilter.doFilter(LoggingFilter.java:101)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.geoserver.filters.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:77)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.geoserver.filters.GZIPFilter.doFilter(GZIPFilter.java:47)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.geoserver.filters.SessionDebugFilter.doFilter(SessionDebugFilter.java:46)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.geoserver.filters.FlushSafeFilter.doFilter(FlushSafeFilter.java:42)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
    at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:374)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1707)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.io.UncheckedIOException: java.io.IOException: Stale file handle
    at org.geowebcache.storage.blobstore.file.LayerMetadataStore.loadLayerMetadata(LayerMetadataStore.java:305)
    at org.geowebcache.storage.blobstore.file.LayerMetadataStore.loadLayerMetadata(LayerMetadataStore.java:337)
    at org.geowebcache.storage.blobstore.file.LayerMetadataStore.putEntry(LayerMetadataStore.java:106)
    at org.geowebcache.storage.blobstore.file.FileBlobStore.putLayerMetadata(FileBlobStore.java:657)
    at org.geowebcache.storage.blobstore.file.FileBlobStore.persistParameterMap(FileBlobStore.java:581)
    at org.geowebcache.storage.blobstore.file.FileBlobStore.lambda$put$0(FileBlobStore.java:477)
    at org.geowebcache.storage.blobstore.file.FileBlobStore.getFileHandleTile(FileBlobStore.java:528)
    at org.geowebcache.storage.blobstore.file.FileBlobStore.put(FileBlobStore.java:479)
    at org.geowebcache.storage.CompositeBlobStore.lambda$put$6(CompositeBlobStore.java:186)
    at org.geowebcache.storage.CompositeBlobStore.lambda$readActionUnsafe$20(CompositeBlobStore.java:508)
    at org.geowebcache.storage.CompositeBlobStore.readFunctionUnsafe(CompositeBlobStore.java:490)
    at org.geowebcache.storage.CompositeBlobStore.readActionUnsafe(CompositeBlobStore.java:505)
    at org.geowebcache.storage.CompositeBlobStore.put(CompositeBlobStore.java:186)
    at org.geoserver.gwc.ConfigurableBlobStore.put(ConfigurableBlobStore.java:225)
    at org.geowebcache.storage.DefaultStorageBroker.put(DefaultStorageBroker.java:85)
    at org.geowebcache.layer.TileLayer.saveTiles(TileLayer.java:543)
    at org.geoserver.gwc.layer.GeoServerTileLayer.getMetatilingReponse(GeoServerTileLayer.java:654)
    ... 119 more
Caused by: java.io.IOException: Stale file handle
    at java.base/java.io.FileInputStream.available0(Native Method)
    at java.base/java.io.FileInputStream.available(FileInputStream.java:330)
    at java.base/java.util.zip.GZIPInputStream.readTrailer(GZIPInputStream.java:232)
    at java.base/java.util.zip.GZIPInputStream.read(GZIPInputStream.java:120)
    at java.base/java.io.FilterInputStream.read(FilterInputStream.java:107)
    at java.base/java.util.Properties$LineReader.readLine(Properties.java:597)
    at java.base/java.util.Properties.load0(Properties.java:418)
    at java.base/java.util.Properties.load(Properties.java:407)
    at org.geowebcache.storage.blobstore.file.LayerMetadataStore.loadLayerMetadata(LayerMetadataStore.java:294)
    ... 135 more

The GKE manifest is the following:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: geoserver-nfs-ha
  namespace: geoserver-ha
spec:
  capacity:
    storage: 20Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteMany
  nfs:
    path: /data
    server: 171.25.2.20
  mountOptions:
    - hard
    - proto=tcp
    - nfsvers=3
    - rsize=65536
    - wsize=65536
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: geoserver-pvc-nfs-prod-ha
  namespace: geoserver-ha
spec:
  storageClassName: ""
  accessModes:
    - ReadWriteMany
  volumeName: geoserver-nfs-ha
  resources:
    requests:
      storage: 20Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: geoserver
  labels:
    app: geoserver-ha
  namespace: geoserver-ha
spec:
  replicas: 2
  selector:
    matchLabels:
      app: geoserver-ha
  template:
    metadata:
      labels:
        app: geoserver-ha
    spec:
      securityContext:
        runAsUser: 1000
        fsGroup: 10001
      containers:
        - name: geoserver
          image: kartoza/geoserver:2.19.0
          imagePullPolicy: Always
          ports:
            - containerPort: 8080
          volumeMounts:
            - name: pvc-geoserver-prod
              mountPath: /opt/geoserver/data_dir
              subPath: "data_dir"
            - name: pvc-geoserver-prod
              mountPath: /opt/geoserver/data_dir/gwc
              subPath: "gwc"
      volumes:
        - name: pvc-geoserver-prod
          persistentVolumeClaim:
            claimName: geoserver-pvc-nfs-prod-ha
            readOnly: false
      restartPolicy: Always
...
...

What could be the problem?

NyakudyaA commented 2 years ago

Seems like a gwc error here. Are you using the default gwc storage? In production environments is recommended to use a database backend for storing gwc configuration. I suggest you change to that https://github.com/kartoza/docker-geoserver#enable-disk-quota-storage-in-postgresql-backend

VP2405 commented 2 years ago

Yes we are using an external database:

Schermata 2022-03-29 alle 12 58 59
NyakudyaA commented 2 years ago

Can you check https://geoserver.geo-solutions.it/multidim/caching_data/diskquotas.html

Maybe you need to add locking like the article above

  <lockProvider>nioLock</lockProvider>

I think it might be some timeout issues happening with your database. Try to look at the database logs to see if they are any errors showing up

VP2405 commented 2 years ago

We have disabled the external database just to do a test, and set the disk quota store type to the in process database (H2) just to see if the exception still occurs, and now it seems working.

Looking at Postgres logs before doing the switch, the only errors shown were the following:

By the way, we still have another error, seen also when using postgres.

When using geoserver intensively, on requests like this:

https://geoserver.ourdomain.com/geoserver/gwc/service/wms?&service=WMS&request=GetMap&layers=gisdb:our_db&styles=our_style&format=image/png&transparent=true&version=1.1.0&tiled=true&cql_filter=sub_cod='39400'&width=256&height=256&srs=EPSG:900913&bbox=1392988.4034690522,5472891.225218621,1394211.395921615,5474114.217671181

sometimes we get CORS errors. Waiting some minutes and retrying the same requests, all works again. CORS are enabled on tomcat and are working fine for low/mid loads.

Any help can be appreciated, thanks.

NyakudyaA commented 2 years ago

@VP2405 for the stale handler this is mainly caused by running the embedded geowebcache with multiple geoserver instances. This is not recommended. They recommend running a stand-alone geowebcache instance which each geoserver instance can be linked to a standalone version.

I will open a ticket and we can investigate this

Martenz commented 2 years ago

Dear @NyakudyaA

after some other tests as @VP2405 said, disabling external database and using H2, we got also this error:

Loading quota store failed, the disk quota subsystem is disabled, please re-configure: Could not open JDBC Connection for transaction; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Database may be already in use: Locked by another process. Possible solutions: close all other connection(s); use the server mode [90020-119])

the error occurs when we update some layers data and force to refresh cache with manual Empty process from interface.

Any hint on this? How is it going with the investigation on the previous issue? Do you think they related in someway ?

regards,

Martino

NyakudyaA commented 2 years ago

@Martenz The H2 database I think is based on sqlite implementation and it doesn't natively support multi writes concurrently. I think because of this that is why it is recommended to use Postgres in a production environment but however when in a clustered environment you should use the stand alone gwc as the only solution for clustering.

Martenz commented 2 years ago

@NyakudyaA thanks, will try with stand alone. meanwhile going back to single instance. Best regards