search

kartoza / docker-geoserver

A docker recipe for GeoServer
GNU General Public License v2.0
622 stars 422 forks source link

GeoServer do not start when upgrading from 2.9.3 to 2.19.0 #250

Closed tlvu closed 3 years ago

tlvu commented 3 years ago

Here is our docker-compose stack for the upgrade https://github.com/bird-house/birdhouse-deploy/pull/136.

A fresh start on GeoServer 2.19.0 without existing data works fine.

But when trying to upgrade from 2.9.3 to 2.19.0 with existing data, GeoServer failed to start.

There is nothing useful in the catalina.log (from docker logs geoserver) and geoserver.log (from docker exec geoserver cat /opt/geoserver/data_dir/logs/geoserver.log).

Are there additional logs file I am missing. Any hints to help debug this problem is appreciated.

Here are the content of the catalina.log and geoserver.log: catalina.log geoserver.log

tlvu commented 3 years ago

I think this is most suspicious log from tomcat:

WARNING: An illegal reflective access operation has occurred                                                                                          
WARNING: Illegal reflective access by org.geotools.image.ImageWorker (file:/usr/local/tomcat/webapps/geoserver/WEB-INF/lib/gt-coverage-25.0.jar) to constructor com.sun.imageio.plugins.jpeg.JPEGImageWriterSpi()                                                                                           
WARNING: Please consider reporting this to the maintainers of org.geotools.image.ImageWorker                                                          
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations                                                 
WARNING: All illegal access operations will be denied in a future release                                                                             
20-May-2021 17:45:48.930 SEVERE [main] org.apache.catalina.core.StandardContext.startInternal One or more listeners failed to start. Full details will be found in the appropriate container log file                                                                                                       
20-May-2021 17:45:48.934 SEVERE [main] org.apache.catalina.core.StandardContext.startInternal Context [/geoserver] startup failed due to previous errors

Can someone share a hint how to diagnose this error?

NyakudyaA commented 3 years ago

@tlvu i will answer you properly tomorrow morning but my guess is that from geoserver 2.9 they are a lot of methods which have been deprecated I.e no data support in rasters etc. if I were you I would maybe do the upgrade process in stages.

This way you can easily figure out what store/ workspace is giving errors

tlvu commented 3 years ago

@NyakudyaA I think I found how to remove the WARNING: Illegal reflective access by org.geotools.image.ImageWorker, see my commit https://github.com/bird-house/birdhouse-deploy/pull/136/commits/601cef7db2e6a4070e898c7277e52934153db207. Honestly I am not sure what that config does but I do not see the warning in the logs anymore. Maybe you could take a quick look if this is a good fix.

The only remaining suspicious warning is a bunch of these below. They were already there with the other "illegal reflective access" but I thought the "illegal reflective access" were more suspicious.

Will also try your suggestion to remove store/workspace gradually. Thanks for the leads.

20-May-2021 20:21:22.865 WARNING [main] org.geoserver.platform.GeoServerExtensions.checkContext Extension lookup 'XStreamPersisterInitializer', but ApplicationContext is unset.
20-May-2021 20:21:22.866 WARNING [main] org.geoserver.platform.GeoServerExtensions.checkContext Extension lookup 'ExtensionFilter', but ApplicationContext is unset.                            
20-May-2021 20:21:22.869 WARNING [main] org.geoserver.platform.GeoServerExtensions.checkContext Extension lookup 'ExtensionProvider', but ApplicationContext is unset.
20-May-2021 20:21:22.870 WARNING [main] org.geoserver.platform.GeoServerExtensions.checkContext Extension lookup 'ExtensionFilter', but ApplicationContext is unset.                                
log4j:WARN File option not set for appender [geoserverlogfile].                                                                                                                                   
log4j:WARN Are you using FileAppender instead of ConsoleAppender?                                                                                                               
20 May 20:21:22 INFO [geoserver.logging] - Suppressing StdOut logging.  If you want to see GeoServer logs, be sure to look in '/opt/geoserver/data_dir/logs/geoserver.log'
20-May-2021 20:21:26.101 WARNING [main] org.geoserver.platform.GeoServerExtensions.checkContext Extension lookup 'MonitorDAO', but ApplicationContext is unset.
20-May-2021 20:21:26.101 WARNING [main] org.geoserver.platform.GeoServerExtensions.checkContext Extension lookup 'ExtensionFilter', but ApplicationContext is unset.
20-May-2021 20:21:26.101 WARNING [main] org.geoserver.platform.GeoServerExtensions.checkContext Extension lookup 'ExtensionProvider', but ApplicationContext is unset.
20-May-2021 20:21:26.102 WARNING [main] org.geoserver.platform.GeoServerExtensions.checkContext Extension lookup 'ExtensionFilter', but ApplicationContext is unset.
20-May-2021 20:21:26.296 WARNING [main] org.geoserver.platform.GeoServerExtensions.checkContext Extension lookup 'GeoServerSecurityProvider', but ApplicationContext is unset.
20-May-2021 20:21:26.296 WARNING [main] org.geoserver.platform.GeoServerExtensions.checkContext Extension lookup 'ExtensionFilter', but ApplicationContext is unset.
20-May-2021 20:21:26.297 WARNING [main] org.geoserver.platform.GeoServerExtensions.checkContext Extension lookup 'ExtensionProvider', but ApplicationContext is unset.
20-May-2021 20:21:26.297 WARNING [main] org.geoserver.platform.GeoServerExtensions.checkContext Extension lookup 'ExtensionFilter', but ApplicationContext is unset.
20-May-2021 20:21:44.810 SEVERE [main] org.apache.catalina.core.StandardContext.startInternal One or more listeners failed to start. Full details will be found in the appropriate container log file
20-May-2021 20:21:44.811 SEVERE [main] org.apache.catalina.core.StandardContext.startInternal Context [/geoserver] startup failed due to previous errors
tlvu commented 3 years ago

Someone else got the same "ApplicationContext unset" and "Illegal reflective access" with Geoserver 2.18.2 in a debian 10 machine with JavaSDK 11 (debian 10 distro) and Tomcat 9 (debian 10 distro): http://osgeo-org.1560.x6.nabble.com/Geoserver-2-18-2-ApplicationContext-unset-and-Illegal-reflective-access-by-org-geotools-image-ImageWr-td5482884.html

Unfortunately no reply in that thread.

tlvu commented 3 years ago

@NyakudyaA I followed your advice and gradually delete stuff in our existing data to see which one is causing Geoserver to fail at starting up. So I wipe everything gradually: workspaces, stores, layers, styles until I got to the extra admin users. It is all the extra admin users that broke the start-up ! Who knows !

I still have no idea why all the existing admin users broke the start-up of the new version. There was nothing related in the logs.