Error in database migration from 3.4.5 to 3.6.0

[juanluisrp]

Describe the bug When deploying GN 3.6.0 with an old database from 3.4.x branch there is an exception that makes the migration fail

Log file

2019-01-23 11:10:06,817 WARN  [geonetwork.databasemigration] -       No migration task found between webapp and database version.
      The system may be unstable or may failed to start if you try to run
      the current GeoNetwork 3.6.1-SNAPSHOT with an older database (ie. 3.4.5
      ). Try to run the migration task manually on the current database
      before starting the application or start with a new empty database.
      Sample SQL scripts for migration could be found in WEB-INF/sql/migrate folder.

2019-01-23 11:10:24,874 ERROR [geonetwork.databasemigration] -           Errors occurs during Java migration file: java.lang.ClassCastException: com.fasterxml.jackson.databind.node.MissingNode cannot be cast to com.fasterxml.jackson.databind.node.ObjectNode
2019-01-23 11:10:24,950 WARN  [geonetwork.databasemigration] -       Error occurs during migration. Check the log file for more details.

Additional context The only two places where the cast to ObjectNode is done are: https://github.com/geonetwork/core-geonetwork/blob/8f5d93ed43be441bdbfdb5e43eae0547ec01f64d/core/src/main/java/org/fao/geonet/migration/JsonDatabaseMigration.java#L113-L120

and https://github.com/geonetwork/core-geonetwork/blob/8f5d93ed43be441bdbfdb5e43eae0547ec01f64d/core/src/main/java/org/fao/geonet/migration/JsonDatabaseMigration.java#L90-L110

[fxprunayre]

We tested migration from 3.4.0>3.6.0 and 3.4.4>3.6.0 on both h2 and postgres and did not get that one. Which version are you running ?

[zer0mode]

Got the same error while trying to migrate from 3.4.2.0 > 3.6.0 :

2019-02-14 16:08:09,655 WARN  [geonetwork.databasemigration] -       No migration task found between webapp and database version.
  The system may be unstable or may failed to start if you try to run 
  the current GeoNetwork 3.6.0-0 with an older database (ie. 3.4.2
  ). Try to run the migration task manually on the current database
  before starting the application or start with a new empty database.
  Sample SQL scripts for migration could be found in WEB-INF/sql/migrate folder.

2019-02-14 16:08:17,798 ERROR [geonetwork.databasemigration] -           Errors occurs during Java migration file: java.lang.ClassCastException: com.fasterxml.jackson.databind.node.MissingNode cannot be cast to com.fasterxml.jackson.databind.node.ObjectNode
2019-02-14 16:08:17,819 WARN  [geonetwork.databasemigration] -       Error occurs during migration. Check the log file for more details.

Ubuntu 16.04.5 LTS

Database was imported in postgres, however with one exception right at the start of importation : sudo -u postgres psql db < importdb.dump

SET
SET
ERROR:  unrecognized configuration parameter "idle_in_transaction_session_timeout"
SET
SET
 set_config 
------------

[zer0mode]

Got the same error while trying to migrate from 3.4.2.0 > 3.6.0 :

2019-02-14 16:08:09,655 WARN  [geonetwork.databasemigration] -       No migration task found between webapp and database version.
  The system may be unstable or may failed to start if you try to run 
  the current GeoNetwork 3.6.0-0 with an older database (ie. 3.4.2
  ). Try to run the migration task manually on the current database
  before starting the application or start with a new empty database.
  Sample SQL scripts for migration could be found in WEB-INF/sql/migrate folder.

2019-02-14 16:08:17,798 ERROR [geonetwork.databasemigration] -           Errors occurs during Java migration file: java.lang.ClassCastException: com.fasterxml.jackson.databind.node.MissingNode cannot be cast to com.fasterxml.jackson.databind.node.ObjectNode
2019-02-14 16:08:17,819 WARN  [geonetwork.databasemigration] -       Error occurs during migration. Check the log file for more details.

Ubuntu 16.04.5 LTS

Database was imported in postgres, however with one exception right at the start of importation : sudo -u postgres psql db < importdb.dump

SET
SET
ERROR:  unrecognized configuration parameter "idle_in_transaction_session_timeout"
SET
SET
 set_config 
------------

update :

After re-examining the error the issue is probably related to schema validation process. It could be the reason which prevented execution of migration. The migration bug might not even exist.

Message | Failed whilst validating schema-ident file /var/lib/tomcat/webapps/geonetwork/WEB-INF/data/config/schema_plugins/iso19110/schema-ident.xml. Exception message if any is XSD Validation error(s): <xsderrors>   <error>     <typeOfError>ERROR</typeOfError>     <errorNumber>1</errorNumber>     <message>cvc-complex-type.2.4.a: Invalid content was found starting with element 'title'. One of '{"http://geonetwork-opensource.org/schemas/schema-ident":appMinorVersionSupported}' is expected. (Element: title with parent element: schema)</message>     <xpath>title[1]</xpath>   </error> </xsderrors>
-- | --
Handler | org.fao.geonet.Geonetwork
Exception | OperationAbortedEx : Failed whilst validating schema-ident file /var/lib/tomcat/webapps/geonetwork/WEB-INF/data/config/schema_plugins/iso19110/schema-ident.xml. Exception message if any is XSD Validation error(s): <xsderrors>   <error>     <typeOfError>ERROR</typeOfError>     <errorNumber>1</errorNumber>     <message>cvc-complex-type.2.4.a: Invalid content was found starting with element 'title'. One of '{"http://geonetwork-opensource.org/schemas/schema-ident":appMinorVersionSupported}' is expected. (Element: title with parent element: schema)</message>     <xpath>title[1]</xpath>   </error> </xsderrors>
Stack | OperationAbortedEx : Failed whilst validating schema-ident file /var/lib/tomcat/webapps/geonetwork/WEB-INF/data/config/schema_plugins/iso19110/schema-ident.xml. Exception message if any is XSD Validation error(s): <xsderrors>   <error>     <typeOfError>ERROR</typeOfError>     <errorNumber>1</errorNumber>     <message>cvc-complex-type.2.4.a: Invalid content was found starting with element 'title'. One of '{"http://geonetwork-opensource.org/schemas/schema-ident":appMinorVersionSupported}' is expected. (Element: title with parent element: schema)</message>     <xpath>title[1]</xpath>   </error> </xsderrors>     at org.fao.geonet.kernel.SchemaManager.processSchema(SchemaManager.java:1302)   at org.fao.geonet.kernel.SchemaManager.configure(SchemaManager.java:224)    at org.fao.geonet.Geonetwork.start(Geonetwork.java:245)     at jeeves.server.JeevesEngine.initAppHandler(JeevesEngine.java:448)     at jeeves.server.JeevesEngine.init(JeevesEngine.java:191)   at jeeves.server.sources.http.JeevesServlet.init(JeevesServlet.java:84)     at javax.servlet.GenericServlet.init(GenericServlet.java:158)   at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1199)  at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1143)  at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1032)     at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5006)    at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5298)    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147)     at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:725)  at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:701)  at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:717)    at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:939)    at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1812)   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748) Caused by: XSDValidationErrorEx : XSD Validation error(s): <xsderrors>   <error>     <typeOfError>ERROR</typeOfError>     <errorNumber>1</errorNumber>     <message>cvc-complex-type.2.4.a: Invalid content was found starting with element 'title'. One of '{"http://geonetwork-opensource.org/schemas/schema-ident":appMinorVersionSupported}' is expected. (Element: title with parent element: schema)</message>     <xpath>title[1]</xpath>   </error> </xsderrors>   at org.fao.geonet.utils.Xml.validate(Xml.java:856)  at org.fao.geonet.kernel.SchemaManager.processSchema(SchemaManager.java:1288)   ... 22 more

I removed the whole /var/lib/tomcat/webapps/geonetwork/WEB-INF/data/config/ (data folder form 3.4.2 version) and copied contents from the 'fresh' 3.6.0 version, just the .config part. After that geonetwork launched normally.

[ReneAgius]

Hi! I just upgraded GN from version 2.2.0 to version 3.6.0 but the h2 database migration failed and a new DB was created. Is there any way how to prevent this from happening please?