search

gentics / mesh-incubator

Project which is home for planned enhancements for Gentics Mesh
3 stars 0 forks source link

Rename Releases to Branches break on big database #180

Closed deckdom closed 4 years ago

deckdom commented 6 years ago

Gentics Mesh Version, operating system, or hardware.

Problem

The RenameReleasesToBranches Job/Handler seems to have some issues with a big database. It seems like it gets a OutOfMemoryError or some other kind of Error which prevents the Job to progress. Sadly, there's no Stacktrace in the Logs, just a paused process indicator:

11:28:28.431 [Nostalgic Ferrothorn] INFO  [main] [RenameReleasesToBranches.java:80] - Migrated {595300} edges
11:28:37.281 [Nostalgic Ferrothorn] INFO  [main] [RenameReleasesToBranches.java:80] - Migrated {595400} edges
11:28:46.340 [Nostalgic Ferrothorn] INFO  [main] [RenameReleasesToBranches.java:80] - Migrated {595500} edges
11:29:17.844 [Nostalgic Ferrothorn] INFO  [main] [RenameReleasesToBranches.java:80] - Migrated {595600} edges
12:37:03.939 [Nostalgic Ferrothorn] INFO  [Thread-0] [MeshImpl.java:352] - Mesh shutting down...

I started mesh with a heapdump output and got one. Will transmit it internally since it contains sensitive data.

Attempting to start the Server after the hickup, results in this Error which prevents it from starting:

13:05:59.447 [Nostalgic Ferrothorn] INFO  [main] [ElasticSearchProvider.java:152] - Waiting for elasticsearch status...
13:06:00.450 [Nostalgic Ferrothorn] INFO  [main] [ElasticSearchProvider.java:145] - Elasticsearch is ready. Releasing lock after 22506 ms
13:06:12.276 [Nostalgic Ferrothorn] INFO  [main] [BootstrapInitializerImpl.java:523] - Invoking database changelog check...
13:06:12.423 [Nostalgic Ferrothorn] INFO  [main] [ChangelogSystem.java:49] - Handling change {53C59D7EBC94483B859D7EBC94083BD6}
13:06:12.424 [Nostalgic Ferrothorn] INFO  [main] [ChangelogSystem.java:50] - Name: ReleaseBranchRenameChange
13:06:12.424 [Nostalgic Ferrothorn] INFO  [main] [ChangelogSystem.java:51] - Description: Renames the release vertices, edges and properties to branch
13:06:12.429 [Nostalgic Ferrothorn] ERROR [main] [ChangelogSystem.java:65] - Error while handling change {53C59D7EBC94483B859D7EBC94083BD6/ReleaseBranchRenameChange}. Invoking rollback..
com.orientechnologies.orient.core.exception.OSchemaException: Cannot change the schema while a transaction is active. Schema changes are not transactional
        DB name="storage"
        at com.orientechnologies.orient.core.metadata.schema.OSchemaShared.saveInternal(OSchemaShared.java:1313) ~[mesh-server-0.24.0.jar:na]
        at com.orientechnologies.orient.core.metadata.schema.OSchemaShared.releaseSchemaWriteLock(OSchemaShared.java:628) ~[mesh-server-0.24.0.jar:na]
        at com.orientechnologies.orient.core.metadata.schema.OClassImpl.releaseSchemaWriteLock(OClassImpl.java:2073) ~[mesh-server-0.24.0.jar:na]
        at com.orientechnologies.orient.core.metadata.schema.OClassImpl.releaseSchemaWriteLock(OClassImpl.java:2068) ~[mesh-server-0.24.0.jar:na]
        at com.orientechnologies.orient.core.metadata.schema.OClassImpl.setSuperClasses(OClassImpl.java:411) ~[mesh-server-0.24.0.jar:na]
        at com.orientechnologies.orient.core.metadata.schema.OClassImpl.setSuperClass(OClassImpl.java:321) ~[mesh-server-0.24.0.jar:na]
        at com.orientechnologies.orient.core.metadata.schema.OClassAbstractDelegate.setSuperClass(OClassAbstractDelegate.java:80) ~[mesh-server-0.24.0.jar:na]
        at com.gentics.mesh.graphdb.OrientDBDatabase.addVertexType(OrientDBDatabase.java:713) ~[mesh-server-0.24.0.jar:na]
        at com.gentics.mesh.changelog.changes.RenameReleasesToBranches.apply(RenameReleasesToBranches.java:24) ~[mesh-server-0.24.0.jar:na]
        at com.gentics.mesh.changelog.ChangelogSystem.applyChanges(ChangelogSystem.java:54) [mesh-server-0.24.0.jar:na]
        at com.gentics.mesh.changelog.ChangelogSystem.applyChanges(ChangelogSystem.java:101) [mesh-server-0.24.0.jar:na]
        at com.gentics.mesh.cli.BootstrapInitializerImpl.invokeChangelog(BootstrapInitializerImpl.java:525) [mesh-server-0.24.0.jar:na]
        at com.gentics.mesh.cli.BootstrapInitializerImpl.initLocalData(BootstrapInitializerImpl.java:204) [mesh-server-0.24.0.jar:na]
        at com.gentics.mesh.cli.BootstrapInitializerImpl.init(BootstrapInitializerImpl.java:279) [mesh-server-0.24.0.jar:na]
        at com.gentics.mesh.cli.MeshImpl.run(MeshImpl.java:139) [mesh-server-0.24.0.jar:na]
        at com.gentics.mesh.cli.MeshImpl.run(MeshImpl.java:103) [mesh-server-0.24.0.jar:na]
13:06:12.430 [Nostalgic Ferrothorn] ERROR [main] [MeshImpl.java:149] - Error while starting mesh
java.lang.RuntimeException: The changelog could not be applied successfully. See log above.
        at com.gentics.mesh.cli.BootstrapInitializerImpl.invokeChangelog(BootstrapInitializerImpl.java:526) ~[mesh-server-0.24.0.jar:na]
        at com.gentics.mesh.cli.BootstrapInitializerImpl.initLocalData(BootstrapInitializerImpl.java:204) ~[mesh-server-0.24.0.jar:na]
        at com.gentics.mesh.cli.BootstrapInitializerImpl.init(BootstrapInitializerImpl.java:279) ~[mesh-server-0.24.0.jar:na]
        at com.gentics.mesh.cli.MeshImpl.run(MeshImpl.java:139) [mesh-server-0.24.0.jar:na]
        at com.gentics.mesh.cli.MeshImpl.run(MeshImpl.java:103) [mesh-server-0.24.0.jar:na]
13:06:12.430 [Nostalgic Ferrothorn] INFO  [main] [MeshImpl.java:352] - Mesh shutting down...
13:06:13.006 [Nostalgic Ferrothorn] INFO  [main] [ElasticSearchProvider.java:258] - Closing Elasticsearch REST client.
13:06:13.006 [Nostalgic Ferrothorn] INFO  [main] [ElasticSearchProvider.java:262] - Stopping Elasticsearch server.
13:06:14.006 [Nostalgic Ferrothorn] INFO  [main] [ElasticsearchProcessManager.java:260] - Terminating Elasticsearch process..
13:06:15.007 [] INFO  [main] [MeshImpl.java:372] - Shutdown completed...
13:07:00.503 [] INFO  [Thread-0] [MeshImpl.java:352] - Mesh shutting down...
13:07:00.504 [] INFO  [Thread-0] [ElasticSearchProvider.java:258] - Closing Elasticsearch REST client.
13:07:00.504 [] INFO  [Thread-0] [ElasticSearchProvider.java:262] - Stopping Elasticsearch server.
13:07:00.504 [] INFO  [Thread-0] [MeshImpl.java:372] - Shutdown completed...

Expected behaviour and actual behaviour

That the Job isn't failing (smaller batch-size?) and/or that the Job is able to continue where it left off the last time, instead of failing.

Jotschi commented 4 years ago

Fixed with 0.38.1