search

convergencelabs / convergence-project

The project used for Convergence Project Management and Issue Reporting
https://convergence.io
42 stars 5 forks source link

Convergence server cannot initialize orientdb schema #269

Closed devarmada closed 2 years ago

devarmada commented 2 years ago

Versions Please fill in all that apply:

Describe the Bug I tried to setup a convergence installation on a Kubernetes cluster copying the docker-compose example configuration. When I start the convergence server, it fails with this error: com.convergencelabs.convergence.server.backend.db.UpgradeException: A state persistence error occurred: The schema is not installed

It seems that the server cannot initialize the orientdb schema at the first start. I am suspecting a memory issue on the database, but I don't know how to investigate further.

This is the exception thrown by the server:

14:33:48 DEBUG ConvergenceDatabaseInitializer - Processing request to ensure the convergence database is initialized
14:33:48 INFO  ConvergenceDatabaseInitializer - Attempting to connect to the database at uri: remote:orientdb:2424
14:33:51 WARN  SystemErr - Apr 29, 2022 2:33:51 PM com.orientechnologies.common.log.OLogManager log
INFO: Detected limit of amount of simultaneously open files is 1048576,  limit of open files for disk cache will be set to 523776

14:33:52 INFO  ConvergenceDatabaseInitializer - Connected to database with Server Admin
14:33:53 INFO  ConvergenceServerActor - Role 'realtimeApi' detected, activating the Realtime API...
14:33:53 DEBUG TcpListener - Successfully bound to /0.0.0.0:8081
14:33:53 INFO  ConvergenceRestApi - Rest API started at: http://0.0.0.0:8081
14:33:53 DEBUG TcpListener - Successfully bound to /0.0.0.0:8080
14:33:53 INFO  ConvergenceRealtimeApi - Realtime API started at: http://0.0.0.0:8080
14:33:55 ERROR ConvergenceServerActor - The was an error starting the ConvergenceServerActor
com.convergencelabs.convergence.server.backend.db.UpgradeException: A state persistence error occurred: The schema is not installed
    at com.convergencelabs.convergence.server.backend.db.ConvergenceDatabaseInitializer.$anonfun$upgradeConvergenceDatabaseIfNeeded$2(ConvergenceDatabaseInitializer.scala:464) ~[com.convergencelabs.convergence-server-1.0.0-rc.12.jar:1.0.0-rc.12]
    at scala.util.Either.fold(Either.scala:190) ~[org.scala-lang.scala-library-2.13.5.jar:?]
    at com.convergencelabs.convergence.server.backend.db.ConvergenceDatabaseInitializer.$anonfun$upgradeConvergenceDatabaseIfNeeded$1(ConvergenceDatabaseInitializer.scala:466) ~[com.convergencelabs.convergence-server-1.0.0-rc.12.jar:1.0.0-rc.12]
    at scala.util.Success.flatMap(Try.scala:258) ~[org.scala-lang.scala-library-2.13.5.jar:?]
    at com.convergencelabs.convergence.server.backend.db.ConvergenceDatabaseInitializer.upgradeConvergenceDatabaseIfNeeded(ConvergenceDatabaseInitializer.scala:452) ~[com.convergencelabs.convergence-server-1.0.0-rc.12.jar:1.0.0-rc.12]
    at com.convergencelabs.convergence.server.backend.db.ConvergenceDatabaseInitializer.$anonfun$processExistingSchema$1(ConvergenceDatabaseInitializer.scala:80) ~[com.convergencelabs.convergence-server-1.0.0-rc.12.jar:1.0.0-rc.12]
    at scala.util.Success.flatMap(Try.scala:258) ~[org.scala-lang.scala-library-2.13.5.jar:?]
    at com.convergencelabs.convergence.server.backend.db.ConvergenceDatabaseInitializer.processExistingSchema(ConvergenceDatabaseInitializer.scala:79) ~[com.convergencelabs.convergence-server-1.0.0-rc.12.jar:1.0.0-rc.12]
    at com.convergencelabs.convergence.server.backend.db.ConvergenceDatabaseInitializer.$anonfun$assertInitialized$3(ConvergenceDatabaseInitializer.scala:69) ~[com.convergencelabs.convergence-server-1.0.0-rc.12.jar:1.0.0-rc.12]
    at com.convergencelabs.convergence.server.backend.db.ConvergenceDatabaseInitializer.$anonfun$assertInitialized$3$adapted(ConvergenceDatabaseInitializer.scala:68) ~[com.convergencelabs.convergence-server-1.0.0-rc.12.jar:1.0.0-rc.12]
    at scala.util.Success.flatMap(Try.scala:258) ~[org.scala-lang.scala-library-2.13.5.jar:?]
    at com.convergencelabs.convergence.server.backend.db.ConvergenceDatabaseInitializer.$anonfun$assertInitialized$2(ConvergenceDatabaseInitializer.scala:68) ~[com.convergencelabs.convergence-server-1.0.0-rc.12.jar:1.0.0-rc.12]
    at scala.util.Success.flatMap(Try.scala:258) ~[org.scala-lang.scala-library-2.13.5.jar:?]
    at com.convergencelabs.convergence.server.backend.db.ConvergenceDatabaseInitializer.assertInitialized(ConvergenceDatabaseInitializer.scala:67) ~[com.convergencelabs.convergence-server-1.0.0-rc.12.jar:1.0.0-rc.12]
    at com.convergencelabs.convergence.server.backend.services.server.ConvergenceDatabaseInitializerActor$.$anonfun$apply$3(ConvergenceDatabaseInitializerActor.scala:51) ~[com.convergencelabs.convergence-server-1.0.0-rc.12.jar:1.0.0-rc.12]
    at akka.actor.typed.internal.BehaviorImpl$ReceiveMessageBehavior.receive(BehaviorImpl.scala:153) ~[com.typesafe.akka.akka-actor-typed_2.13-2.6.15.jar:2.6.15]
    at akka.actor.typed.Behavior$.interpret(Behavior.scala:274) ~[com.typesafe.akka.akka-actor-typed_2.13-2.6.15.jar:2.6.15]
    at akka.actor.typed.Behavior$.interpretMessage(Behavior.scala:230) ~[com.typesafe.akka.akka-actor-typed_2.13-2.6.15.jar:2.6.15]
    at akka.actor.typed.internal.InterceptorImpl$$anon$2.apply(InterceptorImpl.scala:57) ~[com.typesafe.akka.akka-actor-typed_2.13-2.6.15.jar:2.6.15]
    at akka.actor.typed.internal.RestartSupervisor.aroundReceive(Supervision.scala:267) ~[com.typesafe.akka.akka-actor-typed_2.13-2.6.15.jar:2.6.15]
    at akka.actor.typed.internal.InterceptorImpl.receive(InterceptorImpl.scala:85) ~[com.typesafe.akka.akka-actor-typed_2.13-2.6.15.jar:2.6.15]
    at akka.actor.typed.Behavior$.interpret(Behavior.scala:274) ~[com.typesafe.akka.akka-actor-typed_2.13-2.6.15.jar:2.6.15]
    at akka.actor.typed.Behavior$.interpretMessage(Behavior.scala:230) ~[com.typesafe.akka.akka-actor-typed_2.13-2.6.15.jar:2.6.15]
    at akka.actor.typed.internal.InterceptorImpl$$anon$2.apply(InterceptorImpl.scala:57) ~[com.typesafe.akka.akka-actor-typed_2.13-2.6.15.jar:2.6.15]
    at akka.actor.typed.internal.InterceptorImpl.receive(InterceptorImpl.scala:87) ~[com.typesafe.akka.akka-actor-typed_2.13-2.6.15.jar:2.6.15]
    at akka.actor.typed.Behavior$.interpret(Behavior.scala:274) ~[com.typesafe.akka.akka-actor-typed_2.13-2.6.15.jar:2.6.15]
    at akka.actor.typed.Behavior$.interpretMessage(Behavior.scala:230) ~[com.typesafe.akka.akka-actor-typed_2.13-2.6.15.jar:2.6.15]
    at akka.actor.typed.internal.InterceptorImpl$$anon$2.apply(InterceptorImpl.scala:57) ~[com.typesafe.akka.akka-actor-typed_2.13-2.6.15.jar:2.6.15]
    at akka.actor.typed.internal.SimpleSupervisor.aroundReceive(Supervision.scala:123) ~[com.typesafe.akka.akka-actor-typed_2.13-2.6.15.jar:2.6.15]
    at akka.actor.typed.internal.InterceptorImpl.receive(InterceptorImpl.scala:85) ~[com.typesafe.akka.akka-actor-typed_2.13-2.6.15.jar:2.6.15]
    at akka.actor.typed.Behavior$.interpret(Behavior.scala:274) ~[com.typesafe.akka.akka-actor-typed_2.13-2.6.15.jar:2.6.15]
    at akka.actor.typed.Behavior$.interpretMessage(Behavior.scala:230) ~[com.typesafe.akka.akka-actor-typed_2.13-2.6.15.jar:2.6.15]
    at akka.actor.typed.internal.adapter.ActorAdapter.handleMessage(ActorAdapter.scala:131) ~[com.typesafe.akka.akka-actor-typed_2.13-2.6.15.jar:2.6.15]
    at akka.actor.typed.internal.adapter.ActorAdapter.aroundReceive(ActorAdapter.scala:107) ~[com.typesafe.akka.akka-actor-typed_2.13-2.6.15.jar:2.6.15]
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:580) ~[com.typesafe.akka.akka-actor_2.13-2.6.15.jar:2.6.15]
    at akka.actor.ActorCell.invoke(ActorCell.scala:548) ~[com.typesafe.akka.akka-actor_2.13-2.6.15.jar:2.6.15]
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270) ~[com.typesafe.akka.akka-actor_2.13-2.6.15.jar:2.6.15]
    at akka.dispatch.Mailbox.run(Mailbox.scala:231) ~[com.typesafe.akka.akka-actor_2.13-2.6.15.jar:2.6.15]
    at akka.dispatch.Mailbox.exec(Mailbox.scala:243) ~[com.typesafe.akka.akka-actor_2.13-2.6.15.jar:2.6.15]
    at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) [?:?]
    at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) [?:?]
    at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) [?:?]
    at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) [?:?]
    at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183) [?:?]
14:33:55 ERROR ConvergenceServer$ - There was a failure on server start up. Exiting.

And this is what I see on the orientdb logs after the connection:

2022-04-29 14:33:30:519 INFO  OrientDB Studio available at http://10.34.7.179:2480/studio/index.html [OServer]
2022-04-29 14:33:30:520 INFO  OrientDB Server is active v3.0.37 - Veloce (build 6a0e4724c10d51a0b19700fca46da8e41ae006f5, branch UNKNOWN). [OServer]
2022-04-29 14:33:52:819 INFO  Page size for WAL located in /orientdb/databases/convergence is set to 4096 bytes. [OCASDiskWriteAheadLog]
devarmada commented 2 years ago

Replying to my own issue because I found a solution. I deleted the convergence directory in the orientdb container and restarted the server. This time the database was correctly initialised.