Closed mamoedo closed 4 years ago
Were hitting the same timeout after updating from rc3 to 1.0. Unsure if related but we are also seeing the following before hitting the timeout.
[warn] o.t.s.u.Retry [|345a708b] An error occurs (Adding this property for key [_label] and value [Profile] violates a uniqueness constraint [ProfileName]), retrying (5)
[error] o.t.t.m.SchemaUpdaterActor [|] Database with TheHiveSchema schema update failure
org.thp.scalligraph.models.DatabaseException: Violation of database schema
at org.thp.scalligraph.janus.JanusDatabase$$anonfun$1.applyOrElse(JanusDatabase.scala:207)
at org.thp.scalligraph.janus.JanusDatabase$$anonfun$1.applyOrElse(JanusDatabase.scala:205)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:38)
at scala.util.Failure.recoverWith(Try.scala:236)
at org.thp.scalligraph.janus.JanusDatabase.tryTransaction(JanusDatabase.scala:205)
at org.thp.scalligraph.models.BaseDatabase.$anonfun$createSchemaFrom$1(Database.scala:205)
at scala.util.Success.flatMap(Try.scala:251)
at org.thp.scalligraph.models.BaseDatabase.createSchemaFrom(Database.scala:204)
at org.thp.scalligraph.models.Operations.execute(Operation.scala:53)
at org.thp.scalligraph.models.UpdatableSchema.update(Schema.scala:19)
Caused by: org.janusgraph.core.SchemaViolationException: Adding this property for key [_label] and value [Profile] violates a uniqueness constraint [ProfileName]
at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.addProperty(StandardJanusGraphTx.java:817)
at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.addProperty(StandardJanusGraphTx.java:745)
at org.janusgraph.graphdb.vertices.AbstractVertex.property(AbstractVertex.java:152)
at org.janusgraph.core.JanusGraphVertex.property(JanusGraphVertex.java:72)
at org.janusgraph.core.JanusGraphVertex.property(JanusGraphVertex.java:33)
at org.thp.scalligraph.models.BaseDatabase.$anonfun$setSingleProperty$1(Database.scala:330)
at scala.Option.foreach(Option.scala:407)
at org.thp.scalligraph.models.BaseDatabase.setSingleProperty(Database.scala:330)
at org.thp.scalligraph.janus.JanusDatabase.createVertex(JanusDatabase.scala:543)
at org.thp.scalligraph.models.InitialValue.create(Schema.scala:14)```
This errors occurs at each startup ?
This errors occurs at each startup ?
@To-om yes.
I tried several times and I got the same result
Can you enable debug by adding <logger name="org.thp" level="DEBUG" />
in the right place in /etc/thehive/logback.xml and provide me the logs ?
I changed it but I see almost the same logs. This is the few debug log at the beginning of starting thehive:
[debug] o.t.s.m.Database [|] Committing transaction
[debug] o.t.s.m.Database [|] End of transaction
[debug] o.t.t.m.SchemaUpdaterActor [|] Database open instances are: c0a870051268-3f069aa738e91
[info] a.c.s.ClusterSingletonProxy [|] Singleton identified at [akka://application/user/theHiveSchemaUpdaterSingletonManager/singleton]
[info] o.t.t.m.SchemaUpdaterActor [|] Database is ready to be updated
[debug] o.t.s.m.Database [|0649ab9f] Begin of readonly transaction
[debug] o.t.s.m.Database [|0649ab9f] End of readonly transaction
[debug] o.t.s.m.Database [|] Committing transaction
[debug] o.t.s.m.Database [|] End of transaction
[debug] o.t.s.m.Database [|] Committing transaction
[debug] o.t.s.m.Database [|] End of transaction
@mamoedo I can't reproduce this behavior. Try to start TheHive with a fresh database: change the cassandra keyspace in your configuration file (db.janusgraph.storage.keyspace: testKeyspace
in application.conf).
If TheHive works, it means that the problem is related to your data. Otherwise there is no reason I can't reproduce (then I'll investigate on you configuration).
@To-om changing the cassandra keyspace doesn't solve the problem.
TheHive is stuck showing this errors on the logfile, and the web interface isn't up. Cassandra is up.
Caused by: java.lang.IllegalArgumentException (same stack trace as error #24)
35) Error injecting constructor, java.lang.IllegalArgumentException: Could not instantiate implementation: org.janusgraph.diskstorage.cql.CQLStoreManager
at org.thp.scalligraph.janus.JanusDatabase.<init>(JanusDatabase.scala:71)
at org.thp.scalligraph.janus.JanusDatabase.class(JanusDatabase.scala:56)
while locating org.thp.scalligraph.janus.JanusDatabase
while locating org.thp.scalligraph.models.Database
for the 2nd parameter of org.thp.scalligraph.services.config.ApplicationConfig.<init>(ApplicationConfig.scala:20)
at org.thp.scalligraph.services.config.ApplicationConfig.class(ApplicationConfig.scala:19)
while locating org.thp.scalligraph.services.config.ApplicationConfig
for the 2nd parameter of org.thp.thehive.services.TOTPAuthSrvProvider.<init>(TOTPAuthSrv.scala:101)
at org.thp.thehive.services.TOTPAuthSrvProvider.class(TOTPAuthSrv.scala:100)
while locating org.thp.thehive.services.TOTPAuthSrvProvider
Caused by: java.lang.IllegalArgumentException: Could not instantiate implementation: org.janusgraph.diskstorage.cql.CQLStoreManager
at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:64)
at org.janusgraph.diskstorage.Backend.getImplementationClass(Backend.java:440)
at org.janusgraph.diskstorage.Backend.getStorageManager(Backend.java:411)
at org.janusgraph.graphdb.configuration.builder.GraphDatabaseConfigurationBuilder.build(GraphDatabaseConfigurationBuilder.java:50)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:161)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:132)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:112)
at org.thp.scalligraph.janus.JanusDatabase$.openDatabase(JanusDatabase.scala:51)
at org.thp.scalligraph.janus.JanusDatabase.<init>(JanusDatabase.scala:72)
at org.thp.scalligraph.janus.JanusDatabase$$FastClassByGuice$$113881e3.newInstance(<generated>)
at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89)
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:62)
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:213)
at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:184)
at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:111)
at com.google.inject.Guice.createInjector(Guice.java:87)
at com.google.inject.Guice.createInjector(Guice.java:78)
at play.api.inject.guice.GuiceBuilder.injector(GuiceInjectorBuilder.scala:200)
at play.api.inject.guice.GuiceApplicationBuilder.build(GuiceApplicationBuilder.scala:155)
at play.api.inject.guice.GuiceApplicationLoader.load(GuiceApplicationLoader.scala:21)
at play.core.server.ProdServerStart$.start(ProdServerStart.scala:54)
at play.core.server.ProdServerStart$.main(ProdServerStart.scala:30)
at play.core.server.ProdServerStart.main(ProdServerStart.scala)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedConstructorAccessor7.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:58)
... 44 more
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra-2/192.168.112.3:9042 (com.datastax.driver.core.exceptions.TransportException: [cassandra-2/192.168.112.3:9042] Cannot connect))
at com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:268)
at com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:107)
at com.datastax.driver.core.Cluster$Manager.negotiateProtocolVersionAndConnect(Cluster.java:1813)
at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1726)
at com.datastax.driver.core.Cluster.init(Cluster.java:214)
at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:387)
at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:366)
at com.datastax.driver.core.Cluster.connect(Cluster.java:311)
at org.janusgraph.diskstorage.cql.CQLStoreManager.initializeSession(CQLStoreManager.java:320)
at org.janusgraph.diskstorage.cql.CQLStoreManager.<init>(CQLStoreManager.java:182)
... 48 more
Caused by: java.lang.IllegalArgumentException (same stack trace as error #33)
46) Error injecting constructor, java.lang.IllegalArgumentException: Could not instantiate implementation: org.janusgraph.diskstorage.cql.CQLStoreManager
at org.thp.scalligraph.janus.JanusDatabase.<init>(JanusDatabase.scala:71)
at org.thp.scalligraph.janus.JanusDatabase.class(JanusDatabase.scala:56)
while locating org.thp.scalligraph.janus.JanusDatabase
while locating org.thp.scalligraph.models.Database
for the 1st parameter of org.thp.thehive.models.DatabaseProvider.<init>(SchemaUpdaterActor.scala:20)
at org.thp.thehive.models.DatabaseProvider.class(SchemaUpdaterActor.scala:19)
while locating org.thp.thehive.models.DatabaseProvider
while locating org.thp.scalligraph.models.Database annotated with @com.google.inject.name.Named(value=with-thehive-schema)
for the 1st parameter of org.thp.thehive.services.LocalUserSrv.<init>(LocalUserSrv.scala:18)
at org.thp.thehive.services.LocalUserSrv.class(LocalUserSrv.scala:17)
while locating org.thp.thehive.services.LocalUserSrv
Caused by: java.lang.IllegalArgumentException (same stack trace as error #27)
47) Error injecting constructor, java.lang.IllegalArgumentException: Could not instantiate implementation: org.janusgraph.diskstorage.cql.CQLStoreManager
at org.thp.scalligraph.janus.JanusDatabase.<init>(JanusDatabase.scala:71)
at org.thp.scalligraph.janus.JanusDatabase.class(JanusDatabase.scala:56)
while locating org.thp.scalligraph.janus.JanusDatabase
while locating org.thp.scalligraph.models.Database
for the 1st parameter of org.thp.thehive.models.DatabaseProvider.<init>(SchemaUpdaterActor.scala:20)
at org.thp.thehive.models.DatabaseProvider.class(SchemaUpdaterActor.scala:19)
while locating org.thp.thehive.models.DatabaseProvider
while locating org.thp.scalligraph.models.Database annotated with @com.google.inject.name.Named(value=with-thehive-schema)
for the 2nd parameter of org.thp.thehive.connector.cortex.models.DatabaseProvider.<init>(SchemaUpdaterActor.scala:18)
at org.thp.thehive.connector.cortex.models.DatabaseProvider.class(SchemaUpdaterActor.scala:17)
while locating org.thp.thehive.connector.cortex.models.DatabaseProvider
Caused by: java.lang.IllegalArgumentException (same stack trace as error #27)
48) Error injecting constructor, java.lang.IllegalArgumentException: Could not instantiate implementation: org.janusgraph.diskstorage.cql.CQLStoreManager
at org.thp.scalligraph.janus.JanusDatabase.<init>(JanusDatabase.scala:71)
at org.thp.scalligraph.janus.JanusDatabase.class(JanusDatabase.scala:56)
while locating org.thp.scalligraph.janus.JanusDatabase
while locating org.thp.scalligraph.models.Database
for the 2nd parameter of org.thp.scalligraph.services.config.ApplicationConfig.<init>(ApplicationConfig.scala:20)
at org.thp.scalligraph.services.config.ApplicationConfig.class(ApplicationConfig.scala:19)
while locating org.thp.scalligraph.services.config.ApplicationConfig
for the 2nd parameter of org.thp.thehive.services.TOTPAuthSrvProvider.<init>(TOTPAuthSrv.scala:101)
at org.thp.thehive.services.TOTPAuthSrvProvider.class(TOTPAuthSrv.scala:100)
while locating org.thp.thehive.services.TOTPAuthSrvProvider
while locating org.thp.scalligraph.auth.AuthSrv
for the 1st parameter of org.thp.scalligraph.controllers.Entrypoint.<init>(Entrypoint.scala:27)
at org.thp.scalligraph.controllers.Entrypoint.class(Entrypoint.scala:26)
while locating org.thp.scalligraph.controllers.Entrypoint
for the 1st parameter of org.thp.thehive.connector.cortex.controllers.v0.ResponderCtrl.<init>(ResponderCtrl.scala:17)
at org.thp.thehive.connector.cortex.controllers.v0.ResponderCtrl.class(ResponderCtrl.scala:16)
while locating org.thp.thehive.connector.cortex.controllers.v0.ResponderCtrl
Caused by: java.lang.IllegalArgumentException (same stack trace as error #24)
49) Error injecting constructor, java.lang.IllegalArgumentException: Could not instantiate implementation: org.janusgraph.diskstorage.cql.CQLStoreManager
at org.thp.scalligraph.janus.JanusDatabase.<init>(JanusDatabase.scala:71)
at org.thp.scalligraph.janus.JanusDatabase.class(JanusDatabase.scala:56)
while locating org.thp.scalligraph.janus.JanusDatabase
while locating org.thp.scalligraph.models.Database
for the 2nd parameter of org.thp.scalligraph.services.config.ApplicationConfig.<init>(ApplicationConfig.scala:20)
at org.thp.scalligraph.services.config.ApplicationConfig.class(ApplicationConfig.scala:19)
while locating org.thp.scalligraph.services.config.ApplicationConfig
for the 2nd parameter of org.thp.thehive.services.TOTPAuthSrvProvider.<init>(TOTPAuthSrv.scala:101)
at org.thp.thehive.services.TOTPAuthSrvProvider.class(TOTPAuthSrv.scala:100)
while locating org.thp.thehive.services.TOTPAuthSrvProvider
while locating org.thp.scalligraph.auth.AuthSrv
for the 1st parameter of org.thp.scalligraph.controllers.Entrypoint.<init>(Entrypoint.scala:27)
at org.thp.scalligraph.controllers.Entrypoint.class(Entrypoint.scala:26)
while locating org.thp.scalligraph.controllers.Entrypoint
for the 1st parameter of org.thp.thehive.connector.cortex.controllers.v0.AnalyzerCtrl.<init>(AnalyzerCtrl.scala:15)
at org.thp.thehive.connector.cortex.controllers.v0.AnalyzerCtrl.class(AnalyzerCtrl.scala:14)
while locating org.thp.thehive.connector.cortex.controllers.v0.AnalyzerCtrl
Caused by: java.lang.IllegalArgumentException (same stack trace as error #24)
49 errors
at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:554)
at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:188)
at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:111)
at com.google.inject.Guice.createInjector(Guice.java:87)
at com.google.inject.Guice.createInjector(Guice.java:78)
at play.api.inject.guice.GuiceBuilder.injector(GuiceInjectorBuilder.scala:200)
at play.api.inject.guice.GuiceApplicationBuilder.build(GuiceApplicationBuilder.scala:155)
at play.api.inject.guice.GuiceApplicationLoader.load(GuiceApplicationLoader.scala:21)
at play.core.server.ProdServerStart$.start(ProdServerStart.scala:54)
at play.core.server.ProdServerStart$.main(ProdServerStart.scala:30)
at play.core.server.ProdServerStart.main(ProdServerStart.scala)
[info] a.c.s.ClusterSingletonProxy [|] Singleton identified at [akka://application/user/flowSingletonManager/singleton]
Can you confirm that the server on which TheHive is installed can connect to Cassandra ? (check Firewall rules, SElinux/AppArmor policy). Have you configured authentication or SSL for Cassandra ?
@To-om sorry, there wasn't connectivity, TheHive started before Cassandra was fully up, it happens sometimes in my setup.
So I did the testing that you asked me and TheHive is up and running correctly with a new keyspace. I can't see what could be wrong with my data then :( I just migrated from TheHive3 to TheHive4 using the tool. I think my data is too large to check in time for the configured timeouts.
The first start of TheHive (after a migration) can be a little bit long. It indexes data. I was supposing that 5 minutes is enough but if your data is large, you can get a timeout. I can make this timeout configurable.
I think that the debug log level is still disable in your setup. I don't know how you start TheHive but can you check that it is started with the parameter -Dlogger.file=/path/to/logback.xml
? (cf. previous comment). Then you should have more logs with more explaination.
@To-om in my setup it's a bit difficult to edit the logback.xml, because it isn't located in /etc/thehive/logback.xml
but in /opt/thehive/conf/logback.xml
instead (the same applies to logback-migration.xml, which is required to start the migration but it's not in /etc/thehive/
by default.) so I have to manually copy the file and edit it every time I deploy my environment. I enabled it in order to test what you asked me, but it's true that I don't have it enabled by default in my setup.
I was supposing that 5 minutes is enough but if your data is large, you can get a timeout. I can make this timeout configurable.
Thanks! It would help a lot to be able to configure it.
Request Type
Bug
Work Environment
Problem Description
TheHive4 crashes when starting with a not empty database. It makes some checks that seem to be slower than the configured timeout when the database is not empty (or it has few data)
Data size:
Steps to Reproduce
Possible Solutions
Increase the timeout. Although it would be nice to increase the checks speed.
Complementary information