search

TheHive-Project / TheHive

TheHive: a Scalable, Open Source and Free Security Incident Response Platform
https://thehive-project.org
GNU Affero General Public License v3.0
3.36k stars 613 forks source link

[Bug] Authentication Failures and Janugraph indexes stuck in INSTALLED state prevent TheHive from starting up #2080

Open domenicoxs opened 3 years ago

domenicoxs commented 3 years ago

Request Type

Bug

Work Environment

Question Answer
OS version (server) Ubuntu
Virtualized Env. True
Dedicated RAM 12 GB
vCPU 16
TheHive version / git hash 4.1.4-1
Package Type DEB
Database Cassandra
Index type Lucene
Attachments storage Local

Problem Description

Hi everyone, today I wasn't able to access to thehive anymore (Authentication Failure error) so I tried to restart both Cassandra service and thehive. Thehive doesn't start anymore, below the log:

15:26:25.007 [main] INFO ScalligraphApplication - Loading application ...
[info] o.t.s.ScalligraphModule [|] Loading scalligraph module
[info] a.e.s.Slf4jLogger [|] Slf4jLogger started
[info] a.r.a.t.ArteryTcpTransport [|] Remoting started with transport [Artery tcp]; listening on address [akka://application@127.0.0.1:39983] with UID [5432962275019404142]
[info] a.c.Cluster [|] Cluster Node [akka://application@127.0.0.1:39983] - Starting up, Akka version [2.6.10] ...
[info] a.c.Cluster [|] Cluster Node [akka://application@127.0.0.1:39983] - Registered cluster JMX MBean [akka:type=Cluster]
[info] a.c.Cluster [|] Cluster Node [akka://application@127.0.0.1:39983] - Started up successfully
[info] a.c.s.SplitBrainResolver [|] SBR started. Config: strategy [KeepMajority], stable-after [20 seconds], down-all-when-unstable [15 seconds], selfUniqueAddress [akka://application@127.0.0.1:39983#5432962275019404142], selfDc [default].
[info] a.c.Cluster [|] Cluster Node [akka://application@127.0.0.1:39983] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining
[info] o.r.Reflections [|] Reflections took 358 ms to scan 1 urls, producing 160 keys and 2421 values
[info] o.t.t.ClusterSetup [|] Initialising cluster
[info] a.c.Cluster [|] Cluster Node [akka://application@127.0.0.1:39983] - Node [akka://application@127.0.0.1:39983] is JOINING itself (with roles [dc-default], version [0.0.0]) and forming new cluster
[info] a.c.Cluster [|] Cluster Node [akka://application@127.0.0.1:39983] - is the new leader among reachable nodes (more leaders may exist)
[info] a.c.Cluster [|] Cluster Node [akka://application@127.0.0.1:39983] - Leader is moving node [akka://application@127.0.0.1:39983] to [Up]
[info] o.t.t.ClusterListener [|] Member is Up: akka://application@127.0.0.1:39983
[info] a.c.s.SplitBrainResolver [|] This node is now the leader responsible for taking SBR decisions among the reachable nodes (more leaders may exist).
[info] a.c.s.ClusterSingletonManager [|] Singleton manager starting singleton actor [akka://application/system/singletonManagerJanusGraphClusterLeader/JanusGraphClusterLeader]
[info] a.c.s.ClusterSingletonManager [|] ClusterSingletonManager state change [Start -> Oldest]
[debug] o.t.s.j.JanusClusterManagerActor$$anonfun$waitForPeerStatus$1 [|] Cancel timer [TimerKey] with generation [1]
[info] a.c.s.ClusterSingletonProxy [|] Singleton identified at [akka://application/system/singletonManagerJanusGraphClusterLeader/JanusGraphClusterLeader]
[info] c.d.driver.core [|] DataStax Java driver 3.9.0 for Apache Cassandra
[info] c.d.d.c.GuavaCompatibility [|] Detected Guava >= 19 in the classpath, using modern compatibility layer
[info] c.d.d.c.ClockFactory [|] Using native clock to generate timestamps.
[info] c.d.d.c.NettyUtil [|] Found Netty's native epoll transport in the classpath, using it
[info] c.d.d.c.p.DCAwareRoundRobinPolicy [|] Using data-center name 'datacenter1' for DCAwareRoundRobinPolicy (if this is incorrect, please provide the correct datacenter name with DCAwareRoundRobinPolicy constructor)
[info] c.d.d.c.Cluster [|] New Cassandra host /127.0.0.1:9042 added
[info] o.j.c.u.ReflectiveConfigOptionLoader [|] Loaded and initialized config classes: 8 OK out of 13 attempts in PT0.045S
[info] o.j.g.i.UniqueInstanceIdRetriever [|] Generated unique-instance-id=7f00000118261-TH_TA_PROD1
[info] c.d.d.c.ClockFactory [|] Using native clock to generate timestamps.
[info] c.d.d.c.p.DCAwareRoundRobinPolicy [|] Using data-center name 'datacenter1' for DCAwareRoundRobinPolicy (if this is incorrect, please provide the correct datacenter name with DCAwareRoundRobinPolicy constructor)
[info] c.d.d.c.Cluster [|] New Cassandra host /127.0.0.1:9042 added
[info] o.j.d.Backend [|] Configuring index [search]
[info] o.j.d.Backend [|] Initiated backend operations thread pool of size 8
[info] o.j.d.Backend [|] Configuring total store cache size: 814617379
[info] o.j.d.l.k.KCVSLog [|] Loaded unidentified ReadMarker start time 2021-06-10T15:26:39.952Z into org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller@469372d1
[info] o.t.s.j.JanusDatabase [|] Full-text index is available (lucene:/opt/thp/thehive/index) single node
[debug] o.t.s.m.Database [|517f23ac] Begin of transaction
[debug] o.t.s.m.Database [|517f23ac] Get version of thehive
[debug] o.t.s.m.Database [|517f23ac] Committing transaction
[debug] o.t.s.m.Database [|517f23ac] End of transaction
[debug] o.t.s.m.Database [|0814dbf7] Begin of transaction
[debug] o.t.s.m.Database [|0814dbf7] Get version of thehive
[debug] o.t.s.m.Database [|0814dbf7] Committing transaction
[debug] o.t.s.m.Database [|0814dbf7] End of transaction
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model UserConfig
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model TaskUser
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model Data
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model CaseTag
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model ImpactStatus
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model Pattern
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model TaskLog
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model Taxonomy
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model TaxonomyTag
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model AlertCustomField
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model Audit
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model ShareTask
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model CaseTemplateTask
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model ObservableKeyValue
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model PatternPattern
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model LogAttachment
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model CaseTemplateOrganisation
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model ObservableType
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model CaseImpactStatus
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model ReportTag
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model ObservableObservableType
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model CaseTemplateTag
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model AlertCaseTemplate
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model AuditUser
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model CaseTemplate
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model Role
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model RoleOrganisation
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model UserAttachment
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model KeyValue
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model DashboardUser
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model OrganisationConfig
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model ObservableData
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model Dashboard
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model User
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model CaseUser
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model Task
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model AlertTag
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model Tag
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model Config
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model ResolutionStatus
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model ObservableTag
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model CaseResolutionStatus
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model MergedFrom
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model Share
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model ProcedurePattern
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model OrganisationDashboard
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model AlertCase
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model Procedure
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model AuditContext
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model CaseCustomField
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model ShareCase
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model CustomField
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model CaseProcedure
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model Observable
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model ObservableReportTag
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model ObservableAttachment
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model CaseTemplateCustomField
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model ShareProfile
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model OrganisationPage
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model Profile
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model Case
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model UserRole
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model AlertObservable
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model Alert
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model OrganisationTaxonomy
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model Log
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model OrganisationShare
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model ShareObservable
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model RoleProfile
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model OrganisationOrganisation
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model Attachment
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model CaseCaseTemplate
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model AlertOrganisation
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model Organisation
[info] o.t.t.m.TheHiveSchemaDefinition [|] Loading model Page
[debug] o.t.s.m.Database [|mgmt-7f6719c6] Begin of management transaction
[debug] o.t.s.m.Database [|mgmt-7f6719c6] Committing transaction
[debug] o.t.s.m.Database [|mgmt-7f6719c6] End of transaction
[debug] o.t.s.m.Database [|mgmt-6e2c4f66] Begin of management transaction
[debug] o.t.s.m.Database [|mgmt-6e2c4f66] Committing transaction
[debug] o.t.s.m.Database [|mgmt-6e2c4f66] End of transaction
[info] o.t.s.m.Database [|] Wait for the index global to become available
[info] o.j.g.d.m.GraphIndexStatusWatcher [|] Some key(s) on index global do not currently have status(es) [REGISTERED, ENABLED]: date=INSTALLED,organisationId=INSTALLED,data=INSTALLED,endDate=INSTALLED,dueDate=INSTALLED,sighted=INSTALLED,_createdBy=INSTALLED,source=INSTALLED,type=INSTALLED,objectType=INSTALLED,predicate=INSTALLED,caseId=INSTALLED,action=INSTALLED,attachmentId=INSTALLED,contentType=INSTALLED,pap=INSTALLED,order=INSTALLED,group=INSTALLED,read=INSTALLED,caseTemplate=INSTALLED,dataType=INSTALLED,lastSyncDate=INSTALLED,tags=INSTALLED,relatedId=INSTALLED,size=INSTALLED,resolutionStatus=INSTALLED,name=INSTALLED,hashes=INSTALLED,assignee=INSTALLED,sourceRef=INSTALLED,startDate=INSTALLED,impactStatus=INSTALLED,status=INSTALLED,ignoreSimilarity=INSTALLED,flag=INSTALLED,description=INSTALLED,title=INSTALLED,_label=INSTALLED,organisationIds=INSTALLED,requestId=INSTALLED,_createdAt=INSTALLED,_updatedAt=INSTALLED,value=INSTALLED,objectId=INSTALLED,mainAction=INSTALLED,severity=INSTALLED,summary=INSTALLED,_updatedBy=INSTALLED,message=INSTALLED,follow=INSTALLED,colour=INSTALLED,namespace=INSTALLED,tlp=INSTALLED,ioc=INSTALLED,taskId=INSTALLED
[info] o.j.g.d.m.GraphIndexStatusWatcher [|] Some key(s) on index global do not currently have status(es) [REGISTERED, ENABLED]: date=INSTALLED,organisationId=INSTALLED,data=INSTALLED,endDate=INSTALLED,dueDate=INSTALLED,sighted=INSTALLED,_createdBy=INSTALLED,source=INSTALLED,type=INSTALLED,objectType=INSTALLED,predicate=INSTALLED,caseId=INSTALLED,action=INSTALLED,attachmentId=INSTALLED,contentType=INSTALLED,pap=INSTALLED,order=INSTALLED,group=INSTALLED,read=INSTALLED,caseTemplate=INSTALLED,dataType=INSTALLED,lastSyncDate=INSTALLED,tags=INSTALLED,relatedId=INSTALLED,size=INSTALLED,resolutionStatus=INSTALLED,name=INSTALLED,hashes=INSTALLED,assignee=INSTALLED,sourceRef=INSTALLED,startDate=INSTALLED,impactStatus=INSTALLED,status=INSTALLED,ignoreSimilarity=INSTALLED,flag=INSTALLED,description=INSTALLED,title=INSTALLED,_label=INSTALLED,organisationIds=INSTALLED,requestId=INSTALLED,_createdAt=INSTALLED,_updatedAt=INSTALLED,value=INSTALLED,objectId=INSTALLED,mainAction=INSTALLED,severity=INSTALLED,summary=INSTALLED,_updatedBy=INSTALLED,message=INSTALLED,follow=INSTALLED,colour=INSTALLED,namespace=INSTALLED,tlp=INSTALLED,ioc=INSTALLED,taskId=INSTALLED
[info] o.j.g.d.m.GraphIndexStatusWatcher [|] Some key(s) on index global do not currently have status(es) [REGISTERED, ENABLED]: date=INSTALLED,organisationId=INSTALLED,data=INSTALLED,endDate=INSTALLED,dueDate=INSTALLED,sighted=INSTALLED,_createdBy=INSTALLED,source=INSTALLED,type=INSTALLED,objectType=INSTALLED,predicate=INSTALLED,caseId=INSTALLED,action=INSTALLED,attachmentId=INSTALLED,contentType=INSTALLED,pap=INSTALLED,order=INSTALLED,group=INSTALLED,read=INSTALLED,caseTemplate=INSTALLED,dataType=INSTALLED,lastSyncDate=INSTALLED,tags=INSTALLED,relatedId=INSTALLED,size=INSTALLED,resolutionStatus=INSTALLED,name=INSTALLED,hashes=INSTALLED,assignee=INSTALLED,sourceRef=INSTALLED,startDate=INSTALLED,impactStatus=INSTALLED,status=INSTALLED,ignoreSimilarity=INSTALLED,flag=INSTALLED,description=INSTALLED,title=INSTALLED,_label=INSTALLED,organisationIds=INSTALLED,requestId=INSTALLED,_createdAt=INSTALLED,_updatedAt=INSTALLED,value=INSTALLED,objectId=INSTALLED,mainAction=INSTALLED,severity=INSTALLED,summary=INSTALLED,_updatedBy=INSTALLED,message=INSTALLED,follow=INSTALLED,colour=INSTALLED,namespace=INSTALLED,tlp=INSTALLED,ioc=INSTALLED,taskId=INSTALLED
[info] o.j.g.d.m.GraphIndexStatusWatcher [|] Some key(s) on index global do not currently have status(es) [REGISTERED, ENABLED]: date=INSTALLED,organisationId=INSTALLED,data=INSTALLED,endDate=INSTALLED,dueDate=INSTALLED,sighted=INSTALLED,_createdBy=INSTALLED,source=INSTALLED,type=INSTALLED,objectType=INSTALLED,predicate=INSTALLED,caseId=INSTALLED,action=INSTALLED,attachmentId=INSTALLED,contentType=INSTALLED,pap=INSTALLED,order=INSTALLED,group=INSTALLED,read=INSTALLED,caseTemplate=INSTALLED,dataType=INSTALLED,lastSyncDate=INSTALLED,tags=INSTALLED,relatedId=INSTALLED,size=INSTALLED,resolutionStatus=INSTALLED,name=INSTALLED,hashes=INSTALLED,assignee=INSTALLED,sourceRef=INSTALLED,startDate=INSTALLED,impactStatus=INSTALLED,status=INSTALLED,ignoreSimilarity=INSTALLED,flag=INSTALLED,description=INSTALLED,title=INSTALLED,_label=INSTALLED,organisationIds=INSTALLED,requestId=INSTALLED,_createdAt=INSTALLED,_updatedAt=INSTALLED,value=INSTALLED,objectId=INSTALLED,mainAction=INSTALLED,severity=INSTALLED,summary=INSTALLED,_updatedBy=INSTALLED,message=INSTALLED,follow=INSTALLED,colour=INSTALLED,namespace=INSTALLED,tlp=INSTALLED,ioc=INSTALLED,taskId=INSTALLED
[info] o.j.g.d.m.GraphIndexStatusWatcher [|] Some key(s) on index global do not currently have status(es) [REGISTERED, ENABLED]: date=INSTALLED,organisationId=INSTALLED,data=INSTALLED,endDate=INSTALLED,dueDate=INSTALLED,sighted=INSTALLED,_createdBy=INSTALLED,source=INSTALLED,type=INSTALLED,objectType=INSTALLED,predicate=INSTALLED,caseId=INSTALLED,action=INSTALLED,attachmentId=INSTALLED,contentType=INSTALLED,pap=INSTALLED,order=INSTALLED,group=INSTALLED,read=INSTALLED,caseTemplate=INSTALLED,dataType=INSTALLED,lastSyncDate=INSTALLED,tags=INSTALLED,relatedId=INSTALLED,size=INSTALLED,resolutionStatus=INSTALLED,name=INSTALLED,hashes=INSTALLED,assignee=INSTALLED,sourceRef=INSTALLED,startDate=INSTALLED,impactStatus=INSTALLED,status=INSTALLED,ignoreSimilarity=INSTALLED,flag=INSTALLED,description=INSTALLED,title=INSTALLED,_label=INSTALLED,organisationIds=INSTALLED,requestId=INSTALLED,_createdAt=INSTALLED,_updatedAt=INSTALLED,value=INSTALLED,objectId=INSTALLED,mainAction=INSTALLED,severity=INSTALLED,summary=INSTALLED,_updatedBy=INSTALLED,message=INSTALLED,follow=INSTALLED,colour=INSTALLED,namespace=INSTALLED,tlp=INSTALLED,ioc=INSTALLED,taskId=INSTALLED

From what I think I've understood from the following reference there must be a transaction on Jenusgraph which is holding thehive from starting up

https://groups.google.com/g/janusgraph-users/c/GfYM538KsTo

Unfortunately my knowledge about the platform, the architecture and the underlying technology is very limited so I wasn't able to login inside Janusgraph (what configuration file should I specify to connect to it?) inside the gremlin console to rollback the eventual transaction that was causing the issue.

Can you please help me to restore the platform without losing all the data?

nadouani commented 3 years ago

it says Wait for the index global to become available how much time did this process take? Was it working before? How did you move to 4.1.4? is it a migration from version 3 or an upgrade?

domenicoxs commented 3 years ago

Hi Nabil, thanks for the reply. There was no migration, it was installed from scratch and it was working fine before. After waiting patiently the process eventually started, however the previous issue (Authentication Failure) is still there, I tried to login with a user, with api and with admin account I get the same result. Attached the full log full_log.txt

nadouani commented 3 years ago

the indexation doesn't alter the data itself, are you sure of your credentials?

domenicoxs commented 3 years ago

Yes I am, I tried every pair of credentials. It's very strange that also the superadmin ones are not working. Actually I had the same issue in another 4.x instance inside a docker container, but I didn't mind and created another brand new one. I guess something have corrupted somehow the data. Is there any method for resetting/admin superadmin credentials like in version 3?

domenicoxs commented 3 years ago

Any hint on this?