search

TheHive-Project / Cortex

Cortex: a Powerful Observable Analysis and Active Response Engine
https://thehive-project.org
GNU Affero General Public License v3.0
1.32k stars 222 forks source link

[Question] Cortex Error Connecting to Elastic SSL Authentication #358

Closed Canon88 closed 3 years ago

Canon88 commented 3 years ago

Work Environment

Question Answer
OS version (server) Ubuntu
Cortex version / git hash 3.1.1-1
Package Type Docker

Problem Description

I already had an Elastic cluster, so I used Docker to deploy Cortex without Elastic, but I had some problems with it. my Elastic cluster had certificate authentication enabled, and I tried to modify the configuration of the Elastic certificate in Cortex, and the following error was reported at startup.

One note, since my Elastic cluster is certified with ca certificates, I see that Cortex must use JKS certificates. So I did the conversion, is this the right action?

$ keytool -importcert -trustcacerts -file ca.crt -keystore ca.jks

Elastic Config

# XPack
## Transport layer
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.key: /etc/elasticsearch/certs/elasticsearch/elasticsearch.key
xpack.security.transport.ssl.certificate: /etc/elasticsearch/certs/elasticsearch/elasticsearch.crt
xpack.security.transport.ssl.certificate_authorities: [ "/etc/elasticsearch/certs/ca/ca.crt" ]
## HTTP layer
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.verification_mode: certificate
xpack.security.http.ssl.key: /etc/elasticsearch/certs/elasticsearch/elasticsearch.key
xpack.security.http.ssl.certificate: /etc/elasticsearch/certs/elasticsearch/elasticsearch.crt
xpack.security.http.ssl.certificate_authorities: [ "/etc/elasticsearch/certs/ca/ca.crt" ]
## Authentication for Elastic
xpack.security.enabled: true

  1. This is the error reported after docker startup

    cortex_1  | [warn] o.t.c.s.JobRunnerSrv - The package cortexutils for python hasn't been found
cortex_1  | [warn] o.t.c.s.JobRunnerSrv - The package cortexutils for python2 hasn't been found
cortex_1  | [warn] o.t.c.s.JobRunnerSrv - The package cortexutils for python3 hasn't been found
cortex_1  | [info] c.s.e.h.JavaClient$ - Creating HTTP client on https://elasticsearch01:9200
cortex_1  | [info] play.api.Play - Application started (Prod) (no global state)
cortex_1  | [error] o.e.d.DBConfiguration - ElasticSearch request failure: POST:/cortex_6/_search?scroll=60000ms
cortex_1  | StringEntity({"seq_no_primary_term":"true","query":{"bool":{"must":[{"term":{"relations":{"value":"job"}}},{"term":{"status":{"value":"Waiting"}}}]}},"from":0,"sort":[{"_doc":{"order":"desc"}}]},Some(application/json))
cortex_1  |  => ElasticError(security_exception,missing authentication credentials for REST request [/cortex_6/_search?scroll=60000ms],None,None,None,List(ElasticError(security_exception,missing authentication credentials for REST request [/cortex_6/_search?scroll=60000ms],None,None,None,null,None,None,None,List())),None,None,None,List())
cortex_1  | [warn] o.e.d.SearchWithScroll - Search error
cortex_1  | org.elastic4play.InternalError: Unknown error: ElasticError(security_exception,missing authentication credentials for REST request [/cortex_6/_search?scroll=60000ms],None,None,None,List(ElasticError(security_exception,missing authentication credentials for REST request [/cortex_6/_search?scroll=60000ms],None,None,None,null,None,None,None,List())),None,None,None,List())
cortex_1  |     at org.elastic4play.database.DBConfiguration.$anonfun$execute$2(DBConfiguration.scala:158)
cortex_1  |     at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:307)
cortex_1  |     at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:41)
cortex_1  |     at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
cortex_1  |     at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:56)
cortex_1  |     at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:93)
cortex_1  |     at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
cortex_1  |     at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85)
cortex_1  |     at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:93)
cortex_1  |     at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:48)
cortex_1  | [info] o.t.c.s.WorkerSrv - New worker list:
cortex_1  |
cortex_1  |     IPVoid 1.0
cortex_1  |     OpenCTI_SearchObservable 1.0
cortex_1  |     SEKOIAIntelligenceCenter_Indicators 1.0
cortex_1  |     SEKOIAIntelligenceCenter_Context 1.0
cortex_1  |     HIBP_Query 2.0
cortex_1  |     DNSSinkhole 1.0
cortex_1  |     DomainToolsIris_Investigate 1.0
cortex_1  |     Cyberprotect_ThreatScore 1.0
cortex_1  |     Autofocus_SearchJSON 1.0
cortex_1  |     DomainTools_Reputation 2.0
cortex_1  |     VirusTotal_GetReport 3.0
cortex_1  |     AMPforEndpoints_SCDAdd 1.0
cortex_1  |     MaxMind_GeoIP 4.0
cortex_1  |     Crowdstrike_Falcon_Custom_IOC_API 1.0
cortex_1  |     FireEyeiSight 1.0
cortex_1  |     Malwares_GetReport 1.0
cortex_1  |     Mnemonic_pDNS_Public 3.0
cortex_1  |     DomainTools_Risk 2.0
cortex_1  |     PassiveTotal_Osint 2.0
cortex_1  |     CIRCLPassiveDNS 2.0
cortex_1  |     CyberChef_FromHex 1.0
cortex_1  |     PassiveTotal_Passive_Dns 2.1
cortex_1  |     Shodan_Host 1.0
cortex_1  |     SendGrid 1.0
cortex_1  |     DomainTools_WhoisLookupUnparsed 2.0
cortex_1  |     PassiveTotal_Host_Pairs 2.0
cortex_1  |     Hunterio_DomainSearch 1.0
cortex_1  |     CyberChef_FromCharCode 1.0
cortex_1  |     MISPWarningLists 2.0
cortex_1  |     DomainTools_ReverseIPWhois 2.0
cortex_1  |     AbuseIPDB 1.0
cortex_1  |     TorProject 1.0
cortex_1  |     Redmine_Issue 1.0
cortex_1  |     CIRCLPassiveSSL 2.0
cortex_1  |     Fortiguard_URLCategory 2.1
cortex_1  |     Splunk_Search_User_Agent 3.0
cortex_1  |     Yara 2.0
cortex_1  |     EmergingThreats_DomainInfo 1.0
cortex_1  |     DNSDB_DomainName 2.0
cortex_1  |     PhishTank_CheckURL 2.1
cortex_1  |     DNS-RPZ 1.0
cortex_1  |     IPinfo_Hosted_Domains 1.0
cortex_1  |     SpamhausDBL 1.0
cortex_1  |     PassiveTotal_Trackers 2.0
cortex_1  |     ThreatResponse 1.0
cortex_1  |     FileInfo 7.0
cortex_1  |     Maltiverse_Report 1.0
cortex_1  |     BackscatterIO_GetObservations 1.0
cortex_1  |     OTXQuery 2.0
cortex_1  |     Investigate_Sample 1.0
cortex_1  |     MetaDefenderCloud_Reputation 1.0
cortex_1  |     Autofocus_SearchIOC 1.0
cortex_1  |     Splunk_Search_Mail_Email 3.0
cortex_1  |     LastInfoSec 1.0
cortex_1  |     Patrowl_GetReport 1.0
cortex_1  |     NSRL 1.0
cortex_1  |     AMPforEndpoints_MoveGUID 1.0
cortex_1  |     RT4-CreateTicket 1.0
cortex_1  |     PhishingInitiative_Scan 1.0
cortex_1  |     Mailer 1.0
cortex_1  |     C1fApp 1.0
cortex_1  |     RecordedFuture_risk 1.0
cortex_1  |     Nessus 2.0
cortex_1  |     KnowBe4 1.0
cortex_1  |     SecurityTrails_Passive_DNS 1.0
cortex_1  |     JoeSandbox_File_Analysis_Inet 2.0
cortex_1  |     Virusshare 2.0
cortex_1  |     Velociraptor_Flow 0.1
cortex_1  |     GreyNoise 2.3
cortex_1  |     DomainTools_ReverseIP 2.0
cortex_1  |     Yeti 1.0
cortex_1  |     StaxxSearch 1.0
cortex_1  |     SinkDB 1.1
cortex_1  |     MalwareBazaar 1.0
cortex_1  |     DomainToolsIris_AddRiskyDNSTag 1.0
cortex_1  |     Robtex_Forward_PDNS_Query 1.0
cortex_1  |     WOT_Lookup 2.0
cortex_1  |     Splunk_Search_Hash 3.0
cortex_1  |     Autofocus_GetSampleAnalysis 1.0
cortex_1  |     Virustotal_Downloader 0.1
cortex_1  |     VirusTotal_Scan 3.0
cortex_1  |     EmergingThreats_IPInfo 1.0
cortex_1  |     Shodan_ReverseDNS 1.0
cortex_1  |     Shodan_Host_History 1.0
cortex_1  |     Wazuh 1.0
cortex_1  |     PassiveTotal_Whois_Details 2.0
cortex_1  |     Urlscan.io_Search 0.1.1
cortex_1  |     DomainTools_WhoisLookup 2.0
cortex_1  |     ZEROFOX_Close_alert 1.0
cortex_1  |     Minemeld 1.0
cortex_1  |     PassiveTotal_Malware 2.0
cortex_1  |     DomainTools_ReverseNameServer 2.0
cortex_1  |     IntezerCommunity 1.0
cortex_1  |     DNSDB_IPHistory 2.0
cortex_1  |     GoogleSafebrowsing 2.0
cortex_1  |     PassiveTotal_Enrichment 2.0
cortex_1  |     PayloadSecurity_File_Analysis 1.0
cortex_1  |     Msg_Parser 3.0
cortex_1  |     DomainMailSPFDMARC_Analyzer 1.1
cortex_1  |     PassiveTotal_Unique_Resolutions 2.0
cortex_1  |     Splunk_Search_User 3.0
cortex_1  |     CuckooSandbox_Url_Analysis 1.2
cortex_1  |     BackscatterIO_Enrichment 1.0
cortex_1  |     Hashdd_Detail 1.0
cortex_1  |     DomainTools_ReverseWhois 2.0
cortex_1  |     Threatcrowd 1.0
cortex_1  |     Umbrella_Blacklister 1.1
cortex_1  |     ZEROFOX_Takedown_request 1.0
cortex_1  |     CyberCrime-Tracker 1.0
cortex_1  |     EmailRep 1.0
cortex_1  |     URLhaus 2.0
cortex_1  |     MISP 2.1
cortex_1  |     TeamCymruMHR 1.0
cortex_1  |     Hashdd_Status 1.0
cortex_1  |     DShield_lookup 1.0
cortex_1  |     EmergingThreats_MalwareInfo 1.0
cortex_1  |     StopForumSpam 1.0
cortex_1  |     DomainTools_HostingHistory 2.0
cortex_1  |     CyberChef_FromBase64 1.0
cortex_1  |     Abuse_Finder 3.0
cortex_1  |     Investigate_Categorization 1.0
cortex_1  |     SecurityTrails_Whois 1.0
cortex_1  |     DomainTools_WhoisHistory 2.0
cortex_1  |     MetaDefenderCloud_Scan 1.0
cortex_1  |     PassiveTotal_Ssl_Certificate_History 2.0
cortex_1  |     Splunk_Search_Other 3.0
cortex_1  |     Malpedia 1.0
cortex_1  |     MetaDefenderCore_Scan 1.0
cortex_1  |     Splunk_Search_Registry 3.0
cortex_1  |     Crt_sh_Transparency_Logs 1.0
cortex_1  |     IPinfo_Details 1.0
cortex_1  |     CERTatPassiveDNS 2.0
cortex_1  |     Urlscan.io_Scan 0.1.0
cortex_1  |     DomainToolsIris_CheckMaliciousTags 1.0
cortex_1  |     ProofPoint_Lookup 1.0
cortex_1  |     PayloadSecurity_Url_Analysis 1.0
cortex_1  |     Shodan_DNSResolve 1.0
cortex_1  |     Splunk_Search_Mail_Subject 3.0
cortex_1  |     GoogleDNS_resolve 1.0.0
cortex_1  |     DomainToolsIris_Pivot 1.0
cortex_1  |     MetaDefenderCloud_GetReport 1.0
cortex_1  |     Hipposcore 2.0
cortex_1  |     Shodan_InfoDomain 1.0
cortex_1  |     CuckooSandbox_File_Analysis_Inet 1.2
cortex_1  |     JoeSandbox_File_Analysis_Noinet 2.0
cortex_1  |     GoogleVisionAPI_WebDetection 1.0.0
cortex_1  |     TalosReputation 1.0
cortex_1  |     Splunk_Search_IP 3.0
cortex_1  |     TorBlutmagie 1.0
cortex_1  |     SpamAssassin 1.0
cortex_1  |     Splunk_Search_Domain_FQDN 3.0
cortex_1  |     FireHOLBlocklists 2.0
cortex_1  |     NERD 1.0
cortex_1  |     ThreatGrid 1.0
cortex_1  |     Robtex_Reverse_PDNS_Query 1.0
cortex_1  |     PassiveTotal_Ssl_Certificate_Details 2.0
cortex_1  |     AMPforEndpoints_IsolationStart 1.0
cortex_1  |     VMRay 3.0
cortex_1  |     DNSDB_NameHistory 2.0
cortex_1  |     PhishingInitiative_Lookup 2.0
cortex_1  |     AMPforEndpoints_IsolationStop 1.0
cortex_1  |     SoltraEdge 1.0
cortex_1  |     Pulsedive_GetIndicator 1.0
cortex_1  |     QRadar_Auto_Closing_Offense 1.0
cortex_1  |     IBMXForce_Lookup 1.0
cortex_1  |     Splunk_Search_URL_URI_Path 3.0
cortex_1  |     JoeSandbox_Url_Analysis 2.0
cortex_1  |     Censys 1.0
cortex_1  |     Malwares_Scan 1.0
cortex_1  |     Robtex_IP_Query 1.0
cortex_1  |     HippoMore 2.0
cortex_1  |     HybridAnalysis_GetReport 1.0
cortex_1  |     EmlParser 1.2
cortex_1  |     AMPforEndpoints_SCDRemove 1.0
cortex_1  |     ClamAV_FileInfo 1.1
cortex_1  |     ForcepointWebsensePing 1.0
cortex_1  |     Shodan_Search 2.0
cortex_1  |     Umbrella_Report 1.0
cortex_1  |     PassiveTotal_Components 2.0
cortex_1  |     MetaDefenderCore_GetReport 1.0
cortex_1  |     MalwareClustering_Search 1.0
cortex_1  |     Mnemonic_pDNS_Closed 3.0
cortex_1  |     Splunk_Search_File_Filename 3.0
cortex_1  |     UnshortenLink 1.2
cortex_1  |     Onyphe_Summary 1.0
cortex_1  |     AnyRun_Sandbox_Analysis 1.0
cortex_1  |
cortex_1  | [error] o.e.d.DBConfiguration - ElasticSearch request failure: POST:/cortex_6/_search?scroll=60000ms
cortex_1  | StringEntity({"seq_no_primary_term":"true","query":{"bool":{"must":[{"term":{"relations":{"value":"worker"}}},{"match_all":{}}]}},"from":0,"sort":[{"_doc":{"order":"desc"}}]},Some(application/json))
cortex_1  |  => ElasticError(security_exception,missing authentication credentials for REST request [/cortex_6/_search?scroll=60000ms],None,None,None,List(ElasticError(security_exception,missing authentication credentials for REST request [/cortex_6/_search?scroll=60000ms],None,None,None,null,None,None,None,List())),None,None,None,List())
cortex_1  | [warn] o.e.d.SearchWithScroll - Search error
cortex_1  | org.elastic4play.InternalError: Unknown error: ElasticError(security_exception,missing authentication credentials for REST request [/cortex_6/_search?scroll=60000ms],None,None,None,List(ElasticError(security_exception,missing authentication credentials for REST request [/cortex_6/_search?scroll=60000ms],None,None,None,null,None,None,None,List())),None,None,None,List())
cortex_1  |     at org.elastic4play.database.DBConfiguration.$anonfun$execute$2(DBConfiguration.scala:158)
cortex_1  |     at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:307)
cortex_1  |     at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:41)
cortex_1  |     at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
cortex_1  |     at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:56)
cortex_1  |     at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:93)
cortex_1  |     at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
cortex_1  |     at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85)
cortex_1  |     at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:93)
cortex_1  |     at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:48)
cortex_1  | [info] p.c.s.AkkaHttpServer - Enabling HTTP/2 on Akka HTTP server...
cortex_1  | [info] p.c.s.AkkaHttpServer - Listening for HTTP on /0.0.0.0:9001

  2. This is the error when the browser logs in image

  3. Docker Config

    version: "2"
services:
cortex:
image: thehiveproject/cortex:3.1.1-1
environment:
  - job_directory=${job_directory}
  - es_uri=https://elasticsearch01:9200
  - es_hostname=elasticsearch01:192.168.199.11
volumes:
  - './vol/cortex/application.conf:/etc/cortex/application.conf'
  - './vol/cortex/truststore.jks:/etc/cortex/truststore.jks'
  - /var/run/docker.sock:/var/run/docker.sock
  - ${job_directory}:${job_directory}
ports:
  - "0.0.0.0:9001:9001"
extra_hosts:
    - "elasticsearch01:192.168.199.11"

  4. Cortex Config

    
search {
index = cortex
## Authentication configuration
username = "elastic"
password = "HelloWorld"

## SSL configuration
keyStore {
path = "/etc/cortex/truststore.jks"
type = "JKS" # or PKCS12
password = "HelloWorld"
}
trustStore {
path = "/etc/cortex/truststore.jks"
type = "JKS" # or PKCS12
password = "HelloWorld"
}
}

cache.job = 10 minutes

analyzer { urls = [ "https://download.thehive-project.org/analyzers.json" ] fork-join-executor { parallelism-min = 2 parallelism-factor = 2.0 parallelism-max = 4 } }

responder { urls = [ "https://download.thehive-project.org/responders.json" ] fork-join-executor { parallelism-min = 2 parallelism-factor = 2.0 parallelism-max = 4 } }

Canon88 commented 3 years ago

This problem has been solved. Update configuration username -> user

search {
  index = cortex
  ## Authentication configuration
  user = "elastic"
  password = "HelloWorld"

  ## SSL configuration
  keyStore {
    path = "/etc/cortex/truststore.jks"
    type = "JKS" # or PKCS12
    password = "HelloWorld"
  }
  trustStore {
    path = "/etc/cortex/truststore.jks"
    type = "JKS" # or PKCS12
    password = "HelloWorld"
  }
}

cache.job = 10 minutes

analyzer {
  urls = [
    "https://download.thehive-project.org/analyzers.json"
  ]
  fork-join-executor {
    parallelism-min = 2
    parallelism-factor = 2.0
    parallelism-max = 4
  }
}

responder {
  urls = [
    "https://download.thehive-project.org/responders.json"
  ]
  fork-join-executor {
    parallelism-min = 2
    parallelism-factor = 2.0
    parallelism-max = 4
  }
}