Cortex Analyzers unavailable in cortex Web UI

[shehreyarahmedkohati]

Request Type

Issue

Work Environment

Question	Answer
OS version (server)	Ubuntu 16.04 Virtual Machine
Cortex version / git hash	2.1.1
Package Type	Binary

Problem Description

I have tried to configure the cortex analyzers using this guide https://github.com/TheHive-Project/CortexDocs/blob/master/installation/install-guide.md#analyzers-and-responders

My application.conf file is as follows:

# Secret key
# ~~~~~
# The secret key is used to secure cryptographics functions.
# If you deploy your application to several instances be sure to use the same key!
play.http.secret.key="lc9EXWeI68lofpB3X3YVTZmUXpW**ClKuoIKMgfH****7MSaa9FdinaZ2"

analyzer.path = ["/home/digit/Downloads/Cortex-Analyzers/analyzers"]

## ElasticSearch
search {
  # Name of the index
  index = cortex
  # Name of the ElasticSearch cluster
  cluster = hive
  # Address of the ElasticSearch instance
  host = ["127.0.0.1:9300"]
}

## Cache
#
# If an analyzer is executed against the same observable, the previous report can be returned without re-executing the
# analyzer. The cache is used only if the second job occurs within cache.job (the default is 10 minutes).
cache.job = 10 minutes

## Authentication
auth {
    # "provider" parameter contains the authentication provider(s). It can be multi-valued, which is useful
    # for migration.
    # The available auth types are:
    # - services.LocalAuthSrv : passwords are stored in the user entity within ElasticSearch). No
    #   configuration are required.
    # - ad : use ActiveDirectory to authenticate users. The associated configuration shall be done in
    #   the "ad" section below.
    # - ldap : use LDAP to authenticate users. The associated configuration shall be done in the
    #   "ldap" section below.
    provider = [local]

    ad {
        # The Windows domain name in DNS format. This parameter is required if you do not use
        # 'serverNames' below.
        #domainFQDN = "mydomain.local"

        # Optionally you can specify the host names of the domain controllers instead of using 'domainFQDN
        # above. If this parameter is not set, TheHive uses 'domainFQDN'.
        #serverNames = [ad1.mydomain.local, ad2.mydomain.local]

        # The Windows domain name using short format. This parameter is required.
        #domainName = "MYDOMAIN"

        # If 'true', use SSL to connect to the domain controller.
        #useSSL = true
    }

    ldap {
        # The LDAP server name or address. The port can be specified using the 'host:port'
        # syntax. This parameter is required if you don't use 'serverNames' below.
        #serverName = "ldap.mydomain.local:389"

        # If you have multiple LDAP servers, use the multi-valued setting 'serverNames' instead.
        #serverNames = [ldap1.mydomain.local, ldap2.mydomain.local]

        # Account to use to bind to the LDAP server. This parameter is required.
        #bindDN = "cn=thehive,ou=services,dc=mydomain,dc=local"

        # Password of the binding account. This parameter is required.
        #bindPW = "***secret*password***"

        # Base DN to search users. This parameter is required.
        #baseDN = "ou=users,dc=mydomain,dc=local"

        # Filter to search user in the directory server. Please note that {0} is replaced
        # by the actual user name. This parameter is required.
        #filter = "(cn={0})"

        # If 'true', use SSL to connect to the LDAP directory server.
        #useSSL = true
    }
}

## ANALYZERS
#
analyzer {
  # Absolute path where you have pulled the Cortex-Analyzers repository.
  path = ["/home/digit/Downloads/Cortex-Analyzers/analyzers"]

  # Sane defaults. Do not change unless you know what you are doing.
  fork-join-executor {

    # Min number of threads available for analysis.
    parallelism-min = 2

    # Parallelism (threads) ... ceil(available processors * factor).
    parallelism-factor = 2.0

    # Max number of threads available for analysis.
    parallelism-max = 4
  }

responder {
  # Directory that holds responders
  path = ["/home/digit/Downloads/Cortex-Analyzers/responders"]
}
}

---

Any help regarding this?

[nadouani]

Please ready the Quick Start guide

Thanks

[shehreyarahmedkohati]

@nadouani I have followed these guides and also have experience working on training VM's in the past. Hence doing a binary install this time. I got two errors while installing dependencies for the analyzers.

1) urllib2

(Collecting urllib2 (from -r Cortex-Analyzers/analyzers/DShield/requirements.txt (line 2)) Could not find a version that satisfies the requirement urllib2 (from -r Cortex-Analyzers/analyzers/DShield/requirements.txt (line 2)) (from versions: ) No matching distribution found for urllib2 (from -r Cortex-Analyzers/analyzers/DShield/requirements.txt (line 2)

2) aiohttp 2.2.2 (Collecting aiohttp==3.4.4 (from domaintools-api==0.3.1->-r Cortex-Analyzers/analyzers/DomainTools/requirements.txt (line 3)) Could not find a version that satisfies the requirement aiohttp==3.4.4 (from domaintools-api==0.3.1->-r Cortex-Analyzers/analyzers/DomainTools/requirements.txt (line 3)) (from versions: 0.1, 0.2, 0.3, 0.4, 0.4.1, 0.4.2, 0.4.3, 0.4.4, 0.5.0, 0.6.0, 0.6.1, 0.6.2, 0.6.3, 0.6.4, 0.6.5, 0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.8.0, 0.8.1, 0.8.2, 0.8.3, 0.8.4, 0.9.0, 0.9.1, 0.9.2, 0.9.3, 0.10.0, 0.10.1, 0.10.2, 0.11.0, 0.12.0, 0.13.0, 0.13.1, 0.14.0, 0.14.1, 0.14.2, 0.14.3, 0.14.4, 0.15.0, 0.15.1, 0.15.2, 0.15.3, 0.16.0, 0.16.1, 0.16.2, 0.16.3, 0.16.4, 0.16.5, 0.16.6, 0.17.0, 0.17.1, 0.17.2, 0.17.3, 0.17.4, 0.18.0, 0.18.1, 0.18.2, 0.18.3, 0.18.4, 0.19.0, 0.20.0, 0.20.1, 0.20.2, 0.21.0, 0.21.1, 0.21.2, 0.21.4, 0.21.5, 0.21.6, 0.22.0a0, 0.22.0b0, 0.22.0b1, 0.22.0b2, 0.22.0b3, 0.22.0b4, 0.22.0b5, 0.22.0b6, 0.22.0, 0.22.1, 0.22.2, 0.22.3, 0.22.4, 0.22.5, 1.0.0, 1.0.1, 1.0.2, 1.0.3, 1.0.5, 1.1.0, 1.1.1, 1.1.2, 1.1.3, 1.1.4, 1.1.5, 1.1.6, 1.2.0, 1.3.0, 1.3.1, 1.3.2, 1.3.3, 1.3.4, 1.3.5, 2.0.0rc1, 2.0.0, 2.0.1, 2.0.2, 2.0.3, 2.0.4, 2.0.5, 2.0.6, 2.0.6.post1, 2.0.7, 2.1.0, 2.2.0, 2.2.1, 2.2.2, 2.2.3, 2.2.4, 2.2.5, 2.3.0a1, 2.3.0a2, 2.3.0a4, 2.3.0, 2.3.1a1, 2.3.1, 2.3.2b2, 2.3.2b3, 2.3.2, 2.3.3, 2.3.4, 2.3.5, 2.3.6, 2.3.7, 2.3.8, 2.3.9, 2.3.10, 3.0.0b0) No matching distribution found for aiohttp==3.4.4 (from domaintools-api==0.3.1->-r Cortex-Analyzers/analyzers/DomainTools/requirements.txt (line 3)

[nadouani]

@3c7 Any idea about these dependencies?

[shehreyarahmedkohati]

Can you also tell me if the analyzer path in the application.conf file is defined by this command "analyzer.path = ["/home/digit/Downloads/Cortex-Analyzers/analyzers"]" OR by the block of code starting with analyzer {path = ["/home/digit/Downloads/Cortex-Analyzers/analyzers"], etc.... }

Which one should be used @nadouani

[nadouani]

I think that your analyzer.path=XXX is overwritten by the block of code you are referring to.

Add the customizations, like analyzer.path=XXX at the end of the file ;)

[3c7]

Please use both, Python2 and python3. I know that's bad, but analyzer python requirements are not consistent atm. Using python3:

pip install aiohttp==3.4.4
Collecting aiohttp==3.4.4
  Downloading https://files.pythonhosted.org/packages/52/f9/c22977fc95346911d8fe507f90c3c4e4f445fdf339b750be6f03f090498d/aiohttp-3.4.4-cp36-cp36m-manylinux1_x86_64.whl (1.1MB)
    100% |████████████████████████████████| 1.1MB 521kB/s

At least it looks like that's a python version problem, as you can't find the specific module version. I'm able to install aiohttp 3.4.4 using python3. If you install all dependencies like that, all analyzers should work:

cd path/to/Cortex-Analyzers
git pull
for I in analyzers/*/requirements.txt; do sudo -H pip2 install -U -r $I || true; done && \
for I in analyzers/*/requirements.txt; do sudo -H pip3 install -U -r $I || true; done

[shehreyarahmedkohati]

@nadouani I did what you said, and restarted the cortex server but still cant see any analyzers in my cortex web UI. The 2 errors regarding the dependencies rise from the DomainTools And DShield Analyzers. Hence, I removed those two analyzers to check the remaining ones but still cant see any of them. Can you share with me a working application.conf file

[shehreyarahmedkohati]

@3c7 I used your command above, still got the same error. My python versions are Python 2.7.12 and Python 3.5.2 respectively. The two analyzers that needs these dependcies are DomainTools And DShield Analyzers. So I also went cd /Downloads/Cortex-Analyzers/analyzers/DomainTools and did a "sudo pip3 install -r requirements.txt" and got the error below:

Ignoring domaintools-api: markers 'python_version < "3.5"' don't match your environment Collecting git+https://github.com/DomainTools/python_api.git (from -r requirements.txt (line 3)) Cloning https://github.com/DomainTools/python_api.git to /tmp/pip-req-build-ep0aduc6

Requirement already satisfied: cortexutils in /usr/local/lib/python3.5/dist-packages (from -r requirements.txt (line 1)) (1.3.0) Requirement already satisfied: requests in /usr/local/lib/python3.5/dist-packages (from domaintools-api==0.3.1->-r requirements.txt (line 3)) (2.20.0) Collecting aiohttp==3.4.4 (from domaintools-api==0.3.1->-r requirements.txt (line 3)) Could not find a version that satisfies the requirement aiohttp==3.4.4 (from domaintools-api==0.3.1->-r requirements.txt (line 3)) (from versions: 0.1, 0.2, 0.3, 0.4, 0.4.1, 0.4.2, 0.4.3, 0.4.4, 0.5.0, 0.6.0, 0.6.1, 0.6.2, 0.6.3, 0.6.4, 0.6.5, 0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.8.0, 0.8.1, 0.8.2, 0.8.3, 0.8.4, 0.9.0, 0.9.1, 0.9.2, 0.9.3, 0.10.0, 0.10.1, 0.10.2, 0.11.0, 0.12.0, 0.13.0, 0.13.1, 0.14.0, 0.14.1, 0.14.2, 0.14.3, 0.14.4, 0.15.0, 0.15.1, 0.15.2, 0.15.3, 0.16.0, 0.16.1, 0.16.2, 0.16.3, 0.16.4, 0.16.5, 0.16.6, 0.17.0, 0.17.1, 0.17.2, 0.17.3, 0.17.4, 0.18.0, 0.18.1, 0.18.2, 0.18.3, 0.18.4, 0.19.0, 0.20.0, 0.20.1, 0.20.2, 0.21.0, 0.21.1, 0.21.2, 0.21.4, 0.21.5, 0.21.6, 0.22.0a0, 0.22.0b0, 0.22.0b1, 0.22.0b2, 0.22.0b3, 0.22.0b4, 0.22.0b5, 0.22.0b6, 0.22.0, 0.22.1, 0.22.2, 0.22.3, 0.22.4, 0.22.5, 1.0.0, 1.0.1, 1.0.2, 1.0.3, 1.0.5, 1.1.0, 1.1.1, 1.1.2, 1.1.3, 1.1.4, 1.1.5, 1.1.6, 1.2.0, 1.3.0, 1.3.1, 1.3.2, 1.3.3, 1.3.4, 1.3.5, 2.0.0rc1, 2.0.0, 2.0.1, 2.0.2, 2.0.3, 2.0.4, 2.0.5, 2.0.6, 2.0.6.post1, 2.0.7, 2.1.0, 2.2.0, 2.2.1, 2.2.2, 2.2.3, 2.2.4, 2.2.5, 2.3.0a1, 2.3.0a2, 2.3.0a4, 2.3.0, 2.3.1a1, 2.3.1, 2.3.2b2, 2.3.2b3, 2.3.2, 2.3.3, 2.3.4, 2.3.5, 2.3.6, 2.3.7, 2.3.8, 2.3.9, 2.3.10, 3.0.0b0) No matching distribution found for aiohttp==3.4.4 (from domaintools-api==0.3.1->-r requirements.txt (line 3))

and for DShield Analyzer:

Requirement already satisfied: cortexutils in /usr/local/lib/python3.5/dist-packages (from -r requirements.txt (line 1)) (1.3.0) Collecting urllib2 (from -r requirements.txt (line 2)) Could not find a version that satisfies the requirement urllib2 (from -r requirements.txt (line 2)) (from versions: ) No matching distribution found for urllib2 (from -r requirements.txt (line 2))

[shehreyarahmedkohati]

### When I start the cortex server:

sudo bin/cortex -Dconfig.file=/etc/cortex/application.conf[sudo] password for digit: [info] o.r.Reflections - Reflections took 918 ms to scan 2 urls, producing 97 keys and 913 values [info] module - Loading model class org.thp.cortex.models.AuditModel [info] module - Loading model class org.thp.cortex.models.JobModel [info] module - Loading model class org.thp.cortex.models.UserModel [info] module - Loading model class org.elastic4play.services.AttachmentModel [info] module - Loading model class org.thp.cortex.models.WorkerModel [info] module - Loading model class org.thp.cortex.models.ArtifactModel [info] module - Loading model class org.elastic4play.services.DBListModel [info] module - Loading model class org.thp.cortex.models.ReportModel [info] module - Loading model class org.thp.cortex.models.WorkerConfigModel [info] module - Loading model class org.thp.cortex.models.OrganizationModel [info] a.e.s.Slf4jLogger - Slf4jLogger started [info] o.e.p.PluginsService - no modules loaded [info] o.e.p.PluginsService - loaded plugin [org.elasticsearch.index.reindex.ReindexPlugin] [info] o.e.p.PluginsService - loaded plugin [org.elasticsearch.join.ParentJoinPlugin] [info] o.e.p.PluginsService - loaded plugin [org.elasticsearch.percolator.PercolatorPlugin] [info] o.e.p.PluginsService - loaded plugin [org.elasticsearch.script.mustache.MustachePlugin] [info] o.e.p.PluginsService - loaded plugin [org.elasticsearch.transport.Netty3Plugin] [info] o.e.p.PluginsService - loaded plugin [org.elasticsearch.transport.Netty4Plugin] [info] i.n.u.i.PlatformDependent - Your platform does not provide complete low-level API for accessing direct buffers reliably. Unless explicitly requested, heap buffer will always be preferred to avoid potential system instability. [info] o.t.c.s.WorkerSrv - New worker list:

CuckooSandbox_Url_Analysis 1.0
VirusTotal_GetReport 3.0
FireEyeiSight 1.0
Shodan_Search 1.0
Malwares_GetReport 1.0
Mnemonic_pDNS_Public 3.0
Urlscan.io_Search 0.1.0
Onyphe_Datascan 1.0
PassiveTotal_Osint 2.0
CIRCLPassiveDNS 2.0
PassiveTotal_Passive_Dns 2.1
Shodan_Host 1.0
Hunterio_DomainSearch 1.0
TorProject 1.0
CIRCLPassiveSSL 2.0
Fortiguard_URLCategory 2.1
Yara 2.0
Onyphe_Threats 1.0
EmergingThreats_DomainInfo 1.0
URLhaus 1.0
DNSDB_DomainName 2.0
Onyphe_Forward 1.0
PhishTank_CheckURL 2.1
OTXQuery 2.0
Investigate_Sample 1.0
PhishingInitiative_Scan 1.0
C1fApp 1.0
RecordedFuture_risk 1.0
Nessus 2.0
MISP 2.0
JoeSandbox_File_Analysis_Inet 2.0
Virusshare 2.0
CuckooSandbox_File_Analysis_Inet 1.0
Onyphe_Ports 1.0
GreyNoise 2.3
Yeti 1.0
StaxxSearch 1.0
Robtex_Forward_PDNS_Query 1.0
Abuse_Finder 2.0
VirusTotal_Scan 3.0
EmergingThreats_IPInfo 1.0
PassiveTotal_Whois_Details 2.0
PassiveTotal_Malware 2.0
DNSDB_IPHistory 2.0
WOT_Lookup 1.0
GoogleSafebrowsing 2.0
PassiveTotal_Enrichment 2.0
PayloadSecurity_File_Analysis 1.0
PassiveTotal_Unique_Resolutions 2.0
Hashdd_Detail 1.0
Threatcrowd 1.0
UnshortenLink 1.0
CyberCrime-Tracker 1.0
Hashdd_Status 1.0
EmergingThreats_MalwareInfo 1.0
StopForumSpam 1.0
Onyphe_Inetnum 1.0
SinkDB 1.0
Investigate_Categorization 1.0
PassiveTotal_Ssl_Certificate_History 2.0
Malpedia 1.0
Msg_Parser 2.0
Crt_sh_Transparency_Logs 1.0
CERTatPassiveDNS 2.0
ProofPoint_Lookup 1.0
PayloadSecurity_Url_Analysis 1.0
GoogleDNS_resolve 1.0.0
Hipposcore 2.0
JoeSandbox_File_Analysis_Noinet 2.0
TorBlutmagie 1.0
FileInfo 5.0
Cymon_Check_IP 2.1
FireHOLBlocklists 2.0
EmlParser 1.1
Robtex_Reverse_PDNS_Query 1.0
PassiveTotal_Ssl_Certificate_Details 2.0
Onyphe_Reverse 1.0
VMRay 3.0
DNSDB_NameHistory 2.0
Onyphe_Geolocate 1.0
PhishingInitiative_Lookup 2.0
Pulsedive_GetIndicator 1.0
IBMXForce_Lookup 1.0
JoeSandbox_Url_Analysis 2.0
Censys 1.0
Malwares_Scan 1.0
Robtex_IP_Query 1.0
HippoMore 2.0
HybridAnalysis_GetReport 1.0
MISPWarningLists 1.0
Mnemonic_pDNS_Closed 3.0
MaxMind_GeoIP 3.0

[info] play.api.Play - Application started (Prod) [info] p.c.s.AkkaHttpServer - Enabling HTTP/2 on Akka HTTP server... [info] p.c.s.AkkaHttpServer - Listening for HTTP on /0:0:0:0:0:0:0:0:9001

[nadouani]

Hello, what user role do you use? Did you enable the analyzers from the administration page under « Organization > Analyzers »? Do you see them listed?

The logs say that Cortex has found the analyzer definitions.

[shehreyarahmedkohati]

I have made an organization called Digit labs as you can see here with one user

the user screen for that organization is this:

[shehreyarahmedkohati]

I dont see the configuration and analyzers page as the quick start guide screens have.

[3c7]

The user digitlabs has no administrator privileges.

[nadouani]

So, please read the quick start guide again.

You nees to use an OrgAdmin user to activate the analyzers you want to make available to the Cortex users that will run the analyzers.

When you create an organization, all the analyzers are disabled by default because each one requires its own configuration

Thank you

[3c7]

I could only imagine that the required package is not available for python3.5. I'm using 3.6.6 and I can't reproduce the issue. :/

[shehreyarahmedkohati]

Oh that I did not think of. Thank you so much for the enlightenment folks. I can see the config and analyzers tabs now.

[nadouani]

That’s why I’ve invited you to the quick start guide that quickly explains how to use cortex ;)

[shehreyarahmedkohati]

Haha I get it now :)

[azgss]

hi ,I have a problem with the responders no responders found (responders panel is empty) on the other hand in the directory /opt/cortex/Cortex-Analyzers/responders/, there are several responders

I checked -responders path in /etc/cortex/application.conf file which is -directory permission (the analyzers work correctly)