search

TheHive-Project / Cortex-Analyzers

Cortex Analyzers Repository
https://TheHive-Project.github.io/Cortex-Analyzers/
GNU Affero General Public License v3.0
434 stars 375 forks source link

Threatcrowd, TorBlutmagie, TorProject not displayed #414

Closed masual closed 5 years ago

masual commented 5 years ago

Three analyzers not displayed: Threatcrowd, TorBlutmagie, TorProject

Request Type

Bug

Work Environment

Question Answer
OS version (server) Docker image
OS version (client) N/A
Cortex Analyzer Name Threatcrowd, TorBlutmagie, TorProject
Cortex Analyzer Version 1.15.0
Cortex Version 2.1.3-1
Browser type & version Chrome/Firefox

Description

This three analyzers are not displayed in the organization menu. They are placed in the same directory as all other analyzers and their requirements.txt are installed, but they are not present in the Analyzers Config or Analyzers tabs.

Checking the log I found:

[warn] o.t.c.m.WorkerDefinition - Load of worker /opt/Cortex-Analyzers/analyzers/TorBlutmagie/TorBlutmagie.json fails
java.nio.charset.MalformedInputException: Input length = 1
    at java.nio.charset.CoderResult.throwException(CoderResult.java:281)
    at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:339)
    at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
    at java.io.InputStreamReader.read(InputStreamReader.java:184)
    at java.io.BufferedReader.read1(BufferedReader.java:210)
    at java.io.BufferedReader.read(BufferedReader.java:286)
    at java.io.Reader.read(Reader.java:140)
    at scala.io.BufferedSource.mkString(BufferedSource.scala:94)
    at org.thp.cortex.models.WorkerDefinition$.$anonfun$readJsonFile$1(WorkerDefinition.scala:113)
    at scala.util.Try$.apply(Try.scala:209)
[warn] o.t.c.s.WorkerSrv - Worker definition file read error
java.nio.charset.MalformedInputException: Input length = 1
    at java.nio.charset.CoderResult.throwException(CoderResult.java:281)
    at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:339)
    at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
    at java.io.InputStreamReader.read(InputStreamReader.java:184)
    at java.io.BufferedReader.read1(BufferedReader.java:210)
    at java.io.BufferedReader.read(BufferedReader.java:286)
    at java.io.Reader.read(Reader.java:140)
    at scala.io.BufferedSource.mkString(BufferedSource.scala:94)
    at org.thp.cortex.models.WorkerDefinition$.$anonfun$readJsonFile$1(WorkerDefinition.scala:113)
nadouani commented 5 years ago

Hello @masual can you share your /opt/Cortex-Analyzers/analyzers/TorBlutmagie/TorBlutmagie.json file? It looks like Cortex is not able to parse it.

jeromeleonard commented 5 years ago

hello @masual, I was not able to reproduce the issue with any recent versions of Cortex-Analyzers (1.14.4 to latest). Could you please provide us more details as asked by @nadouani like the .json configuration file ?

nadouani commented 5 years ago

Hello @masual, any news?

masual commented 5 years ago

Hello guys, thank you for the feedback. I am using the default version of the analyzers from the repo tag 1.15.0. I can't access my preprod environment at the moment, if you need it I will upload the actual "TorBlutmagie.json" file present in our installation as soon as I can.

masual commented 5 years ago

@nadouani @jeromeleonard sorry for the late response. Here it goes:

{ "name": "TorBlutmagie", "author": "Marc-André DOLL, STARC by EXAPROBE", "license": "AGPL-V3", "url": "https://github.com/TheHive-Project/Cortex-Analyzers", "version": "1.0", "description": "Query http://torstatus.blutmagie.de/query_export.php/Tor_query_EXPORT.csv for TOR exit nodes IP addresses or names.", "dataTypeList": ["ip", "domain", "fqdn"], "command": "TorBlutmagie/tor_blutmagie_analyzer.py", "baseConfig": "TorBlutmagie", "configurationItems": [ { "name": "cache.duration", "description": "Define the cache duration", "type": "number", "multi": false, "required": true, "defaultValue": 3600 }, { "name": "cache.root", "description": "Define the path to the stored data", "type": "string", "multi": false, "required": false } ] }

jeromeleonard commented 5 years ago

asked @To-om if it could be an issue be something related to Cortex in docker

To-om commented 5 years ago

I've tried with thehiveproject/cortex:2.1.3 docker image and I can't reproduce the problem. The analyzers Threatcrowd, TorBlutmagie and TorProject are correctly loaded.

@masual please give us more details on your environment and explain how to reproduce the problem.

jeromeleonard commented 5 years ago

cant reproduce, no more information. closing until we get more details.

Passimist commented 5 years ago

I was having the same problem and found out whats causing it. I just created this github account to let you know. So i´m sorry if commenting on a closed issue isnt the best way to tell you, but i think this should be allright.

The Analyzers dont show up in Cortex because their workers fail to load. And this is happening because of characters like "é" in /Cortex-Analyzers/analyzers/Threatcrowd/Threatcrowd.json I removed those and it worked like a charm.

nadouani commented 5 years ago

Hello @Passimist thanks for the feedback, it's absolutely not a problem to comment a closed issue when you bring useful information.

I think this issue occurs for some people based on how their file charset are read.

Thanks again

Passimist commented 5 years ago

Yea it must be something like that rather than a problem in Cortex, since i got a second instance of thehive running, that is not having this problem. If i happen to find a way to fix it without removing the characters from the files i ll post hit here.

Thanks for your (extremely) fast answer :)

nadouani commented 5 years ago

Yes, we weren't able to reproduce the issue on our side.