search

PaloAltoNetworks / minemeld-misp

MineMeld nodes for MISP
Apache License 2.0
18 stars 16 forks source link

Minemeld MISP Error in New MISP.any local prototype #4

Open da667 opened 6 years ago

da667 commented 6 years ago

Hello. I recently added the minemeld-misp module to my minemeld instance.

Operating System: lsb_release -a:

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.4 LTS
Release:        16.04
Codename:       xenial

uname -a:

Linux [hostname redacted] 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12 21:23:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

I am currently running both MISP AND Minemeld on the same system. Minemeld Version: 0.9.44.post1 MISP Version: 2.4.89

If you're wondering how we're getting minemeld and MISP to exist on the same host, we're currently hacking it by having nginx listen on port 81, and 444 to host minemeld, then using an apache mod_rewrite rule to redirect any requests to /[servername]/minemeld, to https://localhost:444 (where minemeld is listening on SSL). So far, everything works wonderfully. My other modules work. They retrieve data, things work fine. But just because I care about completion, here are the apache rewrite rules, for posterity:

RewriteRule ^/minemeld(.*)$ https://localhost:444$1 [L,P]
RewriteCond expr "%{HTTP_REFERER} -strmatch 'https://%{HTTP_HOST}/minemeld/*'"
RewriteCond expr "! %{REQUEST_URI} -strmatch '/minemeld*'"
RewriteRule ^(.*)$ https://localhost:444$1 [L,P]

I have installed the minemeld-misp module via github option on the Minemeld System>Extensions tab. After installing the module, I enabled the module, and restarted minemeld. I can confirm that the misp.anyEvent prototype got installed. Additionally, per the installation instructions, I created a new minemeldlocal prototype based off of the misp.anyEvent prototype, per the instructions on this github repo. Here is what the prototype parameters look like:

# source name, to identify the origin of the indicators inside MineMeld
source_name: misp.test
# URL of MISP
url: https://[hostname]
# filters for MISP query
# default: none
# this one check for published events with tag tlp:white
# you can specify a time window of the last N days using datefrom: <N>d
# check the search_index API in PyMISP for available filter parameters
filters:
  published: 1
  tag: 'tlp:white'
  # datefrom: 180d
# select specific inidicator types, default: null (any)
# indicator_types: ['URL', 'IPv4', 'IPv6']
indicator_types: null
# honour IDS flag, if true only events with IDS set will be exported
honour_ids_flag: false
# a dictionary of event attributes to be extracted, the value
# of each in key in the dictionary is a JMESPATH expression
# default:
# event_attributes:
#   info: info
#   org: Org
#   orgc: Orgc
#   threat_level_id: threat_level_id
#   uuid: uuid
#   tags: Tag[*].name
# a dictionary of attribute attributes to be extracted, the value
# of each in key in the dictionary is a JMESPATH expression
# default:
# attribute_attributes:
#   uuid: info
#   category: Org
#   comment: Orgc
# prefix to be applied to indicator attributes, default: misp
# prefix: misp
# verify remote certificate, default true
verify_cert: false
# require a client certificate, default false
client_cert_required: false
# age out of indicators
age_out:
  sudden_death: true
  default: last_seen+30d
# flag indicators with share level white, if not specified
# by tag
attributes:
  confidence: 70
  # if not specified in the event, default is white for
  # this prototype
  share_leve: white

I cloned this new prototype, added it as a node to our config, committed the change on the config page, navigate to the nodes page, and edit the settings for our new node, specifically adding an API key for access, and explicitly disabling cert verification.

I then instruct the MISP node to run, and the only indication I get that anything went wrong on the web interface is a red box labeled "error" in the Last Run field of our newly configured MISP node. When I hover over the error box, this is the only text that I get back:

Unknown error (something is very broken server-side:)

So... That tells me absolutely nothing. Hunting through other bugs for minemeld, I decided to check out /opt/minemeld/log/minemeld-engine.log:

2018-03-28T15:12:16 (9888)connectionpool._new_conn INFO: Starting new HTTPS connection (1): [redacted hostname]
/opt/minemeld/engine/current/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
2018-03-28T15:12:16 (9888)connectionpool._new_conn INFO: Starting new HTTPS connection (1): [redacted hostname]
/opt/minemeld/engine/current/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
2018-03-28T15:12:16 (9888)node._build_iterator INFO: MISP_HLABS - query filters: {'tag': 'tlp:white', 'published': 1}
2018-03-28T15:12:16 (9888)connectionpool._new_conn INFO: Starting new HTTPS connection (1): [redacted hostname]
/opt/minemeld/engine/current/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
2018-03-28T15:12:16 (9888)basepoller._poll ERROR: Exception in polling loop for MISP_HLABS: Unknown error (something is very broken server-side: )
Traceback (most recent call last):
  File "/opt/minemeld/engine/core/minemeld/ft/basepoller.py", line 721, in _poll
    performed = self._polling_loop()
  File "/opt/minemeld/engine/core/minemeld/ft/basepoller.py", line 571, in _polling_loop
    iterator = self._build_iterator(now)
  File "/opt/minemeld/local/library/ed995eae-e28b-4b39-840e-abbf5787d8e5/mmmisp/node.py", line 149, in _build_iterator
    r = misp.get_index(filters)
  File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/pymisp/api.py", line 309, in get_index
    return self._check_response(response)
  File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/pymisp/api.py", line 220, in _check_response
    raise PyMISPError('Unknown error (something is very broken server-side: {})'.format(response.text))
PyMISPError: Unknown error (something is very broken server-side: )
2018-03-28T15:12:17 (9888)basepoller._polling_loop INFO: Polling MISP_HLABS
2018-03-28T15:12:17 (9888)connectionpool._new_conn INFO: Starting new HTTPS connection (1): [redacted hostname]
/opt/minemeld/engine/current/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
2018-03-28T15:12:17 (9888)connectionpool._new_conn INFO: Starting new HTTPS connection (1): [redacted hostname]
/opt/minemeld/engine/current/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
2018-03-28T15:12:17 (9888)node._build_iterator INFO: MISP_HLABS - query filters: {'tag': 'tlp:white', 'published': 1}
2018-03-28T15:12:17 (9888)connectionpool._new_conn INFO: Starting new HTTPS connection (1): [redacted hostname]
/opt/minemeld/engine/current/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
2018-03-28T15:12:17 (9888)basepoller._poll ERROR: Exception in polling loop for MISP_HLABS: Unknown error (something is very broken server-side: )
Traceback (most recent call last):
  File "/opt/minemeld/engine/core/minemeld/ft/basepoller.py", line 721, in _poll
    performed = self._polling_loop()
  File "/opt/minemeld/engine/core/minemeld/ft/basepoller.py", line 571, in _polling_loop
    iterator = self._build_iterator(now)
  File "/opt/minemeld/local/library/ed995eae-e28b-4b39-840e-abbf5787d8e5/mmmisp/node.py", line 149, in _build_iterator
    r = misp.get_index(filters)
  File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/pymisp/api.py", line 309, in get_index
    return self._check_response(response)
  File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/pymisp/api.py", line 220, in _check_response
    raise PyMISPError('Unknown error (something is very broken server-side: {})'.format(response.text))
PyMISPError: Unknown error (something is very broken server-side: )

Specifically:

2018-03-28T15:12:16 (9888)node._build_iterator INFO: MISP_HLABS - query filters: {'tag': 'tlp:white', 'published': 1}
2018-03-28T15:12:16 (9888)basepoller._poll ERROR: Exception in polling loop for MISP_HLABS: Unknown error (something is very broken server-side: )
Traceback (most recent call last):
  File "/opt/minemeld/engine/core/minemeld/ft/basepoller.py", line 721, in _poll
    performed = self._polling_loop()
  File "/opt/minemeld/engine/core/minemeld/ft/basepoller.py", line 571, in _polling_loop
    iterator = self._build_iterator(now)
  File "/opt/minemeld/local/library/ed995eae-e28b-4b39-840e-abbf5787d8e5/mmmisp/node.py", line 149, in _build_iterator
    r = misp.get_index(filters)
  File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/pymisp/api.py", line 309, in get_index
    return self._check_response(response)
  File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/pymisp/api.py", line 220, in _check_response
    raise PyMISPError('Unknown error (something is very broken server-side: {})'.format(response.text))
PyMISPError: Unknown error (something is very broken server-side: )

I have no idea what this error is referring to.

When I check /var/log/apache2/access.log on my server, here are the entries that show up when I force the miner node to start:

[redacted IP address] [redacted hostname] - - 443 [28/Mar/2018:15:17:51 -0400] "GET /servers/getPyMISPVersion.json HTTP/1.1" "?/servers/getPyMISPVersion.json" 200 20 "-" "PyMISP 2.4.89 - Python 2.7.12" 771 6008 31849
[redacted IP address] [redacted hostname] - - 443 [28/Mar/2018:15:17:51 -0400] "GET /attributes/describeTypes.json HTTP/1.1" "?/attributes/describeTypes.json" 200 17451 "-" "PyMISP 2.4.89 - Python 2.7.12" 771 23471 31050
[redacted IP address] [redacted hostname] - - 443 [28/Mar/2018:15:17:51 -0400] "POST /events/index HTTP/1.1" "?/events/index" 200 - "-" "PyMISP 2.4.89 - Python 2.7.12" 811 5958 107050
[redacted IP address] [redacted hostname] - - 443 [28/Mar/2018:15:17:54 -0400] "GET /servers/getPyMISPVersion.json HTTP/1.1" "?/servers/getPyMISPVersion.json" 200 20 "-" "PyMISP 2.4.89 - Python 2.7.12" 771 6008 29807
[redacted IP address] [redacted hostname] - - 443 [28/Mar/2018:15:17:54 -0400] "GET /attributes/describeTypes.json HTTP/1.1" "?/attributes/describeTypes.json" 200 17451 "-" "PyMISP 2.4.89 - Python 2.7.12" 771 23471 28855
[redacted IP address] [redacted hostname] - - 443 [28/Mar/2018:15:17:54 -0400] "POST /events/index HTTP/1.1" "?/events/index" 200 - "-" "PyMISP 2.4.89 - Python 2.7.12" 811 5958 108727

I need your help determining WHY the misp miner is not collecting data from the MISP instance hosted on the same system.

node1 node2

da667 commented 6 years ago

think we might have found the solution. Not yet confirmed, but we don't get generic errors anymore.

the search filter parameters for MISP changed at some point so the prototype suggestions:

filters:
  published: 1
  tag: 'tlp:white'
  # datefrom: 180d

need to be changed. Specifically, the parameter published: 1 needs to be searchpublished: 1, and the parameter datefrom: xxxd needs to be searchDatefrom: xxxd (yes, the capitalization of the word Date is important)

Again, no more failures... Need to confirm this is the solution first, however.

da667 commented 6 years ago

seems that our issue revolves the filters section of the prototype. Some time between when the documentation on this project was written and now, the pymisp parameters changed for filtering what data pymisp will retrieve.

https://github.com/MISP/misp-book/blob/master/automation/README.md#filtering-event-metadata

this link has a list of the new parameters that pymisp uses. Out of curiosity, I attempted to create a version of the prototype WITHOUT a filter, and this results in the same error. It appears as though this miner or pymisp requires filters of some sort in order to return data. It also appears as though the searchDatefrom parameter, the new filter parameter that actually works requires a date in yyyy-mm-dd format.

As soon as we switched our filter parameters to:

filters:
  searchpublished: 1
  searchDatefrom: 2018-02-26

and left our other prototype parameters the same, MISP events began to be consumed via the minemeld MISP miner.

So... at this point, I guess we need a feature request for a feature that is able to translate a relative date (e.g. 30d) and turn that into a yyyy-mm-dd date timestamp that MISP's search filter API can understand.

I mean, currently, we could probably ghetto-rig a shell script that changes the parameter for the protoype on disk for us every 30 days via cron, but I feel like theres a better way, and I'm hoping you could assist with that.

alexmoldovanbv commented 6 years ago

Hi,

I've tried the solution provided: searchpublished: 1 & searchDatefrom: 2018-02-26. I get the error 'response'. I couldn't find a description of error: 'response'.

jtschichold commented 6 years ago

Hi @da667, @alexmoldovanbv, sorry for being late on this issue and thanks for digging into this. Just give me some few days to retest the extension with recent versions of MISP/PyMISP. May I ask you how do you plan to use this Miner? any specific requirement on the filter side? or suggestion on which filter to apply by default?

da667 commented 6 years ago

Hey there!

So, we use Minemeld and MISP pretty heavily in our environment. We have a ton of automated malware analysis and cuckoo reports automatically pulled into our MISP instance. Ideally, what we would like to do is pull our MISP data to our minemeld instance and create feeds so that the data can be easily consumed.

What I would like to be able to do is what the minemeld-misp extension states it is capable of doing, e.g. I want to pull all IOCs for the last X amount of days, so that they can be pulled into a minemeld feed, and be automatically aged out after a period of time.

the default filter of "datefrom:30d" is no longer working because 'datefrom' is no longer an option for search_index parameter with pyMISP.

jtschichold commented 6 years ago

Thanks, let me fix this.

jtschichold commented 6 years ago

Hi @da667, I have created a new version with 2.4.89b1 that installs pymisp 2.4.89. I have slightly improved prototypes and UI to let you specify the URL from the GUI without customising the prototype.

I think MISP 2.4.95 intoduces a new API, I have to check if pymisp 2.4.95 is backward compatible with MISP <2.4.95. Otherwise I will create a new tag just for 2.4.95 pinning pymisp 2.4.95.