Problem pulling data from misp

honey4free commented 7 years ago

Hi i get an error with this module. Some indicators are pulled but i after a short time i get an error stating

Unable to connecto to MISP (https://domain) Please make sure the API key and the URL are correct Connection aborted, gaierror (-2 name or service not known)

I know the api key is correct as i am able to pull some of the indicators. There is no limitation to my knowlage that shoud terminate the conection this fast. Is there a fix to this or a log i can lookat for se if im able to figure out the problem.

Im running the lates version of minemeld and misp

jtschichold commented 7 years ago

Hi @honey4free, did you change the default MISP prototype to point to the MISP base url ?

Thanks, luigi

honey4free commented 7 years ago

Yes i changed the "/opt/minemeld/local/prototypes/minemeldlocal.yml"

MISP-XX:
    class: mmmisp.Miner
    config:
        age_out:
            default: null
            sudden_death: true
        attributes:
            confidence: 70
            share_leve: white
        filters:
            datefrom: 180d
            published: 1
        indicator_types: null
        source_name: misp.test
        url: https://MyMispServer
        verify_cert: true
    description: 'Miner for MISP

        '
    development_status: STABLE
    indicator_types:
    - any
    node_type: miner
    tags:
    - extension
    - misp

I run misp in docker (jtschichold/minemeld). I can pull some indicators from MISP then it stops with the error i added. It is totaly wierd. Havent found the right logs to describe why the Connection is terminated or why yet. Tips to what logs i have to read or how to fix the wierdness :)

jtschichold commented 7 years ago

Could you check the minemeld-engine.log file from System > Dashboard > Engine > Logs and search for your Miner name ? Do you see anything relevant to your Miner ? That errors typically means an issue with DNS

honey4free commented 7 years ago

This is super wierd- Pinging the server and curl -vs https://host returns data from the misp server.

One thing tho is that the domain is not in the dns cache

Hostname was NOT found in DNS cache
Trying xx.xx.xx.xx...
Connected to misp.tld (xx.xx.xx.xx) port 443 (#0) Trying ip (correct ip)

But the minemeld-engine.log file still gives me errors and not a real god one.

2017-08-10T07:50:50 (22039)basepoller._polling_loop INFO: Polling MISP-XX 2017-08-10T07:51:06 (22039)basepoller._poll ERROR: Exception in polling loop for MISP-XX: ('Connection aborted.', gaierror(-2, 'Name or service not known'))

witch is super wierd

jtschichold commented 7 years ago

Could it be that you have http_proxy env variables configured on your system/MineMeld env ?

honey4free commented 7 years ago

There is no Proxy varible set on the system. Might it be something missing in the docker image ?. As i said earlier im able to pull som of the indicators befor it stops. I have checked IPS and FW and there is nothing

jtschichold commented 7 years ago

Couple of questions:

is certificate verification enabled in the MISP Miner ? This should not be the issue as the Miner was able to connect at least one time
have you tried running the curl to MISP from inside the container ?

Thanks, luigi

honey4free commented 7 years ago

I have tried with curl and the ssl hanshake is a-oki. I have also tried deactivating ssl and only used http and the problem presists.

Btw do you also know if there will come a cuckoo extention for minemeld ?

honey4free commented 7 years ago

Also i found out that even if the prototype is stuffed with indicators it looks like it is having problems passing the indicators from the aggregator to the output node even tho i used stdlib.feedHCWithValue and changed the confidence Level of the indicator to 80. Smells funky. Is there a way i can look at the raw data of the individual prototypes ?. Like looking at the json or whatever without looking at the output node. I wan't to figure out what is so wrong here. I can't seam to se any errors in my creation of the prototype. And the logs havent given me a Clue yet

jtschichold commented 7 years ago

Hi @honey4free, would you be available for a webmeeting to debug this issue ? You know my email address :-)

honey4free commented 7 years ago

Sendt you an email :).

honey4free commented 7 years ago

Just tested with another output node (stdlib.taxiiDataFeed) and not any of the stdlib.feedWithValue and it looks like the indicators is passed over to the outputnode. Might be som missconfiguration on my part but i don't se what it might be as i use all 3 stdlib.feedWithValue

honey4free commented 7 years ago

I have an update. Misp pull works like a charm on my Ubuntu 14.04 so it looks to me like the problem might be isolated to the docker image

pasket commented 6 years ago

Hi,

I have a similar problem. I' running Minemeld in a docker container with image jtschichold/minemeld.

The exception I get is (obfuscate url with xxxx):

2018-08-21T21:05:45 (491)basepoller._polling_loop INFO: Polling misp_cert
2018-08-21T21:05:45 (491)connectionpool._new_conn INFO: Starting new HTTPS connection (1): xxxxx
2018-08-21T21:05:45 (491)basepoller._poll ERROR: Exception in polling loop for misp_cert: Unable to connect to MISP (https://xxxxx). Please make sure the API key and the URL are correct (http/https is required): ('Connection aborted.', error(111, 'Connection refused'))
Traceback (most recent call last):
  File "/opt/minemeld/engine/0.9.50.post1/local/lib/python2.7/site-packages/minemeld/ft/basepoller.py", line 721, in _poll
    performed = self._polling_loop()
  File "/opt/minemeld/engine/0.9.50.post1/local/lib/python2.7/site-packages/minemeld/ft/basepoller.py", line 571, in _polling_loop
    iterator = self._build_iterator(now)
  File "/opt/minemeld/local/library/116cf440-1f13-4c4c-bc6f-e4b23fa4c63e/mmmisp/node.py", line 129, in _build_iterator
    misp = PyMISP(self.url, self.automation_key, **kwargs)
  File "/opt/minemeld/engine/0.9.50.post1/local/lib/python2.7/site-packages/pymisp/api.py", line 108, in __init__
    raise PyMISPError('Unable to connect to MISP ({}). Please make sure the API key and the URL are correct (http/https is required): {}'.format(self.root_url, e))
PyMISPError: Unable to connect to MISP (https://xxxxxxxx). Please make sure the API key and the URL are correct (http/https is required): ('Connection aborted.', error(111, 'Connection refused'))

My prototype:

Configuration:

source_name: misp.cert
url: https://xxxxxxxxxxx
filters:
  published: 1
  tag: 'tlp:white'
indicator_types: null
honour_ids_flag: true
verify_cert: false
client_cert_required: false
age_out:
  sudden_death: true
  default: null
attributes:
  confidence: 70
  share_leve: white

Inside the container:

$ docker exec -it minemeld-docker_minemeld_1 /bin/bash

Check connectivity:

root@c5db804101c2:/# curl -vs https://xxxxxxxx
* Rebuilt URL to: https://xxxxxx/
* Hostname was NOT found in DNS cache
*   Trying 172.19.32.20...
* Connected to proxy.mycompany.com (172.19.xxx.xxx) port 8080 (#0)
* Establish HTTP proxy tunnel to xxxxxxxxx:443
> CONNECT xxxxxx:443 HTTP/1.1
> Host: xxxxx:443
> User-Agent: curl/7.35.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
<
* Proxy replied OK to CONNECT request
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using AES256-SHA256
* Server certificate:
*    subject: C=ES; L=xxxx; O=xxxxxx; OU=Departamento de Sistemas; CN=*.xxxxxx
*    start date: 2017-03-16 00:00:00 GMT
*    expire date: 2019-06-15 12:00:00 GMT
*    subjectAltName: xxxxxx matched
*    issuer: C=US; O=xxx Inc; CN=xxx SHA2 Secure Server CA
*    SSL certificate verify ok.
> GET / HTTP/1.1
> User-Agent: curl/7.35.0
> Host: xxxxx
> Accept: */*
>
< HTTP/1.1 302 Found
< Date: Tue, 21 Aug 2018 21:09:06 GMT
* Server Apache is not blacklisted
< Server: Apache
< X-Permitted-Cross-Domain-Policies: none
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< X-Frame-Options: sameorigin
< Strict-Transport-Security: max-age=17280000
< Location: https://xxxxxx/users/login
< Content-Length: 0
< Content-Type: text/html; charset=UTF-8
<
* Connection #0 to host proxy.mycompany.com left intact

I've copied API key several times to ensure it's correct.

Thanks.

pasket commented 6 years ago

Hi,

I found the problem.

I saw that I had connectivity and the proxy was configured in the docker container. To do this I launched the container sending the proxy environment (as below) and I checked with curl.

environment:
            - HTTPS_PROXY=https://myproxy:1234

Apparently everything was correct. However, the message was still there: PyMISPError: Unable to connect to MISP (https://xxxxxxxx)

I've edited inside the container /opt/minemeld/engine/0.9.50.post1/local/lib/python2.7/site-packages/pymisp/api.py filed and added some print. I realized that proxies variable was None.

When I hardcoded proxies = { 'https': 'https://myproxy:1234' } in the __init__ function everything started working.

What I haven't solved yet is why this Miner doesn't get the proxy value from the system.

PaloAltoNetworks / minemeld-misp

Problem pulling data from misp #2