search

OpenCTI-Platform / connectors

OpenCTI Connectors
https://www.opencti.io
Apache License 2.0
389 stars 419 forks source link

[MISP] Errors on event recovery #263

Closed Tyrell20 closed 3 years ago

Tyrell20 commented 3 years ago

Description

MISP connector (version 4.2.4) sometimes fails to recover events. From the docker logs I see the following error: "Unknown error: the response is not in JSON". On we

Environment

OS (where OpenCTI server runs): { Red Hat Enterprise Linux Server release 7.9 (Maipo) with Docker Compose } OpenCTI version: { OpenCTI 4.2.4 } OpenCTI client: { Frontend } Other environment details: Elasticsearch 7.11.0 RabbitMQ 3.8.12 Redis 6.0.10 MinIO Latest

Expected Output

Connector always working, as in this case:

INFO:root:Connector` successfully run (2 events have been processed), storing last_run as 1614332592
INFO:root:Reporting work update_received opencti-work--9daad8c2-5983-4fbe-a0c1-ab6e6e55d664
INFO:root:Initiate work for 5ed76dda-9a94-4dcd-b372-112c0a1ea115
INFO:root:Connector last run: 2021-02-26 09:43:12

Actual Output

INFO:root:Fetching MISP events with args: {"tags": {}, "timestamp": 1614335304, "limit": 50, "page": 1}
CRITICAL [api.py:3180 - _check_response() ] Unknown error: the response is not in JSON.
Something is broken server-side, please send us everything that follows (careful with the auth key):
Request headers:
{'User-Agent': 'PyMISP 2.4.138 - Python 3.8', 'Accept-Encoding': 'gzip, deflate', 'Accept': 'application/json', 'Connection': 'keep-alive', 'Cookie': 'MISP-5ed76dda-6dd0-43e5-a3cb-112c0a1ea115=g18dvgruo9vh5oiptbafq62jks', 'Content-Length': '326', 'Authorization': 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', 'content-type': 'application/json'}
Request body:
{"returnFormat": "json", "page": 1, "limit": 50, "tags": {}, "withAttachments": 0, "metadata": 0, "timestamp": 1614335304, "enforceWarninglist": 0, "includeEventUuid": 0, "includeEventTags": 0, "sgReferenceOnly": 0, "includeContext": 0, "headerless": 0, "includeSightings": 0, "includeDecayScore": 0, "includeCorrelations": 0}
Response (if any):
{"name":"An Internal Error Has Occurred.","message":"An Internal Error Has Occurred.","url":"\/events\/restSearch"}
CRITICAL:pymisp:Unknown error: the response is not in JSON.

Additional information

Connector configuration on docker-compose.yml:

connector-misp: image: opencti/connector-misp:latest environment:

smclinden commented 3 years ago

I had the same issue. On the MISP side it was complaining

Error: [InvalidArgumentException] Restsearch queries using GET and no parameters are not allowed. If you have passed parameters via a JSON body, make sure you use POST requests.
Request URL: /events/restsearch
smclinden commented 3 years ago

This reference might help.

https://github.com/MISP/PyMISP/issues/525

Tyrell20 commented 3 years ago

Hello smclinden,

On MISP side (on var/www/MISP/app/tmp/logs/error.log) it seems that I have no errors related to the OpenCTI query.

The DB schema seems OK and the MISP workers are all running.

What else can I check?

Thanks a lot for your support.

smclinden commented 3 years ago

I'm working on revising the connector, now. I'll probably create a PR when I have it working.

smclinden commented 3 years ago

Ok, one thing that can cause this is described in the following link (a POST being rewritten as a GET by mod_rewrite):

https://stackoverflow.com/questions/26728231/post-request-getting-converted-to-get-when-url-rewriting-is-done-in-apache-httpd

AFAIK, if you are using RewriteRule you could either use the flag [R=307] which would cause the Request to be repeated using the same method (e.g. POST) and data or you could combine mod_rewrite and mod_proxy and use the flag [P] which would stop rewriting and cause the Request to be proxied without changes.

This, of course, assumes that you are using a proxy.

Tyrell20 commented 3 years ago

Hello,

I don't use a proxy/reverse proxy on MISP server. I use only a "Redirect permanent" option to redirect from http to https with Apache.

In addition, my MISP instance is integrated with other platforms without problems.

Thank you

smclinden commented 3 years ago

I should modify what I posted, above, to state that the changes that I suggested would apply to browsers but not, likely, APIs.

Assuming that your MISP_URL is not subject to a Redirect but is, in fact, the actual URL, what I wrote would not be applicable.

Tyrell20 commented 3 years ago

I'm working on revising the connector, now. I'll probably create a PR when I have it working.

Hello smclinden,

are you still working on the connector? We are available in case of is needed to do some tests.

Thanks a lot

smclinden commented 3 years ago

Yeah, I got side-tracked by some incident response. I should get back to it by the weekend.

SamuelHassine commented 3 years ago

Hello @smclinden, @Tyrell20,

The problem may be solved in the next release (tomorrow, 4.3.2). Please let me know if it's fixed so we can close this issue.

Kind regards, Samuel

SamuelHassine commented 3 years ago

Seems to be confirmed, closing it, feel free to re-open if the problem persists.

An0Mee commented 1 year ago

I am getting same error with latest version of OCTI 5.8.6. @SamuelHassine

{"timestamp": "2023-06-29T00:09:18.238582Z", "level": "INFO", "name": "pycti.entities", "message": "Listing Threat-Actors with filters null."} {"timestamp": "2023-06-29T00:09:18.242494Z", "level": "DEBUG", "name": "urllib3.connectionpool", "message": "Starting new HTTP connection (1): 192.168.153.131:8080"} {"timestamp": "2023-06-29T00:09:18.366662Z", "level": "DEBUG", "name": "urllib3.connectionpool", "message": "http://192.168.153.131:8080 \"POST /graphql HTTP/1.1\" 200 144"} {"timestamp": "2023-06-29T00:09:18.368246Z", "level": "INFO", "name": "pycti.entities", "message": "Listing Threat-Actors with filters null."} {"timestamp": "2023-06-29T00:09:18.369682Z", "level": "DEBUG", "name": "urllib3.connectionpool", "message": "Starting new HTTP connection (1): 192.168.153.131:8080"} {"timestamp": "2023-06-29T00:09:18.491425Z", "level": "DEBUG", "name": "urllib3.connectionpool", "message": "http://192.168.153.131:8080 \"POST /graphql HTTP/1.1\" 200 144"} {"timestamp": "2023-06-29T00:09:18.685607Z", "level": "DEBUG", "name": "urllib3.connectionpool", "message": "http://192.168.153.131:8080 \"POST /graphql HTTP/1.1\" 200 768"} {"timestamp": "2023-06-29T00:09:18.686441Z", "level": "INFO", "name": "pycti.connector", "message": "Connector registered with ID: 1ce97355-5ced-4163-a6a6-74bb24bf710f"} {"timestamp": "2023-06-29T00:09:18.686977Z", "level": "INFO", "name": "pycti.connector", "message": "Starting ping alive thread"} {"timestamp": "2023-06-29T00:09:18.689688Z", "level": "DEBUG", "name": "pymisp", "message": "GET - https://localhost/servers/getVersion"} {"timestamp": "2023-06-29T00:09:18.690130Z", "level": "DEBUG", "name": "pymisp", "message": ""} {"timestamp": "2023-06-29T00:09:18.690696Z", "level": "DEBUG", "name": "pymisp", "message": "{'User-Agent': 'PyMISP 2.4.167 - Python 3.10', 'Accept-Encoding': 'gzip, deflate', 'Accept': 'application/json', 'Connection': 'keep-alive', 'Authorization': '', 'content-type': 'application/json'}"} {"timestamp": "2023-06-29T00:09:18.691642Z", "level": "DEBUG", "name": "urllib3.connectionpool", "message": "Starting new HTTPS connection (1): localhost:443"} {"timestamp": "2023-06-29T00:09:18.810123Z", "level": "DEBUG", "name": "urllib3.connectionpool", "message": "http://192.168.153.131:8080 \"POST /graphql HTTP/1.1\" 200 252"} {"timestamp": "2023-06-29T00:09:19.383711Z", "level": "DEBUG", "name": "urllib3.connectionpool", "message": "https://localhost:443 \"GET /servers/getVersion HTTP/1.1\" 200 181"} {"timestamp": "2023-06-29T00:09:19.384650Z", "level": "DEBUG", "name": "pymisp", "message": "", "version": "2.4.171", "pymisp_recommended_version": "2.4.171", "perm_sync": true, "perm_sighting": true, "perm_galaxy_editor": true, "request_encoding": ["gzip", "br"], "filter_sightings": true} {"timestamp": "2023-06-29T00:09:19.384849Z", "level": "WARNING", "name": "pymisp", "message": "The version of PyMISP recommended by the MISP instance (2.4.171) is newer than the one you're using now (2.4.167). Please upgrade PyMISP."} {"timestamp": "2023-06-29T00:09:19.385039Z", "level": "DEBUG", "name": "pymisp", "message": "GET - https://localhost/users/view/me"} {"timestamp": "2023-06-29T00:09:19.385177Z", "level": "DEBUG", "name": "pymisp", "message": ""} {"timestamp": "2023-06-29T00:09:19.385901Z", "level": "DEBUG", "name": "pymisp", "message": "{'User-Agent': 'PyMISP 2.4.167 - Python 3.10', 'Accept-Encoding': 'gzip, deflate', 'Accept': 'application/json', 'Connection': 'keep-alive', 'Cookie': 'CAKEPHP=94kbquq1od7ecufe3vljk7361c', 'Authorization': '', 'content-type': 'application/json'}"} {"timestamp": "2023-06-29T00:09:19.528538Z", "level": "DEBUG", "name": "urllib3.connectionpool", "message": "https://localhost:443 \"GET /users/view/me HTTP/1.1\" 200 1568"} {"timestamp": "2023-06-29T00:09:19.529399Z", "level": "DEBUG", "name": "pymisp", "message": "", "User": {"id": "1", "password": "*****", "org_id": "1", "email": "admin@admin.test", "autoalert": false, "invited_by": "0", "gpgkey": null, "certif_public": "", "nids_sid": "4000000", "termsaccepted": false, "newsread": "0", "role_id": "1", "change_pw": false, "contactalert": false, "disabled": false, "expiration": null, "current_login": "1687996919", "last_login": "1687982624", "force_logout": false, "date_created": null, "date_modified": "1687997359"}, "Role": {"id": "1", "name": "admin", "created": "2023-06-28 19:36:59", "modified": "2023-06-28 19:36:59", "perm_add": true, "perm_modify": true, "perm_modify_org": true, "perm_publish": true, "perm_delegate": true, "perm_sync": true, "perm_admin": true, "perm_audit": true, "perm_auth": true, "perm_site_admin": true, "perm_regexp_access": true, "perm_tagger": true, "perm_template": true, "perm_sharing_group": true, "perm_tag_editor": true, "perm_sighting": true, "perm_object_template": true, "default_role": false, "memory_limit": "", "max_execution_time": "", "restricted_to_site_admin": false, "perm_publish_zmq": true, "perm_publish_kafka": true, "perm_decaying": true, "enforce_rate_limit": false, "rate_limit_count": "0", "perm_galaxy_editor": true, "perm_warninglist": false, "permission": "3", "permission_description": "publish"}, "UserSetting": [], "Organisation": {"id": "1", "name": "ORGNAME", "date_created": "2023-06-28 19:59:52", "date_modified": "2023-06-28 19:59:52", "description": "Automatically generated admin organisation", "type": "ADMIN", "nationality": "", "sector": "", "created_by": "0", "uuid": "77e08b5c-80fb-45ef-b185-080dd32c351e", "contacts": null, "local": true, "restricted_to_domain": null, "landingpage": null}} {"timestamp": "2023-06-29T00:09:19.529879Z", "level": "DEBUG", "name": "pymisp", "message": "GET - https://localhost/attributes/describeTypes.json"} {"timestamp": "2023-06-29T00:09:19.530009Z", "level": "DEBUG", "name": "pymisp", "message": ""} {"timestamp": "2023-06-29T00:09:19.530639Z", "level": "DEBUG", "name": "pymisp", "message": "{'User-Agent': 'PyMISP 2.4.167 - Python 3.10', 'Accept-Encoding': 'gzip, deflate', 'Accept': 'application/json', 'Connection': 'keep-alive', 'Cookie': 'CAKEPHP=4nbgggtrutfpoc738lnrbip8fh', 'Authorization': '', 'content-type': 'application/json'}"} {"timestamp": "2023-06-29T00:09:19.770869Z", "level": "DEBUG", "name": "urllib3.connectionpool", "message": "https://localhost:443 \"GET /attributes/describeTypes.json HTTP/1.1\" 200 22194"} {"timestamp": "2023-06-29T00:09:19.775594Z", "level": "INFO", "name": "pycti.api", "message": "Initiate work for 1ce97355-5ced-4163-a6a6-74bb24bf710f"} {"timestamp": "2023-06-29T00:09:20.112567Z", "level": "DEBUG", "name": "urllib3.connectionpool", "message": "http://192.168.153.131:8080 \"POST /graphql HTTP/1.1\" 200 97"} {"timestamp": "2023-06-29T00:09:20.114270Z", "level": "INFO", "name": "pycti.connector", "message": "Connector last run: 2023-06-28T18:39:08.063981+00:00"} {"timestamp": "2023-06-29T00:09:20.114431Z", "level": "INFO", "name": "pycti.connector", "message": "Connector latest event: 2023-06-20T18:59:18+00:00"} {"timestamp": "2023-06-29T00:09:20.114616Z", "level": "INFO", "name": "pycti.connector", "message": "Fetching MISP events with args: {\"timestamp\": 1687273159, \"limit\": 50, \"page\": 1, \"enforce_warninglist\": false}"} {"timestamp": "2023-06-29T00:09:20.114905Z", "level": "DEBUG", "name": "pymisp", "message": "POST - https://localhost/events/restSearch"} {"timestamp": "2023-06-29T00:09:20.115050Z", "level": "DEBUG", "name": "pymisp", "message": "{\"returnFormat\": \"json\", \"page\": 1, \"limit\": 50, \"withAttachments\": 0, \"metadata\": 0, \"timestamp\": 1687273159, \"enforceWarninglist\": 0, \"includeEventUuid\": 0, \"includeEventTags\": 0, \"sgReferenceOnly\": 0, \"includeContext\": 0, \"headerless\": 0, \"includeSightings\": 0, \"includeDecayScore\": 0, \"includeCorrelations\": 0, \"excludeDecayed\": 0}"} {"timestamp": "2023-06-29T00:09:20.115739Z", "level": "DEBUG", "name": "pymisp", "message": "{'User-Agent': 'PyMISP 2.4.167 - Python 3.10', 'Accept-Encoding': 'gzip, deflate', 'Accept': 'application/json', 'Connection': 'keep-alive', 'Cookie': 'CAKEPHP=ggmb9kdcq1sjutektof4211io3', 'Content-Length': '335', 'Authorization': '', 'content-type': 'application/json'}"} {"timestamp": "2023-06-29T00:09:21.439054Z", "level": "DEBUG", "name": "urllib3.connectionpool", "message": "https://localhost:443 \"POST /events/restSearch HTTP/1.1\" 500 115"} {"timestamp": "2023-06-29T00:09:21.440001Z", "level": "CRITICAL", "name": "pymisp", "message": "Unknown error: the response is not in JSON.\nSomething is broken server-side, please send us everything that follows (careful with the auth key):\nRequest headers:\n{'User-Agent': 'PyMISP 2.4.167 - Python 3.10', 'Accept-Encoding': 'gzip, deflate', 'Accept': 'application/json', 'Connection': 'keep-alive', 'Cookie': 'CAKEPHP=ggmb9kdcq1sjutektof4211io3', 'Content-Length': '335', 'content-type': 'application/json'}\nRequest body:\n{\"returnFormat\": \"json\", \"page\": 1, \"limit\": 50, \"withAttachments\": 0, \"metadata\": 0, \"timestamp\": 1687273159, \"enforceWarninglist\": 0, \"includeEventUuid\": 0, \"includeEventTags\": 0, \"sgReferenceOnly\": 0, \"includeContext\": 0, \"headerless\": 0, \"includeSightings\": 0, \"includeDecayScore\": 0, \"includeCorrelations\": 0, \"excludeDecayed\": 0}\nResponse (if any):\n{\"name\":\"An Internal Error Has Occurred.\",\"message\":\"An Internal Error Has Occurred.\",\"url\":\"\/events\/restSearch\"}"} {"timestamp": "2023-06-29T00:09:21.440288Z", "level": "ERROR", "name": "pycti.connector", "message": "Error fetching misp event: Error code 500:\n{\"name\":\"An Internal Error Has Occurred.\",\"message\":\"An Internal Error Has Occurred.\",\"url\":\"\/events\/restSearch\"}"} {"timestamp": "2023-06-29T00:09:21.440543Z", "level": "DEBUG", "name": "pymisp", "message": "POST - https://localhost/events/restSearch"} {"timestamp": "2023-06-29T00:09:21.440704Z", "level": "DEBUG", "name": "pymisp", "message": "{\"returnFormat\": \"json\", \"page\": 1, \"limit\": 50, \"withAttachments\": 0, \"metadata\": 0, \"timestamp\": 1687273159, \"enforceWarninglist\": 0, \"includeEventUuid\": 0, \"includeEventTags\": 0, \"sgReferenceOnly\": 0, \"includeContext\": 0, \"headerless\": 0, \"includeSightings\": 0, \"includeDecayScore\": 0, \"includeCorrelations\": 0, \"excludeDecayed\": 0}"} {"timestamp": "2023-06-29T00:09:21.441395Z", "level": "DEBUG", "name": "pymisp", "message": "{'User-Agent': 'PyMISP 2.4.167 - Python 3.10', 'Accept-Encoding': 'gzip, deflate', 'Accept': 'application/json', 'Connection': 'keep-alive', 'Cookie': 'CAKEPHP=7oldeaeuqudtapn0gvn1ckkjbt', 'Content-Length': '335', 'Authorization': '', 'content-type': 'application/json'}"} {"timestamp": "2023-06-29T00:09:21.743118Z", "level": "DEBUG", "name": "urllib3.connectionpool", "message": "https://localhost:443 \"POST /events/restSearch HTTP/1.1\" 500 115"} {"timestamp": "2023-06-29T00:09:21.744016Z", "level": "CRITICAL", "name": "pymisp", "message": "Unknown error: the response is not in JSON.\nSomething is broken server-side, please send us everything that follows (careful with the auth key):\nRequest headers:\n{'User-Agent': 'PyMISP 2.4.167 - Python 3.10', 'Accept-Encoding': 'gzip, deflate', 'Accept': 'application/json', 'Connection': 'keep-alive', 'Cookie': 'CAKEPHP=7oldeaeuqudtapn0gvn1ckkjbt', 'Content-Length': '335', 'content-type': 'application/json'}\nRequest body:\n{\"returnFormat\": \"json\", \"page\": 1, \"limit\": 50, \"withAttachments\": 0, \"metadata\": 0, \"timestamp\": 1687273159, \"enforceWarninglist\": 0, \"includeEventUuid\": 0, \"includeEventTags\": 0, \"sgReferenceOnly\": 0, \"includeContext\": 0, \"headerless\": 0, \"includeSightings\": 0, \"includeDecayScore\": 0, \"includeCorrelations\": 0, \"excludeDecayed\": 0}\nResponse (if any):\n{\"name\":\"An Internal Error Has Occurred.\",\"message\":\"An Internal Error Has Occurred.\",\"url\":\"\/events\/restSearch\"}"} {"timestamp": "2023-06-29T00:09:21.744205Z", "level": "ERROR", "name": "pycti.connector", "message": "Error fetching misp event again: Error code 500:\n{\"name\":\"An Internal Error Has Occurred.\",\"message\":\"An Internal Error Has Occurred.\",\"url\":\"\/events\/restSearch\"}"} {"timestamp": "2023-06-29T00:09:21.744469Z", "level": "INFO", "name": "pycti.connector", "message": "Connector successfully run (0 events have been processed), storing state (last_run=2023-06-29T00:09:19.775465+00:00, last_event=2023-06-20T18:59:18+00:00, last_event_timestamp=1687273158, current_page=1)"} {"timestamp": "2023-06-29T00:09:21.744618Z", "level": "INFO", "name": "pycti.api", "message": "Reporting work update_processed work_1ce97355-5ced-4163-a6a6-74bb24bf710f_2023-06-29T00:09:19.799Z"} {"timestamp": "2023-06-29T00:09:22.025637Z", "level": "DEBUG", "name": "urllib3.connectionpool", "message": "http://192.168.153.131:8080 \"POST /graphql HTTP/1.1\" 200 107"}