[ti_misp] No data being collected for Threat Attributes

[leandrojmp]

Hello,

I'm trying to use the MISP integration to get the Attributes from our MISP, but no data is being collected.

Since I'm trying to get some historical data, I set the look back interval to get attributes since 2023-01-01, which is almost 110d, and to make sure that the request wouldn't timeout I set the timeout interval to half an hour, 1800s, but even this way no data is being collected and the only logs I have are when I add the integration or remove it, nothing in between.

I have data in our MISP and I used the following python script to test the request and see how long it would take to get this historical data:

import requests

MISP_HOST = "https://MISP-ENDPOINT"
ENDPOINT_ATTRIBUTES = "/attributes/restSearch"
HEADERS = {
    "Accept": "application/json",
    "Content-Type": "application/json",
    "Authorization": "API-KEY"
}
BODY = { 
    "timestamp": "2023-01-01"
}

attributes = requests.post("{}{}".format(MISP_HOST,ENDPOINT_ATTRIBUTES), headers=HEADERS, json=BODY)

It takes around 9 minutes to get everything and this is an example of the data:

>>> len(attributes.json()['response']['Attribute'])
135228
>>> attributes.json()['response']['Attribute'][0]
{'id': '316994', 'event_id': '499', 'object_id': '0', 'object_relation': 'ip-src', 'category': 'Network activity', 'type': 'ip-src', 'to_ids': True, 'uuid': 'cd97298f-2a26-4940-9828-d5d3180efe9c', 'timestamp': '1672542435', 'distribution': '5', 'sharing_group_id': '0', 'comment': 'Source: alienvault', 'deleted': False, 'disable_correlation': False, 'first_seen': None, 'last_seen': None, 'value': '192.241.207.4', 'Event': {'org_id': '1', 'distribution': '3', 'id': '499', 'info': 'Redis bruteforce Attackers [2023-01-01]', 'orgc_id': '1', 'uuid': '32ba355c-9768-3fe7-8cec-43a42bc3cf93'}, 'Tag': [{'id': '63', 'name': 'REDACTED:attack-categorization="Brute Force"', 'colour': '#00b8e0', 'numerical_value': None, 'is_galaxy': False, 'local': False}, {'id': '25', 'name': 'REDACTED:false-positive="low-risk"', 'colour': '#33ff00', 'numerical_value': None, 'is_galaxy': False, 'local': False}, {'id': '12', 'name': 'kill-chain:Reconnaissance', 'colour': '#50003a', 'numerical_value': None, 'is_galaxy': False, 'local': False}, {'id': '15', 'name': 'kill-chain:Exploitation', 'colour': '#a80079', 'numerical_value': None, 'is_galaxy': False, 'local': False}, {'id': '72', 'name': 'misp-galaxy:mitre-attack-pattern="Reconnaissance - TA0043"', 'colour': '#149414', 'numerical_value': None, 'is_galaxy': True, 'local': False}, {'id': '130', 'name': 'misp-galaxy:mitre-attack-pattern="Scanning IP Blocks - T1595.001"', 'colour': '#296d98', 'numerical_value': None, 'is_galaxy': True, 'local': False}, {'id': '79', 'name': 'misp-galaxy:mitre-attack-pattern="Credential Access - TA0006"', 'colour': '#149414', 'numerical_value': None, 'is_galaxy': True, 'local': False}, {'id': '169', 'name': 'misp-galaxy:mitre-attack-pattern="Password Guessing - T1110.001"', 'colour': '#296d98', 'numerical_value': None, 'is_galaxy': True, 'local': False}]}

I already added and removed the integration a couple of times, but still no data is collected and no log besides the logs from adding and removing the integration are generated.

How can I troubleshoot this? There is no logs about what Elastic Agent is doing, but it seems that the first request never finishes.

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[efd6]

@leandrojmp I've sent a change (#5965) to ti_misp that will allow you to examine the request/response conversations between your agent and the MISP server. If you use it to examine this issue, please take note of the warnings in the documentation.

[leandrojmp]

Thanks @efd6, I will wait for it to be merged so I can update the integration.

Currently I'm on 8.5.1, so I think I will be able to use this request tracer feature.

If I understood correctly this will save the requests on a file in the Elastic Agent folder, right?

[efd6]

The logs are written in the working directory of the agent. This will likely change in the future and they will be written into a location that can be collected with elastic-agent diagnostics. This hasn't happened yet.

[andrewkroh]

Several bugs were recently fixed w.r.t. how data is read from the MISP API. It would be good to upgrade to the latest version and re-test. And if there are still problems to troubleshoot then please enable the "request tracer" feature in the integration's advanced settings so we can view the raw requests/responses. Please reopen if you are still having an issue.

[andrewkroh]

FYI the initial_interval takes a duration value whose largest unit type h (hour). So convert anything larger (like days) down to hours.

[leandrojmp]

FYI the initial_interval takes a duration value whose largest unit type h (hour). So convert anything larger (like days) down to hours.

Yeah, that was the main issue in the end, after looking at some files in the integration to try to understand it better I found that the timestamp in the body is created like this:

- set:
    target: body.timestamp
    value: '[[.cursor.timestamp.Unix]]'
    default: '[[ (now (parseDuration "-{{initial_interval}}")).Unix ]]'

I was using 180d in the initial interval, converted it to 4320h it started working, I was using as days because in some places in Kibana you can use durations like this.

It would help however to have this information in the interface below when adding the integration.

Thanks @andrewkroh!

[andrewkroh]

It would help however to have this information in the interface below when adding the integration.

Agree 💯 . Created https://github.com/elastic/integrations/issues/6684