search

elastic / integrations

Elastic Integrations
https://www.elastic.co/integrations
Other
197 stars 427 forks source link

[TI_MISP] MISP integration does not get everything from MISP and stop getting new data #10064

Open leandrojmp opened 4 months ago

leandrojmp commented 4 months ago

Hello,

We are facing a couple of issues with the MISP integration, sometimes it misses data from MISP and does not bring everything it should bring and it also stops working without providing any error, everytime this happens we need to remove the integration and install it again.

Are there any performance limitations for the httpinput json? We can see the data on MISP, we can get the data using a custom python script, but the MISP integration simple misses it and only a reinstall to make it look back again in time will bring the missed data.

How can we troubleshoot it? Enabling the request tracing would be enough?

elasticmachine commented 3 months ago

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

efd6 commented 3 months ago

@leandrojmp Have you taken a look at the request trace logging from the integration? When you collect the data from the python, can you correlate the data that you do get from that that is missing from the integration collection?

leandrojmp commented 3 months ago

Have you taken a look at the request trace logging from the integration?

We saw no errors when we enabled request trace logging, but we cannot let this enabled because of the amount of logs that it may generate and we also do not know when this issue will happen.

We only know that it missed some documents when we look on MISP and see some attributes, but could not find them on our Elastic.

When you collect the data from the python, can you correlate the data that you do get from that that is missing from the integration collection?

Not sure if I understand, but using a simple python script I can see both the data that was on our Elastic and the documents it missed.

When we find that the integration missed some data we need to remove the integration and add it again telling it to look back in time to see if will now get the missed data, sometime it works, sometime the integration does not bring anything anymore and we need to remove and add it again until it works.

When it does not work we also have no errors on the agent logs nor the request tracing, it simple does not brings anything and the cursor does not change.