search

elastic / integrations

Elastic Integrations
https://www.elastic.co/integrations
Other
42 stars 451 forks source link

[Infoblox BloxOne DDI] Use of Fingerprint Processor #7842

Open MakoWish opened 1 year ago

MakoWish commented 1 year ago

The Infoblox BloxOne DDI integration currently uses a Fingerprint processor on events, and the timestamp of events is being set based on the created_at value of the event. This is causing visibility issues into events, because if an object was created in Infoblox a year ago, you need to set the time window in Kibana back to a year to see this object. We also ran into this same issue with the Tenable.io integration, and the Fingerprint processor was ultimately removed. I think the issue is compounded by the fact that events are queried based on the "created_at" value, so once an object is created, it will only be queried once, so any changes to that object will likely never be seen in Elastic.

I honestly do not know the best solution to resolve these issues, but I feel a discussion should be started to help figure out the best way to get accurate and up-to-date data ingested.

Eric

elasticmachine commented 1 year ago

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

bhapas commented 1 year ago

@MakoWish

The @timestamp is using last_updated value.

https://github.com/elastic/integrations/blob/80579aa022841f78d5020a274ef103249f6af810/packages/infoblox_bloxone_ddi/data_stream/dhcp_lease/elasticsearch/ingest_pipeline/default.yml#L139-L142

Looks like the fingerprint processor in Infoblox BloxOne DDI is using a combination https://github.com/elastic/integrations/blob/80579aa022841f78d5020a274ef103249f6af810/packages/infoblox_bloxone_ddi/data_stream/dhcp_lease/elasticsearch/ingest_pipeline/default.yml#L23-L29

This creates new document every time it is updated to a new value and fingerprint is unique and document should be ingested with latest update time as the timestamp and queryable using this time.

Am I missing something here?