crowdstrike: timestamp-based filters do not appear to follow their documented semantics

[efd6]

This is for v1.39.0 of the integration.

We have an instance of duplicated alert event ingestion leading to unuseful alerting due to the duplicate. Duplicate events that are collected from the API are de-duplicated by the fingerprinting mechanism, but this does not allow deduplication of events across index roll-over events.

The overall logic flow of the collector is to check if it has a cursor state and use that to make a filtered request to the Crowdstrike API. The cursor has two timestamp values, the last_timestamp (used for setting the starting point for a new polling) and the first_timestamp (used for progression through a CEL input want_more loop). We use the Falcon Query Language > operator with these values to make a filter, filter=timestamp:>TIMESTAMP to start the API from after the TIMESTAMP. The response that we get includes events with the TIMESTAMP indicating that the > operator is behaving as a >= operator. Note that it may be the case that it is working as described, but only at second resolution. The only documentation that I have access to is this which shows timestamps at second resolution.

This means that we will receive duplicate events at the start of the requested interval. It appears that it may also result in data stream non-progression.

We can work around this by filtering for events that have timestamps that are strictly greater than the cursor timestamp than we used in the query.

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)