search

fleetdm / fleet

Open-source platform for IT, security, and infrastructure teams. (Linux, macOS, Chrome, Windows, cloud, data center)
https://fleetdm.com
Other
3.16k stars 432 forks source link

Add "after" parameter to API endpoints that return timestamped results #21765

Open ddribeiro opened 2 months ago

ddribeiro commented 2 months ago

Problem

customer-honoria has a workflow where they pull data from the Fleet API into a homegrown system they use for alerting. They are using the API endpoints for Activities, Hosts, Software, Vulnerabilities and would like to only ingest data that is not already in their system.

For example, the current response for Activities appears to include all activities that have occurred since the start of a Fleet server. This is a large and continuously growing response to ingest into a system multiple times a day.

What have you tried?

The customer looked for a way they could query the API to respond only with items after a certain date. They were able to find one for the hosts endpoint, but are also ingesting Activities, Software, and Vulnerabilities.

Potential solutions

Limiting results in the API response by date seems like a good way to address this.

Add an after parameter to any API endpoint that returns timestamped results. Similar to what is already available for the List hosts endpoint.

What is the expected workflow as a result of your proposal?

A Fleet admin would be able to specify an `after` date to limit the results of an API response to include only items that are new since the last time the API was queried. They would then ingest the results into their system, which would only include items that are new since the last time the workflow was run. This makes the workflow more efficient and less to filter on the client side.
JoStableford commented 2 months ago

Related to a Slack conversation

noahtalerman commented 2 months ago

customer-honoria has a workflow where they pull data from the Fleet API into a homegrown system they use for alerting.

Hey @ddribeiro, what's an example when this workflow is used?

My guess: customer-honoria polls the API and the system is built to trigger an event (ex. webhook) if a new failed login activity is found.

If that's close to reality, do you think customer-honoria could use Fleet's activity automations (webhooks) instead of polling the API?

zayhanlon commented 2 months ago

@ddribeiro @pintomi1989 we're not able to take this ask into the upcoming sprint due to capacity but please bring this back if you feel it becomes workflow blocking in the short term. thank you!