search

fleetdm / fleet

Open-source platform for IT, security, and infrastructure teams. (Linux, macOS, Chrome, Windows, cloud, data center)
https://fleetdm.com
Other
3.12k stars 431 forks source link

Include last modified timestamp for query results per host #22406

Open ddribeiro opened 1 month ago

ddribeiro commented 1 month ago
JoStableford commented 1 month ago

Linked to Unthread ticket:

Inquiry about Query Result Timestamp Behavior #3004)

ddribeiro commented 1 month ago

Similar to #21045

noahtalerman commented 1 month ago
  • @noahtalerman: User requested this because TODO
    • @noahtalerman: In the interim TODO
    • @noahtalerman: Eventually TODO

Hey @ddribeiro do we have another Gong snippet in which the customer explains why they want the modified timestamp?

I'm trying to fill out the "User requested this because" and it's not clear, from the snippet, why they're asking for it.

ddribeiro commented 1 month ago

@noahtalerman I'm going to search previous Gong recordings to see if it was discussed in more detail, but I think this was primarily relayed to us over Slack.

When returning a response from /api/v1/fleet/queries/:id/report each result will contain when the values for a particular host was last_fetched but not when it was last modified. It would be great if fleet kept track of when each value was last modified in addition to when it was last fetched. It would allow automation which monitors the data in the query to invoke only once instead of for every time fleet gets new data

If we want to dig in a bit more, we can discuss it on our next call with the customer.

noahtalerman commented 1 month ago

When returning a response from /api/v1/fleet/queries/:id/report each result will contain when the values for a particular host was last_fetched but not when it was last modified.

Hey @mostlikelee for query reports API is the last_fetched time also the last modified time?

My understanding: Fleet stores the latest result for each host. And, if Fleet gets some new data from a host it will update the last_fetched time. Do we update the last_fetched time if the data doesn't change? If not, the last_fetched time would also be the last_modified time.

cc @ddribeiro

mostlikelee commented 1 month ago

Fleet stores the latest result for each host. And, if Fleet gets some new data from a host it will update the last_fetched time.

This is correct.

Do we update the last_fetched time if the data doesn't change?

Yes. last_fetched is updated every time Fleet receives a new query result for a host.

Also keep in mind that query results are discarded when a report hits the max rows limit (1000 rows by default).

noahtalerman commented 1 month ago

I think this was primarily relayed to us over Slack

Hey @ddribeiro can you please share the Slack thread? Please add it to the top of the issue description by the Gong link.

It would allow automation which monitors the data in the query to invoke only once instead of for every time fleet gets new data

I'm specifically interested in what this "automation" does.

noahtalerman commented 1 month ago

Hey @ddribeiro, just giving you another ping! Can you please share the Slack thread when you get the chance? Thanks :)

ddribeiro commented 1 month ago

@noahtalerman https://fleetdm.slack.com/archives/C072L58U878/p1727295855196009?cid=C072L58U878&thread_ts=1727181622.337299

noahtalerman commented 1 month ago

Moved the original issue description here for safekeeping:

Problem

As a Fleet admin with automations that are triggered based on query results, I'd like to know when the results for a query were last modified from their previous values.

This would allow automations to trigger only once when results have changed instead of every time Fleet gets new data.

What have you tried?

I checked the API response GET /api/v1/fleet/queries/:id/report and saw each result contains values for when a particular host was last_fetched but not last_modified.

Potential solutions

Fleet could log a timestamp for when query results were last modified from their previous values and return that timestamp in response for query related API endpoints.

What is the expected workflow as a result of your proposal?

As a result of this workflow, a customer using automations based on query results would use the `last_modified` timestamp to detect if query results have changed since the last time the automation has run. If the `last_modified` timestamp is more recent than the last time the automations ran, then the automations will run again. Otherwise, the automations will not run as they are not needed.
noahtalerman commented 1 month ago

Thanks for the Slack thread @ddribeiro.

Even w/ the Slack thread, the problem that the user is having is not clear. I think this is because the user isn't the one who sent the Slack message. It sounds like the request is coming from someone else.

@pintomi1989 if you think we should think about this one some more can you please follow up w/ the requestor?