search

fleetdm / fleet

Open-source platform for IT, security, and infrastructure teams. (Linux, macOS, Chrome, Windows, cloud, data center)
https://fleetdm.com
Other
3.05k stars 422 forks source link

Add functionality to splay policy queries #4048

Open macEar opened 2 years ago

macEar commented 2 years ago

Goal

Idk if all policy queries currently run at the same time or not. If they do, it'd be cool to add functionality to splay queries within some interval specified in percentages. The same as --schedule_splay_percent osquery flag does (https://osquery.readthedocs.io/en/3.4.0/installation/cli-flags/).

The main purpose of this feature is to lower total osquery cpu/memory utilization.

Also it'd nice if we can see performance impact of those queries like we see it for regular queries:

image

noahtalerman commented 2 years ago

Idk if all policy queries currently run at the same time or not

@chiiph do all policy queries, for an individual host, run at the same time?

it'd nice if we can see performance impact of those queries

This makes sense as a potential future improvement to policies in Fleet.

@macEar are policies currently consuming too much cpu/memory on your hosts? Here, by "too much" I mean you're hesitant to continue using the policy feature.

If not, I think it maybe makes sense to 1st reveal the performance impact of policy queries and 2nd provide configuration to lower resource utilization of policies.

macEar commented 2 years ago

Policies are very useful, and all performance impact is caused by queries. So, I'll continue using the policy feature for sure.

It definetly makes sense to follow your suggestion and 1st reveal the impact and then configure.

But nevertheless we have no tool to monitor policy queries performance like with regular queries and osquery_schedule osquery table. Policy queries are ran in ad-hoc mode and then aren't displayed in osquery_schedule. The only option is query profiling, maybe it's sufficient. Im not sure if policy query monitoring is feasible.

chiiph commented 2 years ago

@chiiph do all policy queries, for an individual host, run at the same time?

The are sent to osquery at the same time. How they get executed on the host varies, I believe.

macEar commented 2 years ago

May we add functionality to specify a cron schedule for each policy we have. If we have lots of policies and as @chiiph said they are sent at the same time, we may face performance issues as when osquery reports:

osqueryd worker (27676) stopping: Maximum sustainable CPU utilization limit exceeded: 12

I've profiled each policy query and they are really light, not heavy. So, there osquery watcher probably killed osquery because of the policy queries' simultaneousness. May we investigate this question?