Spread out calculation of policies on agents to improve performance when there are lots of policies

[sharon-fdm]

This issue's remaining effort can be completed in ≤1 sprint. It will be valuable even if nothing else ships.

Goal

As the customer of Fleet who uses Fleet policies, I would like Fleet to trigger policy queries on my agents in a way that will use agent's resources in a sustainable way so that the agents will stay healthy and not crash.

Changes

• Currently Fleet agent runs all policies at once
• Since we now have hundreds of them due to CIS benchmarks this may not be a sustainable way and we need to rethink this. TODO: Brainstorm solution options Please refer to #11797 for customer issues and current workaround.

This issue's estimation includes completing:

• [ ] UI changes: TODO
• [ ] CLI usage changes: TODO
• [ ] REST API changes: TODO
• [ ] Permissions changes: TODO
• [ ] Database schema migrations: TODO
• [ ] Outdated documentation changes: TODO
• [ ] Scope transparency changes? TODO
• [ ] Breaking changes requiring major version bump? TODO
• [ ] Changes to paid features or tiers? TODO
• [ ] QA complete?
• [ ] ...

ℹ️  Please read this issue carefully and understand it. Pay special attention to UI wireframes, especially "dev notes".

Context

• Requestor(s): _____

QA

Risk assessment

• [ ] Requires load testing TODO

Risk level: Low / High TODO

Risk description: TODO

Automated:

• Fleet: Cover / Will not cover
• QAWolf: Cover / Will not cover

Manual testing steps

1. Step 1
2. Step 2
3. Step 3

Testing notes

Confirmation

1. [ ] Engineer (@____): Added comment to user story confirming succesful completion of QA.
2. [ ] QA (@____): Added comment to user story confirming succesful completion of QA.

[sharon-fdm]

cc: @zayhanlon @marcosd4h @lucasmrod @zhumo @zwass

[zwass]

One approach to look at might be making osquery take a break between executing distributed queries. IIRC there may actually be a flag that does something like this already?

[lucasmrod]

Yes. --table_delay, though it applies to all queries, not just distributed, and it involves a delay between scans so a query that uses multiple tables will be delayed longer.

https://github.com/osquery/osquery/blob/2e3495837d4fe4db3554c6ef76494bc86c74c099/osquery/sql/virtual_table.cpp#L28C1-L31

(PS: I found it while troubleshooting watchdog killing some of the macOS CIS queries. So this is on my list of things to try as a workaround.)