search

fleetdm / fleet

Open-source platform for IT, security, and infrastructure teams. (Linux, macOS, Chrome, Windows, cloud, data center)
https://fleetdm.com
Other
3.12k stars 431 forks source link

Ensure installers specified in GitOps are what an admin expects, and don't duplicate already-downloaded installers on every GitOps apply #23497

Open iansltx opened 1 week ago

iansltx commented 1 week ago

Problem

Right now Fleet pulls all installers (twice; see #23494) for all teams every time GitOps runs, even if the set of installers is identical. This is necessary because there's no way to specify when an installer is already the correct version without downloading it, since the installer's only identifier is the download URL, and the contents behind the download URL may change.

Another issue here is that, once I have a known-correct installer, if I re-apply GitOps later the installer URL needs to be accessible, and continue returning the correct installer, which is more fragile than the UI-based workflow, where once I upload an installer it's good to go. Thinking as an admin, if I'm using GitOps I want the tools to have GitOps runs be as predictable as possible, versus having an installer change out from under me because GitOps got run.

Split from #22300.

Potential solutions

One solution to allow specification of installer hashes in YAML, using the same hashing mechanism we use for storage IDs for the installers (e.g. have a key named sha256). That would allow us to reuse installers already in object storage rather than needing to re-download, and would provide a higher level of assurance on which installer is being pulled that admins could opt into even after they've pulled installers initially. If we did need to re-download because the hash didn't match what we had in the database (or because we hadn't downloaded the installer yet), a hash mismatch would be treated as an error in the batch upload.

There are other solutions that address the duplicated download (e.g. storing/checking ETags) but allowing admins to opt into providing a hash is the easiest to explain, probably the easiest to implement, and most resilient to varying levels of sophistication for where we're downloading installers from (vs. relying on those servers to provide ETags or return 304s when we indicate that we already have a current version of the installer). This also maintains existing behavior if admins don't opt in, which they wouldn't for installer URLs that intentionally serve the latest version of an installer (and we could add logic to deduplicate downloads for those separately later).

What is the expected workflow as a result of your proposal?

I can specify an installer hash in GitOps YAML, and if all installers I'm referencing have hashes matching previously uploaded installers the batch upload completes quickly without duplication. If hashes don't match, the batch upload fails with a descriptive error message.

lukeheath commented 1 week ago

I'm pulling this off the drafting board to track on the new #help-engineering board.