Equip users who would like to verify fleetd releases

[willmayhone88]

• customer-figali: Gong snippet: https://us-65885.app.gong.io/call?id=7283736297840441495&highlights=%5B%7B%22type%22%3A%22SHARE%22%2C%22from%22%3A150%2C%22to%22%3A960%7D%5D
• @noahtalerman: Users will expect that Fleet's agent (fleetd) releases will be signed with a string that can be accessed via GitHub and later programmatically via GitHub's API.
  • customer-figali: "What's important to us is that GitHub is signing whatever is built (example), -- ideally we're not doing verification ourselves or building custom verification -- rather we want it to just be done with whatever GitHub Actions/GitHub Releases supports natively. We'll probably use this string to use the GitHub command-line tool to the string in our custom script that deploys new versions of agents to laptops. For server images/ECR, we might use a different approach longer-term"
  • "This hello world looks pretty good: https://github.com/fleetdm/fleet/issues/20219#issuecomment-2220285202"
  • Protocol? "not sure if there are good alternatives to SLSA, ecosystem support of SLSA is good, open to ideas if we find something better, github's default support for SLSA would give us a lot of confidence that the release is real"
  • Process of verification: programmatic vs manual? "Not deadset right now, eventually programmatic would be cool (but multiple ways of blasting out the agent depending on target)"

• @lukeheath: Could be a good one for Scott to take to free up capacity

User stories

• 23825

[lukeheath]

@lucasmrod @sharon-fdm Will you please review and consider if it's possible for our fleetd build and release to TUF process to meet these requirements and TUF requirements around security keys? Have we considered leveraging a Cloud HSM?

The challenge here is that per #5 of on the TUF FAQ:

5. Which roles can use online keys? The Timestamp and Snapshot roles can use online keys to facilitate continuous delivery of updates on the typical repository. All other roles should rely on offline keys to prevent attackers from signing for malicious packages in the event of a repo compromise.

Perhaps we could sign the binaries locally, but still use GitHub Actions to upload them?

[noahtalerman]

Luke: Proposed changes to accomplish this:

• New process to pull fleetd down locally, sign it, and upload to GitHub
• Add new GitHub action to push to Fleet’s TUF repo. Runs when fleetd is uploaded to GitHub
  • This way the customer can verify that the fleetd in GitHub is the same fleetd downloaded from TUF server

Alternatively we considered adding a GitHub action to do the signing AND pushing to TUF. We decided not to do this because it violates TUF: keys would be in GitHub (keys have to be local/offline).

[noahtalerman]

cc @willmayhone88 ^

[lucasmrod]

Hi folks!

As far as I can see, this requirement is an addition to the current build process (to add build provenance attestation) and we don't need to modify how how we release to Fleet's TUF.

fleetd has three components: orbit, Fleet Desktop and osqueryd

• orbit and Fleet Desktop components that are uploaded to Fleet's TUF are built by Fleet Github actions.
• osqueryd uploaded to Fleet's TUF is downloaded from osquery's Github release page (its build process is not controlled by Fleet).

We can add build provenance attestation to orbit and Fleet Destop (see the linked docs in the description) by adding a step on our Github actions. And then we could provide tooling/documention so that users/customers can verify the artifacts uploaded to our TUF repository (so they can be sure that what's on Fleet's TUF was built by a Github action).

We should also open an issue on osquery's repository to do the same for osqueryd.

[lucasmrod]

Here's an example:

I have a hello-world repository that builds a hello-world executable and I added attestation to the build CI action: https://github.com/lucasmrod/hello-world/blob/f9e80026cbc6183d7a7c1acdb993743d3569f0a4/.github/workflows/build-hello-world.yml#L36-L39

Then I download the executable from the action artifacts and we can verify that the downloaded executable hasn't been tampered with (as long as we trust Github and organizations member of the Sigstore’s Public Good Instance, which I think is reasonable :):

$ gh attestation verify /Users/luk/Downloads/hello-world --repo lucasmrod/hello-world

Loaded digest sha256:6466805c10276cdf69ebceb90fc3060ed67b152c510166120762dbe77b495ca0 for file:///Users/luk/Downloads/hello-world
Loaded 1 attestation from GitHub API
✓ Verification succeeded!

sha256:6466805c10276cdf69ebceb90fc3060ed67b152c510166120762dbe77b495ca0 was attested by:
REPO                   PREDICATE_TYPE                  WORKFLOW
lucasmrod/hello-world  https://slsa.dev/provenance/v1  .github/workflows/build-hello-world.yml@refs/heads/main

[lucasmrod]

Here are the tasks that I see so far:

• Update github actions that build orbit, Fleet Desktop and the Fleet server (executables and docker images) to do build attestation.
• Create an issue in https://github.com/osquery/osquery to do the same for osquery artifacts.
• Create automation that runs daily that checks for attestation on pushed binaries to TUF and our Fleet server releases.

[JoStableford]

Related to a Slack conversation

[lukeheath]

@lucasmrod Let's make sure to review any attestation we have in place to make sure we're consistent with with the standard we're using. Do we want to adopt SLSA, or do we already have something in place? Want to make sure we don't use two different standards in different places.

[lucasmrod]

Let's make sure to review any attestation we have in place to make sure we're consistent with with the standard we're using.

AFAICS we don't have any build provenance attestation in place for TUF components (https://github.com/fleetdm/fleet/attestations is empty).

Do we want to adopt SLSA, or do we already have something in place?

IMO we should adopt SLSA given we build all our TUF artifacts in Github Actions and Github supports it, see here.

[lukeheath]

@lucasmrod Thanks for the info! Agreed SLSA looks like the way to go if we don't have something else in place.

[mikermcneil]

Focusing on the first bullet here looks to be the way forward once we're ready to build a user story(ies) for this:https://github.com/fleetdm/fleet/issues/20219#issuecomment-2220289174 (the other two can be left out for now, since this is about equipping users with the ability to do the checking themselves)

See updated bullets at top of issue description for reason why.

i.e. @noahtalerman see what you think, but I believe this should do it, provided the way we do it addresses the bullets we added above the line in the issue description:

Update github actions that build orbit, Fleet Desktop and the Fleet server (executables and docker images) to do build attestation.

[noahtalerman]

Moving the original issue description here for safekeeping:

"Add SLSA provenance attestations"

Request: As a security engineer, I want there to be a little string associated with whatever version of fleetd I am about to install, where I could hypothetically use that string to verify (manually or via a programmatic check) there hasn't been a person-in-the-middle-attack that replaced it with something nasty, so that I can describe it as a potential security risk reduction in my CISO's risk register.

Problem

In simple terms, I want to verify that the fleetd uploaded to GitHub is the same fleetd downloaded from TUF server.

Currently, there is no mechanism in Fleet to verify that the binaries we use are built and uploaded using GitHub Actions from the Fleet repository at a specific commit SHA. This creates a potential security risk, as there is no assurance that the code reviewed is the code that is run, leaving room for possible discrepancies or tampering unless GitHub itself is compromised.

Potential solutions

Implement SLSA provenance attestations for the Fleet client binary artifacts and server container image to enable verification that the binaries are built and uploaded using GitHub Actions from the Fleet repository at a particular commit SHA. GitHub now offers native support for SLSA provenance attestations, making it feasible to integrate this feature with relative ease. This addition would enhance the security and integrity of the software by ensuring that the reviewed code matches the deployed binaries, thereby mitigating the risk of supply chain attacks.

See GitHub docs here.

[noahtalerman]

@Patagonia121 heads up that we peeled off this user story and pulled it into the upcoming design sprint.