Remotely configure fleetd update channels in agent options

[mikermcneil]

Goal

User story
As a Fleet user,
I want to provide the update channels for Orbit and osquery (two components of fleetd) remotely
so that I deploy one package instead of having to generate a unique package per team.

Example team YAML of a customer w/ self-managed agent updates:

apiVersion: v1
kind: team
spec:
  name: deploy-s4
  agent_options:      
    update_channels:
      osqueryd: s4
      orbit: s4

Example team YAML of a customer w/ Fleet managed agent updates:

apiVersion: v1
kind: team
spec:
  name: Servers (canary)
  agent_options:      
    update_channels:
      osqueryd: edge
      orbit: edge

apiVersion: v1
kind: team
spec:
  name: Servers (production)
  agent_options:      
    update_channels:
      osqueryd: 5.10.2
      orbit: stable

Changes

• Fleet Premium only

Product

• [ ] CLI usage changes: Add an update_channels object to agent_options in the config and team YAML.
  • Add update_channels key to the YAML validation. Error if the key is specified but empty or null.
  • update_channels accepts osqueryd and orbit. Each key updates their respective update channel (ex. osqueryd updates osqueryd-channel. Add these keys to the YAML validation. Error if either key is empty or null.
  • If only one of the channels are specified in the YAML, the channel that's not specified is set to stable (default).
  • If a host is on a team, this host receives the update_channels set for it's team.
• [ ] fleetd changes:
  • Add validation. If one or both of the update channels are invalid or don't exist, orbit logs errors until the update channel is changed to a valid channel in the Fleet YAML or the self-managed TUF repo
  • Add validation to fleetd startup. If the osquery and Orbit versions set remotely (in Fleet YAML) are below the fleetd agents current version, the agent is downgraded and fleetd logs a clear warning message that the agent was downgraded.
• [ ] UI changes: On the Settings > Agent options page, the YAML accepts the CLI changes specified above.
• [ ] Permissions changes: Only admins and GitOps roles (global and team) can edit agent options. Only admins (global and team) can read agent options.
• [ ] Outdated documentation changes: Add a new "Update channels" section to agent configuration documentation reference.
  • Make sure to document what the default behavior is (if not specified) and what the behavior is if the channel doesn't exist.
• [ ] Changes to paid features or tiers: Update pricing page

Engineering

• [ ] REST API changes: TODO
• [ ] Database schema migrations: TODO
• [ ] Documentation changes complete

QA

Risk assessment

• Requires load testing: No (TODO: confirm this wehn bringint into sprint)
• Any risk of accidental log spew come to mind with regards to this change? Yes / No TODO
• Risk level: Low (TODO: confirm this wehn bringint into sprint)
• Risk description: TODO

Manual testing steps

• On all scenarios, fleetd must be tested in the three OSs.
• On all scenarios fleetd should be tested with and without Fleet Desktop (prioritizing testing without Fleet Desktop enabled).
• The changes are in branch 13825-remotely-configure-fleetd-update-channels or in main if already merged.

Scenarios to test

A. New fleetd with latest released fleet server 4.42.0. Should all behave the same as before (no channels should be updated in fleetd).

1. Deploy fleet fleet-v4.42.0.
2. Build orbit v1.19.0 by running main.sh on fleet-v4.42.0.
3. Checkout 13825-remotely-configure-fleetd-update-channels/main and compile+push orbit:

   GOOS=darwin GOARCH=amd64 go build -o orbit-darwin ./orbit/cmd/orbit
   ./tools/tuf/test/push_target.sh macos orbit orbit-darwin 43
   GOOS=linux GOARCH=amd64 go build -o orbit-linux ./orbit/cmd/orbit
   ./tools/tuf/test/push_target.sh linux orbit orbit-linux 43
   GOOS=windows GOARCH=amd64 go build -o orbit.exe ./orbit/cmd/orbit
   ./tools/tuf/test/push_target.sh windows orbit orbit.exe 43

4. Fleetd should auto-update without issues.

B. New fleetd with new fleet server 4.43.0 without update_channels configuration.

1. Perform steps in A, followed by upgrading fleet from 4.43.0 to 13825-remotely-configure-fleetd-update-channels/main
2. fleetd and fleet should work without issues.

C. New fleetd with new fleet server 4.43.0 with update_channels configuration.

1. Perform steps in A+B and then set update_channels.
2. The update_channels feature should work as documented.

D. fleetd v1.19.0 with new fleet server v4.43.0 without update_channels configuration.

E. fleetd v1.19.0 with new fleet server v4.43.0 with update_channels configuration.

F. Set an unexistent channel in the configuration for all three orbit, osqueryd anddesktop`. Thereafter, change back to an existent channel and it should work as expected.

G. Attempt to set a null update_channels key in agent settings. By null we mean update_channels: <new line>

H. Attempt to set a null orbit, osqueryd and desktop in update_channels, should fail. By null/empty we mean orbit: <new line> or osqueryd: ''<new line>.

I. Set configuration on teams, only team hosts should pick up such configuration.

J. Specify only two channels, e.g. orbit and osqueryd but not desktop, therefore desktop should be configured to `stable.

K. Test that the setting cannot be set in Fleet Free.

L. Change setting to a channel that doesn't exist (test the three components: orbit, osqueryd and desktop). Then push to such channel (effectively creating such channel). Eventually fleetd should auto-update (without requiring restart). To create a channel you just push a component to it:

GOOS=darwin GOARCH=amd64 go build -o orbit-darwin ./orbit/cmd/orbit
./tools/tuf/test/push_target.sh macos orbit orbit-darwin new-channel-name

M. Set desktop channel and check that when using fleetd with Fleet Desktop disabled doesn't cause any issues. (Same thing, but without setting desktop: at all.)

N. Test nothing happens/break if configuring update_channels and fleetd agents were built with --updates-disabled=true.

O. Reproduce auto-update startup loop described in the agent configuration docs added in the PR and test the documented ways to fix it.

Confirmation

1. [X] Engineer (@lucasmrod): Added comment to user story confirming succesful completion of QA.
2. [x] QA (@____): Added comment to user story confirming succesful completion of QA.

[noahtalerman]

Zay: # 1 for blanco

[noahtalerman]

Zay: Part of packs => teams migration for blanco. Currently in "testing" phase.

[noahtalerman]

@noahtalerman get w/ Mike to understand this. Do we air guitar this?

[noahtalerman]

Feature fest: Blocking the customer from moving past "testing" phase for teams. Let's air guitar this.

[noahtalerman]

Hey @zayhanlon heads up, we pulled this into the upcoming design sprint as an air guitar.

[noahtalerman]

For future: Target osquery flags based on label. Target osquery flags based on device attribute.

apiVersion: v1
kind: team
spec:
  name: deploy-stage-4
  agent_options:      
    command_line_flags:
      disable_watchdog: false
      logger_path: /path/to/logger
      osqueryd_update_channel: stage-4
      orbit_update_channel: stage-4
    command_line_flags:
      labels:
        - Hosts with docker installed
      logger_path: /path/to/different/logger

[noahtalerman]

Noah: Any reason to limit which flags you can remotely configure? For example, what happens when I update --enroll-secret?

Zach:

• Why we might not want to allow updating all fleetd flags at first.
  • Fleetd has read/write features while osquery doesn't.
  • We don't want users to enable script execution remotely.
  • You could get fleetd to install something malicious.
  • Folks can shoot themselves in the foot by downgrading osquery
• There's some fleetctl package only flags: --local-wix-dir and --type, --notarize, --identifier
• Zach: Some flags weren't built with the idea of updating after initial installation so we should test this to make sure we don't have to change the way Orbit works.

Noah: Start w/ update channels. Come back to other options later.

[rachaelshaw]

@zayhanlon this issue didn't make it into the current sprint, bringing this back to Feature Fest.

[lucasmrod]

@noahtalerman

fleetd changes: Add validation to fleetd startup. If one or both of the update channels don't exist, fleetd logs an error and doesn't change it's channels.

Actually what will happen is: orbit will write errors to the log about the inexistent/invalid channel/s and then will not auto-update the component/s until the channel is changed in Fleet's update_channels setting or the user pushes something to the channel.

[lucasmrod]

@xpkoala I've just added manual steps to test this feature.

[noahtalerman]

orbit will write errors to the log about the inexistent/invalid channel/s and then will not auto-update the component/s until the channel is changed in Fleet's update_channels setting or the user pushes something to the channel.

@lucasmrod makes sense. I updated this story's description to the following so that the expected behavior is QA'd:

fleetd changes: If one or both of the update channels are invalid or don't exist, orbit logs errors until the update channel is changed to a valid channel in the Fleet YAML or the self-managed TUF repo

Please feel free to correct it if it's inaccurate.

[noahtalerman]

Still TODO:

• Remove mention of "Fleet's osquery installer" in docs: https://github.com/fleetdm/fleet/pull/15872/files#diff-8418e1a527974b8181195b29ac3127e342a2317dd5b7e22f2097c4d583ee7783R287
• Pricing page update

[lucasmrod]

Remove mention of "Fleet's osquery installer" in docs: https://github.com/fleetdm/fleet/pull/15872/files#diff-8418e1a527974b8181195b29ac3127e342a2317dd5b7e22f2097c4d583ee7783R287

Agree, we should also update that on the command_line_flags option (and docs).

[lucasmrod]

Do you want me to update this now? Or is someone else doing this?

[noahtalerman]

Remove mention of "Fleet's osquery installer" in docs: https://github.com/fleetdm/fleet/pull/15872/files#diff-8418e1a527974b8181195b29ac3127e342a2317dd5b7e22f2097c4d583ee7783R287

Agree, we should also update that on the command_line_flags option (and docs).

@lucasmrod please go ahead and make these updates! Thanks.

TODO @noahtalerman: Update the pricing page

[lucasmrod]

OK, remove it from the docs only?

Do we want to remove the comment from the app too?

[noahtalerman]

@lucasmrod hey! Sorry I missed your latest comment.

OK, remove it from the docs only?

Do we want to remove the comment from the app too?

I think we want to remove it from both the docs and the UI.

That said, I think let's bring this change through the normal feature fest => drafting => implementation process. I filed a feature request and added it to feature fest here: https://github.com/fleetdm/fleet/issues/16512

This way, we can take them time to check if it makes sense to make other changes too (like removing the command_line_flags comment.

[noahtalerman]

Pricing page update is here: https://github.com/fleetdm/fleet/pull/16513

[noahtalerman]

Hey @Patagonia121 and @pintomi1989 this customer feature request was shipped in Fleet 4.43

[noahtalerman]

C&C: Let's close after the pricing page update is merged in.

[lucasmrod]

@noahtalerman Friendly reminder to close :)

[fleet-release]

Configured remotely, Fleetd channels bring updates, One package, not many.