Upgrading fleet-osquery overwrites custom /etc/default/orbit

[jamesps-ebi]

Fleet version: 4.48.3 Fleetctl version: 4.48.3

Web browser and operating system: Chrome 123.0.6312.122, Ubuntu 22.04.4

Expected behaviour

Installing a new version of fleet-osquery should not touch the /etc/default/orbit file if it has been modified.

💥  Actual behavior

Installing a new version of fleet-osquery over an existing installation overwrites the /etc/default/orbit contents, even if it has been modified by an admin.

🧑‍💻  Steps to reproduce

1. Install an older version of fleet-osquery on Debian/Ubuntu
2. Edit the /etc/default/orbit file
3. Install a newer version of fleet-osquery
4. The /etc/default/orbit edits will be replaced by the default file.

🕯️ More info (optional)

I think this is partly caused by the .deb control files. Specifically the postrm installation script contains the following line:

rm -rf /var/lib/orbit /var/log/orbit /usr/local/bin/orbit /etc/default/orbit /usr/lib/systemd/system/orbit.service /opt/orbit

This would clear the config file upon upgrade. Some other process must be recreating it with the default contents afterwards.

This documentation might help fix the issue: https://www.debian.org/doc/debian-policy/ch-maintainerscripts.html

[sharon-fdm]

@noahtalerman @lucasmrod is that the expected behaviour that we do not modify this file if it was modified by admins? If so, how do we know it was modified?

[noahtalerman]

@sharon-fdm and @lucasmrod I'm not sure what the expected behavior is.

My gut says we should document the current behavior (new installs overwrite /etc/default/orbit.

[noahtalerman]

Thanks for tracking this @jamesps-ebi. Sorry you're running into this.

I would love to better understand your current workflow.

What values are you modifying in /etc/default/orbit?

Also, did you know that you can specify the agent's version via agent options? Docs are here. This way, you can upgrade the agent w/o reinstalling it.

Does this method work for you?

[jamesps-ebi]

@noahtalerman This isn't a massive problem in my environment since we use Puppet config management to keep /etc/default/orbit in a specific state. But I do think this behaviour somewhat goes against general .deb package standards.

Typically when installing a .deb package which provides a default /etc/package/config file, if the file already exists at that location and differs from the one in the package, you would get a prompt asking whether to overwrite or keep the existing file.

I wonder if removing /etc/default/orbit during the postrm script is causing this flow to be skipped.

The reason we're managing /etc/default/orbit manually instead of specifying the values at package generation time is becuase we host the fleet-osquery package in our own custom APT repo so our machines can retrieve the package automatically. This way we don't expose our enrolment token or any other potentially sensitive data in the package repo.

# This file is managed by Puppet. DO NOT EDIT.
# Standard settings. Shouldn't need to change these.
ORBIT_UPDATE_URL=https://tuf.fleetctl.com
ORBIT_ORBIT_CHANNEL=stable
ORBIT_OSQUERYD_CHANNEL=stable
ORBIT_UPDATE_INTERVAL=15m0s
# Implementation specific. These should be changed as appropriate.
ORBIT_FLEET_URL=https://fleet.example.com
ORBIT_FLEET_CERTIFICATE=/etc/ssl/certs/ca-certificates.crt
ORBIT_ENROLL_SECRET=poajfdoinancoqoqno
ORBIT_ENABLE_SCRIPTS=true
ORBIT_FLEET_DESKTOP=false

[noahtalerman]

Thanks @jamesps-ebi!

The reason we're managing /etc/default/orbit manually instead of specifying the values at package generation time is becuase we host the fleet-osquery package in our own custom APT repo so our machines can retrieve the package automatically. This way we don't expose our enrolment token or any other potentially sensitive data in the package repo.

This workflow makes sense. Thanks for sharing.

Typically when installing a .deb package which provides a default /etc/package/config file, if the file already exists at that location and differs from the one in the package, you would get a prompt asking whether to overwrite or keep the existing file.

@lucasmrod, @jamesps-ebi shared this expected behavior with us. Do you know why fleetd does something different? (overwriting always)

[lucasmrod]

Do you know why fleetd does something different? (overwriting always)

No reason. We didn't know of this flow (it's not on our docs.) And at the time, overwriting was probably the initial implementation of deb packages because simplicity.

[noahtalerman]

Thanks @lucasmrod :)

@jamesps-ebi sorry for the delay!

I think this sounds like we're after a best practice for the "config-less fleetd agent deployment" flow for Linux (deb).

Here's an article we wrote on the "config-less" flow for macOS and Windows: https://fleetdm.com/guides/config-less-fleetd-agent-deployment

While frustrating, overwriting /etc/default on Ubuntu is the expected behavior. So, I updated this issue from a bug to a feature request (story).

@jamesps-ebi, if the article I linked to seems like the right place, would you be willing to help add a "Linux" section to the article here? Please feel free to open a PR.

This will help us better understand the best practice workflow so we can build the right improvement.

[noahtalerman]

Hey @jamesps-ebi heads up, this improvement didn't get selected during our latest feature fest ritual.

That said, please feel free to open a PR with the suggested improvement!