search

gravitational / teleport

The easiest, and most secure way to access and protect all of your infrastructure.
https://goteleport.com
GNU Affero General Public License v3.0
17.66k stars 1.77k forks source link

[teleport-update] Add support for version pinning #49307

Closed sclevine closed 1 day ago

sclevine commented 1 day ago

This PR implements manual version pinning for agent auto-updates by adding a pin subcommand to the teleport-update command. This command installs, verifies, and locks the Teleport agent version, preventing teleport-update update from changing the version automatically. teleport-update pin takes the same arguments as teleport-update enable, and can be used to bootstrap an agent installation without auto-updates. The pin subcommand can be run with an explicit version (--force-version), but by default it will pin the cluster's recommended version. Unlike enable, the --force-version flag is not hidden for pin. The teleport-update pin subcommand always queries the proxy to determine enterprise/fips status.

This PR also implements automatic version pinning. If teleport-update update fails due to an issue with the new installation (e.g., crashing), the failed version is recorded in status in update.yaml. That version is skipped until the version is successfully installed with pin or enable.

The teleport-update binary will be used to enable, disable, and trigger automatic Teleport agent updates. The new auto-updates system manages a local installation of the cluster-specified version of Teleport stored in /var/lib/teleport/versions.

RFD: https://github.com/gravitational/teleport/pull/47126 Goal (internal): https://github.com/gravitational/cloud/issues/10289

Example: Pinning the version away from a broken installation

ubuntu@legendary-mite:~$ sudo ./teleport-update pin --force-version 16.4.3
2024-11-21T02:58:53Z INFO [UPDATER]   Initiating installation. target_version:16.4.3 active_version:17.0.1 agent/updater.go:362
2024-11-21T02:58:54Z INFO [UPDATER]   Downloading Teleport tarball. url:https://cdn.teleport.dev/teleport-ent-v16.4.3-linux-arm64-bin.tar.gz size:163411740 agent/installer.go:322
2024-11-21T02:58:59Z INFO [UPDATER]   Extracting Teleport tarball. path:/var/lib/teleport/versions/16.4.3 size:648488960 agent/installer.go:360
2024-11-21T02:59:02Z INFO [UPDATER]   Executing new teleport-update binary to update configuration. agent/updater.go:191
2024-11-21T02:59:04Z INFO [UPDATER]   Systemd configuration synced. unit:teleport-update.timer agent/process.go:255
2024-11-21T02:59:04Z INFO [UPDATER]   Service enabled. unit:teleport-update.timer agent/process.go:272
2024-11-21T02:59:04Z INFO [UPDATER]   Finished executing new teleport-update binary. agent/updater.go:197
2024-11-21T02:59:04Z INFO [UPDATER]   Target version successfully installed. target_version:16.4.3 agent/updater.go:594
2024-11-21T02:59:04Z ERRO [UPDATER]   [stderr] Job for teleport.service failed. agent/process.go:353
2024-11-21T02:59:04Z ERRO [UPDATER]   [stderr] See "systemctl status teleport.service" and "journalctl -xeu teleport.service" for details. agent/process.go:353
2024-11-21T02:59:04Z ERRO [UPDATER]   Error running systemctl. args:[reload teleport.service] code:1 agent/process.go:300
2024-11-21T02:59:04Z WARN [UPDATER]   Service ungracefully restarted. Connections potentially dropped. unit:teleport.service agent/process.go:108
2024-11-21T02:59:04Z INFO [UPDATER]   Monitoring PID file to detect crashes. unit:teleport.service agent/process.go:113
2024-11-21T02:59:18Z INFO [UPDATER]   Backup version set. backup_version:17.0.1 agent/updater.go:620
2024-11-21T02:59:18Z WARN [UPDATER]   More than 2 versions of Teleport installed. Version directory may need cleanup to save space. count:4 agent/updater.go:629
2024-11-21T02:59:18Z INFO [UPDATER]   Configuration updated. agent/updater.go:37
sclevine commented 1 day ago

Realized I forgot to push teleport-update unpin to this branch last night 😅 It's a simple command with no arguments that just sets spec.pinned to false in the config.

https://github.com/gravitational/teleport/pull/49307/files#diff-cf29b17bebb0e24bf48bacda13ae9ab300a3c718ccf38a97ea0cccbb2d5c11dfR400-R417