search

rust-lang / infra-team

Coordination repository for the Rust infra team
https://www.rust-lang.org/governance/teams/infra
Apache License 2.0
20 stars 10 forks source link

Automate server updates #134

Open MarcoIeni opened 2 months ago

MarcoIeni commented 2 months ago

Updating our servers (e.g. the dev-desktops) at the moment consists in logging in into the machines and running apt commands.

Automate packages update

A first step would be writing a cronjob that tries to run:

sudo apt update && sudo apt upgrade && sudo apt autoremove && sudo reboot now

This can run periodically. Not too often otherwise it would be bad for devs productivity. Say 1 per month.

EDIT: not sure we want to reboot because it would imply downtime.

Automate ubuntu update

It would be great if we could automate the distro upgrade too (e.g. 22 -> 24). Running sudo do-release-upgrade manually on every server takes a lot of time. It would be cool if we had a way to define the Ubuntu version from IaC (e.g. terraform).

At the moment, terraform thinks that we are running ubuntu, while in reality we are in the process of manually upgrading to ubuntu 24 everywhere.

Challenges

This is challenging, because at the moment the home volume is not separated from the root one, so upgrading the OS image from terraform would probably result in rustc developers loosing their files.

Tests

Ideally we should have some tests or deployment strategy so that we check that the system is working before rotating VMs.

EDIT: we probably don't want to automate do-release-upgrade because there are breaking changes, such as packages missing.

Kobzol commented 2 months ago

What about using ubuntu's own system for automating package updates (https://phoenixnap.com/kb/automatic-security-updates-ubuntu? Seems a bit more elegant than running a cron job for this.

MarcoIeni commented 2 months ago

oh, I didn't know about it. Definitely!

EDIT: another idea suggested here is to have two ansible playbooks, one for updates and one for the server initialization.

We could run the update playbook periodically from CI.