akash-network / support

Akash Support and Issue Tracking
Apache License 2.0
5 stars 4 forks source link

docs: providers must disable unattended upgrades #131

Closed andy108369 closed 11 months ago

andy108369 commented 1 year ago

Ref. https://discord.com/channels/747885925232672829/1111749248527114322/1157754908624298074


@everyone :boom: Providers! Especially the GPU-providers (not limited to :warning: )

Make sure you have disabled the unattended upgrades!

Unattended upgrades can bring all sorts of uncertainty/troubles such as upgrade your nvidia drivers and "lock-up" your K8s cluster. (nvidia-smi will hang on the host/pod; nvdp plugin will stuck and hence K8s cluster will be running in a non-desired state where closed deployments will be stuck in Terminating status)

This impacts me, what do I do now?

Check your provider, and if you experience any of these issues (nvidia-smi not hangs, pods stuck in Terminating state), then just reboot your impacted K8s nodes, preferably after disabling the unattended upgrades (see next step).

How to disable the unattended upgrades?

To disable the unattended upgrades, execute these two commands on your worker & control plane Ubuntu/Debian-based nodes:

echo -en 'APT::Periodic::Update-Package-Lists "0";\nAPT::Periodic::Unattended-Upgrade "0";\n' | tee /etc/apt/apt.conf.d/20auto-upgrades

apt remove unattended-upgrades

Verify

These commands should output 0 like in this example:

# apt-config dump APT::Periodic::Unattended-Upgrade
APT::Periodic::Unattended-Upgrade "0";

# apt-config dump APT::Periodic::Update-Package-Lists
APT::Periodic::Update-Package-Lists "0";

cc @ScottCarruthers#8207

cc @chainzero

chainzero commented 1 year ago

Akash doc site entry has been made for instructions listed in this issue here:

https://docs.akash.network/providers/build-a-cloud-provider/akash-cloud-provider-build-with-helm-charts/step-11-disable-unattended-upgrades

arno01 commented 11 months ago

Verified the nodes and applied where applicable:

Have updated the internal doc for Ubuntu Jammy installation config.