Important: GPU-enabled node pools in the October 2022 preview update - Githubissues

Azure / aksArc

# Welcome to the Azure Kubernetes Service on Azure Stack HCI repo This is where the AKS-HCI team will track features and issues with AKS-HCI. We will monitor this repo in order to engage with our community and discuss questions, customer scenarios, or feature requests. Checkout our projects tab to see the roadmap for AKS-HCI!

MIT License

109 stars 45 forks source link

Important: GPU-enabled node pools in the October 2022 preview update #272

Open baziwane opened 1 year ago

baziwane commented 1 year ago

Customers deploying the October 2022 GPU preview update will notice that the node pool will fail to start after creating a workload cluster.

Symptoms

The node pool will not start with most pods failing due to missing /usr/bin/nvidia-container-runtime (this is in the containerd/config.toml)

Root cause

A new base package was published by Nvidia that contains key files that are missing in the October build ( https://github.com/NVIDIA/nvidia-container-toolkit/commit/8de95932091c0f936b268477e16a1c62afa4483e)

Workaround

SSH into each worker node
Run sudo su
Run tdnf install nvidia-container-toolkit-base

Recommendations

Install the September 2022 update if you want to run GPU enabled node pools
For customers already on September 2022 GPU preview or earlier, we recommend delaying upgrading until a fix is made available in an upcoming release.
Use the workaround below if you are already running on the October 2022 GPU preview update.