Add a check and update existing Pathways clusters to use CloudDNS. - Githubissues

AI-Hypercomputer / xpk

xpk (Accelerated Processing Kit, pronounced x-p-k,) is a software tool to help Cloud developers to orchestrate training jobs on accelerators such as TPUs and GPUs on GKE.

Apache License 2.0

70 stars 18 forks source link

Add a check and update existing Pathways clusters to use CloudDNS. #151

Closed RoshaniN closed 3 months ago

RoshaniN commented 4 months ago

Fixes / Features

Update existing Pathways clusters to use CloudDNS without user intervention.
This would avoid DNS resolution failures on existing Pathways clusters.
Add a check to find if the cluster is using CloudDNS or not and upgrade automatically, as needed.
Upgrade reflects after 3 steps:
- Step 1: Update cluster dns to use cloud dns
- Step 2: Update GKE control plane version
- Step 3: Update nodepools, which recreates nodes.

Testing / Documentation

Upgrade existing cluster during the first "cluster create-pathways".
Upgrade existing cluster during the first "workload create-pathways".
No extra steps needed from the user

[ y] Tests pass
[ y ] Appropriate changes to documentation are included in the PR