Fix autoprovisioning with spot nodes - Githubissues

AI-Hypercomputer / xpk

xpk (Accelerated Processing Kit, pronounced x-p-k,) is a software tool to help Cloud developers to orchestrate training jobs on accelerators such as TPUs and GPUs on GKE.

Apache License 2.0

81 stars 23 forks source link

Fix autoprovisioning with spot nodes #187

Open avrittrohwer opened 1 month ago

avrittrohwer commented 1 month ago

Fixes / Features

Fixes workload rendering when using spot, without this change xpk workload create errors like:

[XPK] Waiting for `Creating Workload`, for 0 seconds
error: error parsing /tmp/tmp242uhnfs: error converting YAML to JSON: yaml: line 33: could not find expected ':'
[XPK] Task: `Creating Workload` terminated with code `1`

Adds required pod tolerations when using node auto-provisioning with spot nodes. Without the tolerations cluster autoscaler will not create new spot node pools.

Testing / Documentation

[ y ] Tests pass
[ y, not needed ] Appropriate changes to documentation are included in the PR

Node auto-provisioning with spot

Created a xpk cluster with --spot and autoprovisioning flags.
Created a workload with a different topology than the cluster default.
Observed a nodepool being created with the new workload topology using spot TPU nodes.

Node auto-provisioning without spot

Created a xpk cluster with --spot and autoprovisioning flags.
Created a workload with a different topology than the cluster default and --on-demand flag.
Validated generated YAML does not specify spot node-selector and tolerations
Observed a nodepool being created with the new workload topology using on-demand TPU nodes.

Not auto-provisioning with spot

Created a xpk cluster with --spot flag.
Validated nodepool was created with spot nodes
Created a workload and validated it ran.

avrittrohwer commented 1 month ago

zone: 'us-central2'>] finished with error: Try a different location, or try again later: Google Compute Engine does not have enough resources available to fulfill request: us-central2-b