Prevent Pathways SIGTERMs from counting against backoffLimit

AI-Hypercomputer / xpk

xpk (Accelerated Processing Kit, pronounced x-p-k,) is a software tool to help Cloud developers to orchestrate training jobs on accelerators such as TPUs and GPUs on GKE.

Apache License 2.0

81 stars 23 forks source link

Prevent Pathways SIGTERMs from counting against backoffLimit #134

Open SujeethJinesh opened 7 months ago

SujeethJinesh commented 7 months ago

Fixes / Features

Add podFailurePolicy so SIGTERMed Pathways workers do not count against the default backoffLimit (4).

Testing / Documentation

TODO

[ y ] Tests pass
[ y ] Appropriate changes to documentation are included in the PR