Create single reserved instance node for critical pods

[mattjmcnaughton]

Issue To save money, all of my k8s nodes are spot instances. However, even if I set my maxPrice equal to the on-demand price, there are still windows in which I don't get spot instances. I have critical pods (i.e. online services like my blog) running on these spot instances. In addition, my monitoring/alerting nodes (i.e. prometheus and alertmanager) als run on these spot instances. So I don't even receive alerts when the spot instances are not available.

Impl

• Purchase a single "reserved instance" node.
• Create a new kops instance group which will not use spot instances and will use the reserved instance.
• Use affinities/tolerations to schedule critical pods to this node.

Definition of Done

• [x] Critical pods are running on a non-spot instance.
• [x] File ticket to consider if should use spot instance => schedule for 1 month from now.
• [x] Blog post.

[RmMsr]

Would you mind, posting a link to the blog post? Your issue ranks interestingly high on searches for dedicating k8s nodes to critical workloads. Thanks

[mattjmcnaughton]

Sure! https://mattjmcnaughton.com/post/saving-money-while-maintaining-performance-with-tolerations-on-k8s/

Please let me know if you have any questions.