Availability zones causing problems with mounting volumes to Kubernetes pods

[jpolchlo]

Our Kubernetes cluster was configured with multiple availability zones (if my memory is correct, there is a reason for this, but I can't recall it now). This is causing some problems when attempting to mount previously-created persistent volumes, as their backing stores (EBS or EFS) have to be in the same AZ as the node hosting the pod. This may be solvable with node affinities, but a solution is needed if we are to run with multiple AZs and persistent volumes.

• [ ] Confirm that we need multiple AZs
  • If not, maybe move to just one
• [ ] Try to set node affinities to launch pods on nodes with the same AZ as the backing EBS volumes

[jpolchlo]

It appears to be the accepted wisdom that multiple AZs are the solution to instability, but we probably don't need a high-availability system. According to this article, an AZ failure is rare. We are introducing possible complexity for not much gain. The use of multiple AZs definitely affects volumes, where EBS volumes must be located in the same AZ as the mounting pod. EFS volumes don't have to be in the same zone, but cross-zone traffic (which happens when the EFS volume is in a different zone from the calling pod) will incur traffic charges, which can rack up quickly.

I've encountered some chatter to the effect of "the k8s scheduler will put pods mounting a PV into the same zone as the PV", but the current issue came up because this wasn't happening. I've since upgraded to Kubernetes version 1.23 and JupyterHub chart version 2.0.0, which may have quietly fixed the problem. (JH 2.0.0 uses a more recent kubelet agent, which solved some other issues that I encountered with specifying volumes, so this may affect the problem that this issue is about.) All that said, this issue is possibly moot in that we moved to storing notebooks on S3 using s3contents.

Possibly relevant for the future is the idea of setting up a shared volume for all Jupyter notebook pods to mount which holds a copy of some NWM files that are important to the project—namely the retrospective NWM data. This may trigger this problem again. So, it makes sense to keep this issue on the radar. But it's not clear that this needs to stay open. I'll probably end up solving this problem the next time I add a static volume anyway.

Reopen this if the spirit moves you.