search

freelawproject / courtlistener

A fully-searchable and accessible archive of court data including growing repositories of opinions, oral arguments, judges, judicial financial records, and federal filings.
https://www.courtlistener.com
Other
542 stars 149 forks source link

"volume node affinity conflict" when upgrading Elastic AMIs #3245

Open mlissner opened 1 year ago

mlissner commented 1 year ago

Last week I updated our elastic AMIs by using the button in the EKS console. That seemed fine at the time, but I didn't notice until today that it took down one of our Elastic pods. The problem it would appear is that:

"volume node affinity conflict"

That error appeared in the description (describe pod) for the failing Elastic pod.

What that means is that the pod wanted to start, but didn't have the right volumes available on any node. Digging in, I noticed some things:

So that makes sense, in its way. The pod can't start b/c the storage ain't there, and the storage ain't there because the node is in the wrong place vs. the location of the storage that was created formerly, and the node is in the wrong place because it's somewhat randomly assigned.

So, OK, fine. How do we fix this? A few things:

  1. The solution that eventually worked is a hack. I upped our desired number of nodes from 3 to 4, forcing EKS to make a new node. I guessed it'd do it in the needed availability zone, and it did. Once that node was up, the pod started working and I set the number of nodes back to 3, which deleted the problematic node, keeping the new, good one.

  2. k8s has a system of "affinities" that allows you to put nodes and storage and stuff in certain places.

    Right now, we have storage in us-west-2a, us-west-2b, and us-west-2c, so if we're going to have three nodes, those nodes in the same places. I kind of don't know how that's possible.

    One fix, I guess would be to say that all of our nodes only go into 2a, say, and then move the storage to 2a somehow (or recreate it). But it feels like there should be a better answer. The link above indicates how to pin your nodes and storage to certain zones, but, like, don't we want them spread around?

I'm a bit lost about the next step here, but it's near the end of my day, so I guess it's about time I set it aside, and perhaps tomorrow will yield an answer.

mlissner commented 1 year ago

Hm, it looks like moving volumes isn't so hard: https://awsbytes.com/how-to-move-ebs-volume-to-a-different-ec2-instance-across-availability-zones/

So I guess the trick is to put everything in one zone, though that kind of annoys me too because that's all our eggs in one basket.

mlissner commented 1 year ago

I asked a question about this here: https://stackoverflow.com/questions/77291136/whats-the-best-approach-to-availability-zones-for-an-elastic-search-cluster

We'll see if folks are up to reading all that!

mlissner commented 1 year ago

The best other info I've found is this: https://stackoverflow.com/questions/51946393/kubernetes-pod-warning-1-nodes-had-volume-node-affinity-conflict

mlissner commented 1 year ago

So my StackOverflow question was downvoted and closed, and I read today about how Stackoverflow is laying off a bunch of people, so I thought to myself, "Hm, maybe I'll try ChatGPT":

https://chat.openai.com/share/f2e2b355-e28c-48ca-b65d-117208b2a601

I mean, it's so damned confident, you can't really tell when it's wrong, but it has a couple strategies that do make sense:

All in all, I sort of feel like this is a pain to fix, and while I think we should probably do so, it's not a huge deal if we don't. The fix I used above (just scaling up to 4 nodes temporarily) worked pretty well, and we don't reboot nodes often, so I think I'm going to let this be for a bit.

Maybe when we finish indexing all the content, I'll re-evaluate and we can decide if we want to move everything to a single AZ at that point.