search

Five-Borough-Fedi-Project / masto.nyc-docean

masto.nyc IoC v2 hosted in Digital Ocean
2 stars 0 forks source link

Current k8s volume don't allow for multi-pod scaling #2

Closed seano-vs closed 5 days ago

seano-vs commented 4 weeks ago

Ever since the cloud migration, we've been getting multi-attach errors when scaling out the web service. This is probably due to the RWOnce nature of the volumes. This will need to be resolved to scale.... like, at all

seano-vs commented 3 weeks ago

Right now, this means we can only scale deployments down to 0 and back up to 1 instead of doing a rolling restart

seano-vs commented 3 weeks ago

I found this >_< https://github.com/digitalocean/csi-digitalocean/issues/136

seano-vs commented 3 weeks ago

also, now the web and sidekiq pod need to be on the same node, so I've hard coded it into the deployments. I'm concerned that this may not survive a mandatory docean k8s cluster upgrade

https://github.com/mastodon/mastodon/issues/1016

seano-vs commented 5 days ago

OOF. This is bad. Hard-coding the node name does not persist across upgrades >_<

This is now going to be high priority to fix. I just applied an update to some of the k8s config and it brought everything down ;___;

seano-vs commented 5 days ago

fixed in https://github.com/Five-Borough-Fedi-Project/masto.nyc-docean/commit/27ba392458377241adf8a881e2df38594bfcf6c1