search

zalando-stups / stups-etcd-cluster

Etcd cluster appliance for the STUPS (AWS) environment
Other
29 stars 9 forks source link

Question regarding upgrades #45

Open lutzh opened 7 years ago

lutzh commented 7 years ago

Hi,

I find the upgrade instructions a bit vague. It says:

In order to perform a minor or major upgrade without downtime you need to terminate all EC2 instances one-by-one.

So the suggestion is to do a "rolling update", if I understand correctly. The senza "version" would remain the same, as this is used as a service identifier / DNS entry in this case that should remain stable. Could you elaborate how such a rolling update would be performed? The quote above is confusing, as terminating EC2 instances by hand would just result in new ones being created, because they belong to the same autoscaling group and it would attempt to maintain the number of nodes. So you'd have to manually add an EC2 instance with a newer etcd version first, and then remove one with the old version?

Could you maybe elaborate a bit on, or even provide an example for, the update of the etcd cluster?

CyberDem0n commented 7 years ago

The senza "version" would remain the same, as this is used as a service identifier / DNS entry in this case that should remain stable.

This is a stateful application, you can't just "deploy" a new stack for upgrade and it's not related to DNS.

The quote above is confusing, as terminating EC2 instances by hand would just result in new ones being created, because they belong to the same autoscaling group and it would attempt to maintain the number of nodes.

What is confusing here? After senza update there will be a different Launch Configuration, containing a reference to a new docker image. When ASG spawns a new instance, it will download and start a new docker image.

So you'd have to manually add an EC2 instance with a newer etcd version first, and then remove one with the old version?

You should never ever do that.

Could you maybe elaborate a bit on, or even provide an example for, the update of the etcd cluster?

Please read official documentation: https://github.com/coreos/etcd/blob/master/Documentation/upgrades/upgrading-etcd.md

lutzh commented 7 years ago

Thanks so much @CyberDem0n for the quick response! I think my confusion stems from a lack of senza understanding.

This is a stateful application, you can't just "deploy" a new stack for upgrade and it's not related to DNS.

That makes sense, thanks for the clarification.

After senza update there will be a different Launch Configuration, containing a reference to a new docker image.

Sounds good! I did not understand this process from the current description. Right now, even after your explanation, I still wouldn't really know how to do it. You say

After senza update

I can't find any information on an update command, or how to update configurations, on https://github.com/zalando-stups/senza . Maybe you could point me to the right documentation?

Thanks!

P.S.: Once I get my head around it, I'll happily provide a PR with a more detailed description of the process, e.g. first update the the launch configuration with senza, then kill a node..