Closed jazzl0ver closed 6 years ago
Could you please check the EC2 instance type and the number of availability zones? Did you deploy other services?
Instance type - t2.medium AZs are from a to e (no f) No other services. Just a clean firecamp env
On Jan 25, 2018 21:18, "JuniusLuo" notifications@github.com wrote:
Could you please check the EC2 instance type and the number of availability zones? Did you deploy other services?
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/cloudstax/firecamp/issues/18#issuecomment-360553533, or mute the thread https://github.com/notifications/unsubscribe-auth/ACVLj8VJMjUyS-jkIdxLCPNZB5uZhOSqks5tOMV3gaJpZM4RtQ4S .
Thanks. How many nodes in the cluster? 5 or 3?
3 nodes
On Jan 25, 2018 21:27, "JuniusLuo" notifications@github.com wrote:
Thanks. How many nodes in the cluster? 5 or 3?
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/cloudstax/firecamp/issues/18#issuecomment-360556165, or mute the thread https://github.com/notifications/unsubscribe-auth/ACVLj6MEtSfmVni9o0p2e3HqlWIL4wsrks5tOMekgaJpZM4RtQ4S .
This might be the issue. Currently when creating the service, FireCamp does not check if there is node running in one zone. FireCamp manage service simply assigns the service replicas to zones in the round-robin mode. So the replica may be assigned to the zone that no node is running.
Is there any reason that you want the cluster over 5 zones while has only 3 nodes? If you have 5 nodes on 5 zones or 3 nodes on 3 zones, this issue would not show up.
The main reason was future scaling. I thought having configured multiple AZs might be used when we need to increase a number of instances - they will be created in that AZs. I'll try tomorrow to equalize AZs and instances amount. Thank you for your help!
On Thu, Jan 25, 2018 at 9:44 PM, JuniusLuo notifications@github.com wrote:
This might be the issue. Currently when creating the service, FireCamp does not check if there is node running in one zone. FireCamp manage service simply assigns the service replicas to zones in the round-robin mode. So the replica may be assigned to the zone that no node is running.
Is there any reason that you want the cluster over 5 zones while has only 3 nodes? If you have 5 nodes on 5 zones or 3 nodes on 3 zones, this issue would not show up.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/cloudstax/firecamp/issues/18#issuecomment-360560634, or mute the thread https://github.com/notifications/unsubscribe-auth/ACVLjyRAxVnb520nl_jmqVnxk69UexOTks5tOMt0gaJpZM4RtQ4S .
How do you want to scale? Want to scale ZooKeeper to 5 nodes on 5 AZs?
There is one limitation by AutoScalingGroup and EBS. If the cluster has 5 AZs and 3 instances, ASG may create the new instance in the 4th AZ when one instance goes down. But the previous EBS volume is not in the 4th AZ. So one member will fail to start.
You could start with 3 AZs and 3 instances. In the future release, we will support scaling the AZs. We could add the new AZs to the ASG, and update the new AZs to the FireCamp manage service. The manage service will create the new replicas in the new AZs when scaling the ZooKeeper service.
After shrinking AZs number to 3, everything worked like a charm! Thank you!
close this issue, as it works with the correct number of nodes. Scaling the cluster is an advanced feature in the later release.
I'm sorry for bothering you, but this 0.9.2 release is a headache for me. Can you please check if you can start zookeeper in ECS with the following command:
I'm getting:
And finally have one zookeeper container running only. Service events show:
Firecamp log ends up with:
Any ideas what's going on?