benofben
commented
6 years ago This is subtle --- the timeout is happening for the tags operation. That's causing the script to pick the wrong rally point. It guessed badly and picked a non-inited node which caused it to block until I added that manually.
n1 oniguruma.x86_64 0:5.9.1-3.1.2.amzn1
Complete! This node is part of the autoscaling group that contains the rally point.
An error occurred (RequestLimitExceeded) when calling the DescribeTags operation (reached max retries: 4): Request limit exceeded.
An error occurred (RequestLimitExceeded) when calling the DescribeTags operation (reached max retries: 4): Request limit exceeded.
An error occurred (RequestLimitExceeded) when calling the DescribeTags operation (reached max retries: 4): Request limit exceeded.
An error occurred (RequestLimitExceeded) when calling the DescribeTags operation (reached max retries: 4): Request limit exceeded.
An error occurred (RequestLimitExceeded) when calling the DescribeTags operation (reached max retries: 4): Request limit exceeded. % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 42 100 42 0 0 6825 0 --:--:-- --:--:-- --:--:-- 7000 Using the settings: adminUsername 'couchbase' adminPassword 'foo123!' services 'data,index,query,fts' stackName 'prebaked' rallyPublicDNS 'ec2-54-85-58-168.compute-1.amazonaws.com' region 'us-east-1' instanceID 'i-08c7ff49764abe9df' nodePublicDNS 'ec2-34-207-221-187.compute-1.amazonaws.com' Running couchbase-cli node-init node-init output 'SUCCESS: Node initialized' Running couchbase-cli server-add server-add output 'ERROR: Cluster is not initialized, use cluster-init to initialize the cluster' server-add output 'ERROR: Cluster is not initialized, use cluster-init to initialize the cluster' server-add output 'ERROR: Cluster is not initialized, use cluster-init to initialize the cluster' server-add output 'ERROR: Cluster is not initialized, use cluster-init to initialize the cluster' server-add output 'ERROR: Cluster is not initialized, use cluster-init to initialize the cluster' server-add output 'ERROR: Cluster is not initialized, use cluster-init to initialize the cluster'
It looks like the CFT runs about against ddos protections for the metadata service at scale (100 nodes):
Complete! This node is part of the autoscaling group that contains the rally point.
An error occurred (RequestLimitExceeded) when calling the DescribeTags operation (reached max retries: 4): Request limit exceeded.
An error occurred (RequestLimitExceeded) when calling the DescribeInstances operation (reached max retries: 4): Request limit exceeded. % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 41 100 41 0 0 80078 0 --:--:-- --:--:-- --:--:-- 41000 Using the setting