search

crosscloudci / cross-cloud

Cross-Cloud - multi-cloud K8s provisioner for CNCF CI Project
https://cncf.ci
Apache License 2.0
166 stars 59 forks source link

ATD-8 Add OCI into cross-cloud #168

Closed hbscharp closed 6 years ago

hbscharp commented 6 years ago

Add the OCI cloud controller manager, flex volume driver, and volume provisioner to OCI xcloud clusters. We had to fork the 1.10 templates in order to work around a CoreOS hostname issue in our downrev image version and add kubelet back onto the master in order for our modules to function. The OCI-specific addons are handled with the addition of a vendor-specific TF module 'addons' that copies and renders the required files to the container's data mount. The additional steps necessary to install the addons are then added in a vendor-specific conditional in the Gitlab CI script.

denverwilliams commented 6 years ago

Conflicts resolved, ready for merge.

hbscharp commented 6 years ago

Thanks for the bug report, Denver. I'm sorry you ran into trouble, and that it took this long for me to respond. This issue will be at the top of our backlog Monday morning. We'll send a PR when we can.

-Ben

On Mon, Sep 24, 2018 at 10:17 PM Denver Williams notifications@github.com wrote:

After some testing we seem to be running into an error with the oci-volume-provisioner, it looks like the config file is missing.

bash-4.4# kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE kube-system kube-proxy-oci-master-1.oci.oracle.local 1/1 Running 0 17m kube-system kube-proxy-oci-worker-1.oci.oracle.local 1/1 Running 0 16m kube-system oci-cloud-controller-manager-gpf64 1/1 Running 0 15m kube-system oci-flexvolume-driver-worker-mrcz2 1/1 Running 0 14m kube-system oci-volume-provisioner-585c45b794-fz96l 0/1 CrashLoopBackOff 7 14m

bash-4.4# kubectl logs oci-volume-provisioner-585c45b794-fz96l --namespace=kube-system W0925 01:44:38.636182 1 client_config.go:552] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. I0925 01:44:38.646529 1 main.go:99] Starting volume provisioner in oracle.com/oci mode F0925 01:44:38.646593 1 provisioner.go:75] Unable to load volume provisioner configuration file: /etc/oci/config.yaml

Steps to reproduce

  1. Deploy k8s cluster to oracle cloud using provision.sh
  2. Create rbac roles kubectl create -f /cncf/rbac
  3. Create secrets

kubectl create secret generic oci-flexvolume-driver \ -n kube-system \ --from-file=config.yaml=/cncf/data/addons/create/oci-fvd-secret.yaml

kubectl create secret generic oci-cloud-controller-manager \ -n kube-system \ --from-file=cloud-provider.yaml=/cncf/data/addons/create/oci-ccm-secret.yaml

kubectl create secret generic oci-volume-provisioner \ -n kube-system \ --from-file=config.yaml=/cncf/data/addons/create/oci-vp-secret.yaml

  1. Lastly deploy the addons and check the logs for the oci-volume-provisioner kubectl create -f /cncf/data/addons/apply/ kubectl get pods --namespace=kube-system

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/crosscloudci/cross-cloud/pull/168#issuecomment-424184926, or mute the thread https://github.com/notifications/unsubscribe-auth/ACzCx8pqag5BQDGsPEZ3uPHTTP5s3P8aks5ueZI9gaJpZM4WiMbQ .

denverwilliams commented 5 years ago

Hey @hbscharp

We have been running into some new issues the the OCI provider, i suspect maybe there are some old/duplicate resources on the oci cloud account?

Below error occurs when running 

/cncf/provision.sh oci-deploy test file oci
Error: Error applying plan:

1 error(s) occurred:

* module.master_lb.oci_load_balancer.lb-k8smaster: 1 error(s) occurred:

* oci_load_balancer.lb-k8smaster: Service error:LimitExceeded. The limit for this tenancy has been exceeded.. http status code: 400. Opc request id: 504215c0151aef0c6fe740015f1e2b0b/6D129982F1532F8A6FD849575653BA49/4209C3C83E6A477758F0CC575BC5BDC3

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.
hbscharp commented 5 years ago

Denver,

The issue is likely that nothing in your jobs is running the destroy step as documented in the OCI readme. The allocated compartment had consumed its quota. There's enough there for 4-5 concurrent iterations, and I think that's enough. If you run the provision script in the container with the data volume that contains the terraform plan of that iteration the OCI resources will be deleted and the quota restored. I have run that manually and I believe it still works. I have deleted everything in your compartment. You should have resources to continue again. Please let me know if we need to contribute the change to run the destroy job at the correct spot in your pipeline.

Thanks,

-Ben

On Tue, Oct 9, 2018 at 9:24 PM Denver Williams notifications@github.com wrote:

Hey @hbscharp https://github.com/hbscharp

We have been running into some new issues the the OCI provider, i suspect maybe there are some old/duplicate resources on the oci cloud account?

Below error occurs when running

/cncf/provision.sh oci-deploy test file oci

Error: Error applying plan:

1 error(s) occurred:

  • module.master_lb.oci_load_balancer.lb-k8smaster: 1 error(s) occurred:

  • oci_load_balancer.lb-k8smaster: Service error:LimitExceeded. The limit for this tenancy has been exceeded.. http status code: 400. Opc request id: 504215c0151aef0c6fe740015f1e2b0b/6D129982F1532F8A6FD849575653BA49/4209C3C83E6A477758F0CC575BC5BDC3

Terraform does not automatically rollback in the face of errors. Instead, your Terraform state file has been partially updated with any resources that successfully completed. Please address the error above and apply again to incrementally change your infrastructure.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/crosscloudci/cross-cloud/pull/168#issuecomment-428406221, or mute the thread https://github.com/notifications/unsubscribe-auth/ACzCx81CtKoVln2s3Utsmz56RN7tP8zSks5ujUxbgaJpZM4WiMbQ .

denverwilliams commented 5 years ago

@hbscharp Thanks for the Help!

denverwilliams commented 5 years ago

We have since run into some new troubles, with etcd this time (only happening with multi-master deploys)

Node 1 

Oct 11 03:16:38 oci-master-1.oci.oracle.local systemd[1]: Starting etcd...
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=http://oci-master-1.oci.oracle.local:4001
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: recognized and used environment variable ETCD_DATA_DIR=/var/etcd/data
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: recognized and used environment variable ETCD_DISCOVERY_SRV=oci.oracle.local
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=http://oci-master-1.oci.oracle.local:2380
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=new
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: recognized and used environment variable ETCD_INITIAL_CLUSTER_TOKEN=oci-etcd
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:4001
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: recognized and used environment variable ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: recognized and used environment variable ETCD_NAME=oci
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: etcd Version: 3.2.8
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: Git SHA: e211fb6
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: Go Version: go1.8.3
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: Go OS/Arch: linux/amd64
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: setting maximum number of CPUs to 8, total number of available CPUs is 8
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: listening for peers on http://0.0.0.0:2380
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: listening for client requests on 0.0.0.0:4001
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: got bootstrap from DNS for etcd-server at 0=http://oci-master-3.oci.oracle.local:2380
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: got bootstrap from DNS for etcd-server at 1=http://oci-master-2.oci.oracle.local:2380
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: got bootstrap from DNS for etcd-server at oci=http://oci-master-1.oci.oracle.local:2380
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: resolving oci-master-1.oci.oracle.local:2380 to 10.0.15.5:2380
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: resolving oci-master-1.oci.oracle.local:2380 to 10.0.15.5:2380
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: name = oci
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: data dir = /var/etcd/data
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: member dir = /var/etcd/data/member
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: heartbeat = 100ms
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: election = 1000ms
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: snapshot count = 100000
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: advertise client URLs = http://oci-master-1.oci.oracle.local:4001
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: initial advertise peer URLs = http://oci-master-1.oci.oracle.local:2380
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: initial cluster = 0=http://oci-master-3.oci.oracle.local:2380,1=http://oci-master-2.oci.oracle.local:2380,oci=http://oci-master-1.oci.oracle.local:2380
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: starting member 20945348073f2613 in cluster 89cb34bd51d6a9f3
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: 20945348073f2613 became follower at term 0
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: newRaft 20945348073f2613 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: 20945348073f2613 became follower at term 1
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: simple token is not cryptographically signed
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: starting peer 4aef4be2f926f4ff...
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: started HTTP pipelining with peer 4aef4be2f926f4ff
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: started streaming with peer 4aef4be2f926f4ff (writer)
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: started streaming with peer 4aef4be2f926f4ff (writer)
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: started peer 4aef4be2f926f4ff
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: added peer 4aef4be2f926f4ff
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: starting peer a2bafbe2ed664069...
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: started HTTP pipelining with peer a2bafbe2ed664069
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: started streaming with peer 4aef4be2f926f4ff (stream MsgApp v2 reader)
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: started streaming with peer 4aef4be2f926f4ff (stream Message reader)
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: started streaming with peer a2bafbe2ed664069 (writer)
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: started streaming with peer a2bafbe2ed664069 (writer)
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: started peer a2bafbe2ed664069
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: added peer a2bafbe2ed664069
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: started streaming with peer a2bafbe2ed664069 (stream MsgApp v2 reader)
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: starting server... [version: 3.2.8, cluster version: to_be_decided]
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: started streaming with peer a2bafbe2ed664069 (stream Message reader)
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: added member 20945348073f2613 [http://oci-master-1.oci.oracle.local:2380] to cluster 89cb34bd51d6a9f3
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: added member 4aef4be2f926f4ff [http://oci-master-2.oci.oracle.local:2380] to cluster 89cb34bd51d6a9f3
Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: added member a2bafbe2ed664069 [http://oci-master-3.oci.oracle.local:2380] to cluster 89cb34bd51d6a9f3
Oct 11 03:16:39 oci-master-1.oci.oracle.local etcd[1222]: 20945348073f2613 is starting a new election at term 1
Oct 11 03:16:39 oci-master-1.oci.oracle.local etcd[1222]: 20945348073f2613 became candidate at term 2
Oct 11 03:16:39 oci-master-1.oci.oracle.local etcd[1222]: 20945348073f2613 received MsgVoteResp from 20945348073f2613 at term 2
Oct 11 03:16:39 oci-master-1.oci.oracle.local etcd[1222]: 20945348073f2613 [logterm: 1, index: 3] sent MsgVote request to 4aef4be2f926f4ff at term 2
Oct 11 03:16:39 oci-master-1.oci.oracle.local etcd[1222]: 20945348073f2613 [logterm: 1, index: 3] sent MsgVote request to a2bafbe2ed664069 at term 2
Oct 11 03:16:41 oci-master-1.oci.oracle.local etcd[1222]: 20945348073f2613 is starting a new election at term 2
Oct 11 03:16:41 oci-master-1.oci.oracle.local etcd[1222]: 20945348073f2613 became candidate at term 3
Oct 11 03:16:41 oci-master-1.oci.oracle.local etcd[1222]: 20945348073f2613 received MsgVoteResp from 20945348073f2613 at term 3
Oct 11 03:16:41 oci-master-1.oci.oracle.local etcd[1222]: 20945348073f2613 [logterm: 1, index: 3] sent MsgVote request to 4aef4be2f926f4ff at term 3
Oct 11 03:16:41 oci-master-1.oci.oracle.local etcd[1222]: 20945348073f2613 [logterm: 1, index: 3] sent MsgVote request to a2bafbe2ed664069 at term 3
Oct 11 03:16:42 oci-master-1.oci.oracle.local etcd[1222]: 20945348073f2613 is starting a new election at term 3
Oct 11 03:16:42 oci-master-1.oci.oracle.local etcd[1222]: 20945348073f2613 became candidate at term 4
Oct 11 03:16:42 oci-master-1.oci.oracle.local etcd[1222]: 20945348073f2613 received MsgVoteResp from 20945348073f2613 at term 4
Oct 11 03:16:42 oci-master-1.oci.oracle.local etcd[1222]: 20945348073f2613 [logterm: 1, index: 3] sent MsgVote request to 4aef4be2f926f4ff at term 4
Oct 11 03:16:42 oci-master-1.oci.oracle.local etcd[1222]: 20945348073f2613 [logterm: 1, index: 3] sent MsgVote request to a2bafbe2ed664069 at term 4
Oct 11 03:16:43 oci-master-1.oci.oracle.local etcd[1222]: health check for peer 4aef4be2f926f4ff could not connect: dial tcp 10.0.15.3:2380: getsockopt: connection refused
Oct 11 03:16:43 oci-master-1.oci.oracle.local etcd[1222]: health check for peer a2bafbe2ed664069 could not connect: dial tcp 10.0.15.4:2380: getsockopt: connection refused
Oct 11 03:16:44 oci-master-1.oci.oracle.local etcd[1222]: 20945348073f2613 is starting a new election at term 4
Oct 11 03:16:44 oci-master-1.oci.oracle.local etcd[1222]: 20945348073f2613 became candidate at term 5
Oct 11 03:16:44 oci-master-1.oci.oracle.local etcd[1222]: 20945348073f2613 received MsgVoteResp from 20945348073f2613 at term 5
Oct 11 03:16:44 oci-master-1.oci.oracle.local etcd[1222]: 20945348073f2613 [logterm: 1, index: 3] sent MsgVote request to 4aef4be2f926f4ff at term 5
Oct 11 03:16:44 oci-master-1.oci.oracle.local etcd[1222]: 20945348073f2613 [logterm: 1, index: 3] sent MsgVote request to a2bafbe2ed664069 at term 5
Oct 11 03:16:45 oci-master-1.oci.oracle.local etcd[1222]: publish error: etcdserver: request timed out

Node 2

Oct 11 03:17:20 oci-master-2.oci.oracle.local systemd[1]: Starting etcd...
Oct 11 03:17:20 oci-master-2.oci.oracle.local etcd[1220]: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=http://oci-master-1.oci.oracle.local:4001
Oct 11 03:17:20 oci-master-2.oci.oracle.local etcd[1220]: recognized and used environment variable ETCD_DATA_DIR=/var/etcd/data
Oct 11 03:17:20 oci-master-2.oci.oracle.local etcd[1220]: recognized and used environment variable ETCD_DISCOVERY_SRV=oci.oracle.local
Oct 11 03:17:20 oci-master-2.oci.oracle.local etcd[1220]: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=http://oci-master-1.oci.oracle.local:2380
Oct 11 03:17:20 oci-master-2.oci.oracle.local etcd[1220]: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=new
Oct 11 03:17:20 oci-master-2.oci.oracle.local etcd[1220]: recognized and used environment variable ETCD_INITIAL_CLUSTER_TOKEN=oci-etcd
Oct 11 03:17:20 oci-master-2.oci.oracle.local etcd[1220]: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:4001
Oct 11 03:17:20 oci-master-2.oci.oracle.local etcd[1220]: recognized and used environment variable ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380
Oct 11 03:17:20 oci-master-2.oci.oracle.local etcd[1220]: recognized and used environment variable ETCD_NAME=oci
Oct 11 03:17:20 oci-master-2.oci.oracle.local etcd[1220]: etcd Version: 3.2.8
Oct 11 03:17:20 oci-master-2.oci.oracle.local etcd[1220]: Git SHA: e211fb6
Oct 11 03:17:20 oci-master-2.oci.oracle.local etcd[1220]: Go Version: go1.8.3
Oct 11 03:17:20 oci-master-2.oci.oracle.local etcd[1220]: Go OS/Arch: linux/amd64
Oct 11 03:17:20 oci-master-2.oci.oracle.local etcd[1220]: setting maximum number of CPUs to 8, total number of available CPUs is 8
Oct 11 03:17:20 oci-master-2.oci.oracle.local etcd[1220]: listening for peers on http://0.0.0.0:2380
Oct 11 03:17:20 oci-master-2.oci.oracle.local etcd[1220]: listening for client requests on 0.0.0.0:4001
Oct 11 03:17:20 oci-master-2.oci.oracle.local etcd[1220]: got bootstrap from DNS for etcd-server at oci=http://oci-master-1.oci.oracle.local:2380
Oct 11 03:17:20 oci-master-2.oci.oracle.local etcd[1220]: got bootstrap from DNS for etcd-server at 0=http://oci-master-2.oci.oracle.local:2380
Oct 11 03:17:20 oci-master-2.oci.oracle.local etcd[1220]: got bootstrap from DNS for etcd-server at 1=http://oci-master-3.oci.oracle.local:2380
Oct 11 03:17:20 oci-master-2.oci.oracle.local etcd[1220]: resolving oci-master-1.oci.oracle.local:2380 to 10.0.15.5:2380
Oct 11 03:17:20 oci-master-2.oci.oracle.local etcd[1220]: resolving oci-master-1.oci.oracle.local:2380 to 10.0.15.5:2380
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: name = oci
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: data dir = /var/etcd/data
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: member dir = /var/etcd/data/member
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: heartbeat = 100ms
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: election = 1000ms
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: snapshot count = 100000
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: advertise client URLs = http://oci-master-1.oci.oracle.local:4001
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: initial advertise peer URLs = http://oci-master-1.oci.oracle.local:2380
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: initial cluster = 0=http://oci-master-2.oci.oracle.local:2380,1=http://oci-master-3.oci.oracle.local:2380,oci=http://oci-master-1.oci.oracle.local:2380
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: starting member 20945348073f2613 in cluster 89cb34bd51d6a9f3
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: 20945348073f2613 became follower at term 0
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: newRaft 20945348073f2613 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: 20945348073f2613 became follower at term 1
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: simple token is not cryptographically signed
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: starting peer 4aef4be2f926f4ff...
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: started HTTP pipelining with peer 4aef4be2f926f4ff
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: started streaming with peer 4aef4be2f926f4ff (writer)
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: started streaming with peer 4aef4be2f926f4ff (writer)
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: started peer 4aef4be2f926f4ff
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: added peer 4aef4be2f926f4ff
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: starting peer a2bafbe2ed664069...
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: started streaming with peer 4aef4be2f926f4ff (stream MsgApp v2 reader)
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: started streaming with peer 4aef4be2f926f4ff (stream Message reader)
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: started HTTP pipelining with peer a2bafbe2ed664069
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: started streaming with peer a2bafbe2ed664069 (writer)
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: started streaming with peer a2bafbe2ed664069 (writer)
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: started streaming with peer a2bafbe2ed664069 (stream MsgApp v2 reader)
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: started peer a2bafbe2ed664069
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: started streaming with peer a2bafbe2ed664069 (stream Message reader)
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: added peer a2bafbe2ed664069
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: starting server... [version: 3.2.8, cluster version: to_be_decided]
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: added member 20945348073f2613 [http://oci-master-1.oci.oracle.local:2380] to cluster 89cb34bd51d6a9f3
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: added member 4aef4be2f926f4ff [http://oci-master-2.oci.oracle.local:2380] to cluster 89cb34bd51d6a9f3
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: started HTTP pipelining with peer 20945348073f2613
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: added member a2bafbe2ed664069 [http://oci-master-3.oci.oracle.local:2380] to cluster 89cb34bd51d6a9f3
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: 20945348073f2613 [term: 1] received a MsgVote message with higher term from 20945348073f2613 [term: 32]
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: 20945348073f2613 became follower at term 32
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: 20945348073f2613 [logterm: 1, index: 3, vote: 0] cast MsgVote for 20945348073f2613 [logterm: 1, index: 3] at term 32
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: peer 20945348073f2613 became active
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3
Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3
Oct 11 03:17:22 oci-master-2.oci.oracle.local etcd[1220]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3

Node 3 

-- Logs begin at Thu 2018-10-11 03:16:08 UTC, end at Thu 2018-10-11 03:25:09 UTC. --
Oct 11 03:16:57 oci-master-3.oci.oracle.local systemd[1]: Starting etcd...
Oct 11 03:16:57 oci-master-3.oci.oracle.local etcd[1221]: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=http://oci-master-1.oci.oracle.local:4001
Oct 11 03:16:57 oci-master-3.oci.oracle.local etcd[1221]: recognized and used environment variable ETCD_DATA_DIR=/var/etcd/data
Oct 11 03:16:57 oci-master-3.oci.oracle.local etcd[1221]: recognized and used environment variable ETCD_DISCOVERY_SRV=oci.oracle.local
Oct 11 03:16:57 oci-master-3.oci.oracle.local etcd[1221]: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=http://oci-master-1.oci.oracle.local:2380
Oct 11 03:16:57 oci-master-3.oci.oracle.local etcd[1221]: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=new
Oct 11 03:16:57 oci-master-3.oci.oracle.local etcd[1221]: recognized and used environment variable ETCD_INITIAL_CLUSTER_TOKEN=oci-etcd
Oct 11 03:16:57 oci-master-3.oci.oracle.local etcd[1221]: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:4001
Oct 11 03:16:57 oci-master-3.oci.oracle.local etcd[1221]: recognized and used environment variable ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380
Oct 11 03:16:57 oci-master-3.oci.oracle.local etcd[1221]: recognized and used environment variable ETCD_NAME=oci
Oct 11 03:16:57 oci-master-3.oci.oracle.local etcd[1221]: etcd Version: 3.2.8
Oct 11 03:16:57 oci-master-3.oci.oracle.local etcd[1221]: Git SHA: e211fb6
Oct 11 03:16:57 oci-master-3.oci.oracle.local etcd[1221]: Go Version: go1.8.3
Oct 11 03:16:57 oci-master-3.oci.oracle.local etcd[1221]: Go OS/Arch: linux/amd64
Oct 11 03:16:57 oci-master-3.oci.oracle.local etcd[1221]: setting maximum number of CPUs to 8, total number of available CPUs is 8
Oct 11 03:16:57 oci-master-3.oci.oracle.local etcd[1221]: listening for peers on http://0.0.0.0:2380
Oct 11 03:16:57 oci-master-3.oci.oracle.local etcd[1221]: listening for client requests on 0.0.0.0:4001
Oct 11 03:16:58 oci-master-3.oci.oracle.local etcd[1221]: got bootstrap from DNS for etcd-server at 0=http://oci-master-3.oci.oracle.local:2380
Oct 11 03:16:58 oci-master-3.oci.oracle.local etcd[1221]: got bootstrap from DNS for etcd-server at 1=http://oci-master-2.oci.oracle.local:2380
Oct 11 03:16:58 oci-master-3.oci.oracle.local etcd[1221]: got bootstrap from DNS for etcd-server at oci=http://oci-master-1.oci.oracle.local:2380
Oct 11 03:16:58 oci-master-3.oci.oracle.local etcd[1221]: resolving oci-master-1.oci.oracle.local:2380 to 10.0.15.5:2380
Oct 11 03:16:58 oci-master-3.oci.oracle.local etcd[1221]: resolving oci-master-1.oci.oracle.local:2380 to 10.0.15.5:2380
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: name = oci
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: data dir = /var/etcd/data
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: member dir = /var/etcd/data/member
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: heartbeat = 100ms
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: election = 1000ms
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: snapshot count = 100000
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: advertise client URLs = http://oci-master-1.oci.oracle.local:4001
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: initial advertise peer URLs = http://oci-master-1.oci.oracle.local:2380
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: initial cluster = 0=http://oci-master-3.oci.oracle.local:2380,1=http://oci-master-2.oci.oracle.local:2380,oci=http://oci-master-1.oci.oracle.local:2380
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: starting member 20945348073f2613 in cluster 89cb34bd51d6a9f3
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: 20945348073f2613 became follower at term 0
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: newRaft 20945348073f2613 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: 20945348073f2613 became follower at term 1
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: simple token is not cryptographically signed
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: starting peer 4aef4be2f926f4ff...
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: started HTTP pipelining with peer 4aef4be2f926f4ff
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: started streaming with peer 4aef4be2f926f4ff (writer)
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: started streaming with peer 4aef4be2f926f4ff (writer)
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: started peer 4aef4be2f926f4ff
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: added peer 4aef4be2f926f4ff
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: started streaming with peer 4aef4be2f926f4ff (stream MsgApp v2 reader)
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: started streaming with peer 4aef4be2f926f4ff (stream Message reader)
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: starting peer a2bafbe2ed664069...
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: started HTTP pipelining with peer a2bafbe2ed664069
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: started streaming with peer a2bafbe2ed664069 (writer)
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: started streaming with peer a2bafbe2ed664069 (writer)
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: started peer a2bafbe2ed664069
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: started streaming with peer a2bafbe2ed664069 (stream MsgApp v2 reader)
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: started streaming with peer a2bafbe2ed664069 (stream Message reader)
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: added peer a2bafbe2ed664069
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: starting server... [version: 3.2.8, cluster version: to_be_decided]
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: added member 20945348073f2613 [http://oci-master-1.oci.oracle.local:2380] to cluster 89cb34bd51d6a9f3
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: added member 4aef4be2f926f4ff [http://oci-master-2.oci.oracle.local:2380] to cluster 89cb34bd51d6a9f3
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: started HTTP pipelining with peer 20945348073f2613
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: added member a2bafbe2ed664069 [http://oci-master-3.oci.oracle.local:2380] to cluster 89cb34bd51d6a9f3
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: 20945348073f2613 [term: 1] received a MsgVote message with higher term from 20945348073f2613 [term: 15]
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: 20945348073f2613 became follower at term 15
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: 20945348073f2613 [logterm: 1, index: 3, vote: 0] cast MsgVote for 20945348073f2613 [logterm: 1, index: 3] at term 15
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: peer 20945348073f2613 became active
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: 20945348073f2613 [term: 15] received a MsgVote message with higher term from 20945348073f2613 [term: 16]
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: 20945348073f2613 became follower at term 16
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: 20945348073f2613 [logterm: 1, index: 3, vote: 0] cast MsgVote for 20945348073f2613 [logterm: 1, index: 3] at term 16
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3
Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3
Oct 11 03:17:00 oci-master-3.oci.oracle.local etcd[1221]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3
hbscharp commented 5 years ago

Thanks, Denver. I've opened ticket ATD-210 internally to track this issue. I'm not surprised it doesnt work, since we didn't test that before releasing to you. I was unaware that multi-master was a test case. Seems strange to me since we're limited to a single node for the ONAP project?

On Wed, Oct 10, 2018 at 11:26 PM Denver Williams notifications@github.com wrote:

We have since run into some new troubles, with etcd this time (only happening with multi-master deploys)

Node 1

Oct 11 03:16:38 oci-master-1.oci.oracle.local systemd[1]: Starting etcd... Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=http://oci-master-1.oci.oracle.local:4001 Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: recognized and used environment variable ETCD_DATA_DIR=/var/etcd/data Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: recognized and used environment variable ETCD_DISCOVERY_SRV=oci.oracle.local Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=http://oci-master-1.oci.oracle.local:2380 Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=new Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: recognized and used environment variable ETCD_INITIAL_CLUSTER_TOKEN=oci-etcd Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:4001 Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: recognized and used environment variable ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380 Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: recognized and used environment variable ETCD_NAME=oci Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: etcd Version: 3.2.8 Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: Git SHA: e211fb6 Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: Go Version: go1.8.3 Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: Go OS/Arch: linux/amd64 Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: setting maximum number of CPUs to 8, total number of available CPUs is 8 Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: listening for peers on http://0.0.0.0:2380 Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: listening for client requests on 0.0.0.0:4001 Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: got bootstrap from DNS for etcd-server at 0=http://oci-master-3.oci.oracle.local:2380 Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: got bootstrap from DNS for etcd-server at 1=http://oci-master-2.oci.oracle.local:2380 Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: got bootstrap from DNS for etcd-server at oci=http://oci-master-1.oci.oracle.local:2380 Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: resolving oci-master-1.oci.oracle.local:2380 to 10.0.15.5:2380 Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: resolving oci-master-1.oci.oracle.local:2380 to 10.0.15.5:2380 Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: name = oci Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: data dir = /var/etcd/data Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: member dir = /var/etcd/data/member Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: heartbeat = 100ms Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: election = 1000ms Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: snapshot count = 100000 Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: advertise client URLs = http://oci-master-1.oci.oracle.local:4001 Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: initial advertise peer URLs = http://oci-master-1.oci.oracle.local:2380 Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: initial cluster = 0=http://oci-master-3.oci.oracle.local:2380,1=http://oci-master-2.oci.oracle.local:2380,oci=http://oci-master-1.oci.oracle.local:2380 Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: starting member 20945348073f2613 in cluster 89cb34bd51d6a9f3 Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: 20945348073f2613 became follower at term 0 Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: newRaft 20945348073f2613 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0] Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: 20945348073f2613 became follower at term 1 Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: simple token is not cryptographically signed Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: starting peer 4aef4be2f926f4ff... Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: started HTTP pipelining with peer 4aef4be2f926f4ff Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: started streaming with peer 4aef4be2f926f4ff (writer) Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: started streaming with peer 4aef4be2f926f4ff (writer) Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: started peer 4aef4be2f926f4ff Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: added peer 4aef4be2f926f4ff Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: starting peer a2bafbe2ed664069... Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: started HTTP pipelining with peer a2bafbe2ed664069 Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: started streaming with peer 4aef4be2f926f4ff (stream MsgApp v2 reader) Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: started streaming with peer 4aef4be2f926f4ff (stream Message reader) Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: started streaming with peer a2bafbe2ed664069 (writer) Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: started streaming with peer a2bafbe2ed664069 (writer) Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: started peer a2bafbe2ed664069 Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: added peer a2bafbe2ed664069 Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: started streaming with peer a2bafbe2ed664069 (stream MsgApp v2 reader) Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: starting server... [version: 3.2.8, cluster version: to_be_decided] Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: started streaming with peer a2bafbe2ed664069 (stream Message reader) Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: added member 20945348073f2613 [http://oci-master-1.oci.oracle.local:2380] to cluster 89cb34bd51d6a9f3 Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: added member 4aef4be2f926f4ff [http://oci-master-2.oci.oracle.local:2380] to cluster 89cb34bd51d6a9f3 Oct 11 03:16:38 oci-master-1.oci.oracle.local etcd[1222]: added member a2bafbe2ed664069 [http://oci-master-3.oci.oracle.local:2380] to cluster 89cb34bd51d6a9f3 Oct 11 03:16:39 oci-master-1.oci.oracle.local etcd[1222]: 20945348073f2613 is starting a new election at term 1 Oct 11 03:16:39 oci-master-1.oci.oracle.local etcd[1222]: 20945348073f2613 became candidate at term 2 Oct 11 03:16:39 oci-master-1.oci.oracle.local etcd[1222]: 20945348073f2613 received MsgVoteResp from 20945348073f2613 at term 2 Oct 11 03:16:39 oci-master-1.oci.oracle.local etcd[1222]: 20945348073f2613 [logterm: 1, index: 3] sent MsgVote request to 4aef4be2f926f4ff at term 2 Oct 11 03:16:39 oci-master-1.oci.oracle.local etcd[1222]: 20945348073f2613 [logterm: 1, index: 3] sent MsgVote request to a2bafbe2ed664069 at term 2 Oct 11 03:16:41 oci-master-1.oci.oracle.local etcd[1222]: 20945348073f2613 is starting a new election at term 2 Oct 11 03:16:41 oci-master-1.oci.oracle.local etcd[1222]: 20945348073f2613 became candidate at term 3 Oct 11 03:16:41 oci-master-1.oci.oracle.local etcd[1222]: 20945348073f2613 received MsgVoteResp from 20945348073f2613 at term 3 Oct 11 03:16:41 oci-master-1.oci.oracle.local etcd[1222]: 20945348073f2613 [logterm: 1, index: 3] sent MsgVote request to 4aef4be2f926f4ff at term 3 Oct 11 03:16:41 oci-master-1.oci.oracle.local etcd[1222]: 20945348073f2613 [logterm: 1, index: 3] sent MsgVote request to a2bafbe2ed664069 at term 3 Oct 11 03:16:42 oci-master-1.oci.oracle.local etcd[1222]: 20945348073f2613 is starting a new election at term 3 Oct 11 03:16:42 oci-master-1.oci.oracle.local etcd[1222]: 20945348073f2613 became candidate at term 4 Oct 11 03:16:42 oci-master-1.oci.oracle.local etcd[1222]: 20945348073f2613 received MsgVoteResp from 20945348073f2613 at term 4 Oct 11 03:16:42 oci-master-1.oci.oracle.local etcd[1222]: 20945348073f2613 [logterm: 1, index: 3] sent MsgVote request to 4aef4be2f926f4ff at term 4 Oct 11 03:16:42 oci-master-1.oci.oracle.local etcd[1222]: 20945348073f2613 [logterm: 1, index: 3] sent MsgVote request to a2bafbe2ed664069 at term 4 Oct 11 03:16:43 oci-master-1.oci.oracle.local etcd[1222]: health check for peer 4aef4be2f926f4ff could not connect: dial tcp 10.0.15.3:2380: getsockopt: connection refused Oct 11 03:16:43 oci-master-1.oci.oracle.local etcd[1222]: health check for peer a2bafbe2ed664069 could not connect: dial tcp 10.0.15.4:2380: getsockopt: connection refused Oct 11 03:16:44 oci-master-1.oci.oracle.local etcd[1222]: 20945348073f2613 is starting a new election at term 4 Oct 11 03:16:44 oci-master-1.oci.oracle.local etcd[1222]: 20945348073f2613 became candidate at term 5 Oct 11 03:16:44 oci-master-1.oci.oracle.local etcd[1222]: 20945348073f2613 received MsgVoteResp from 20945348073f2613 at term 5 Oct 11 03:16:44 oci-master-1.oci.oracle.local etcd[1222]: 20945348073f2613 [logterm: 1, index: 3] sent MsgVote request to 4aef4be2f926f4ff at term 5 Oct 11 03:16:44 oci-master-1.oci.oracle.local etcd[1222]: 20945348073f2613 [logterm: 1, index: 3] sent MsgVote request to a2bafbe2ed664069 at term 5 Oct 11 03:16:45 oci-master-1.oci.oracle.local etcd[1222]: publish error: etcdserver: request timed out

Node 2

Oct 11 03:17:20 oci-master-2.oci.oracle.local systemd[1]: Starting etcd... Oct 11 03:17:20 oci-master-2.oci.oracle.local etcd[1220]: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=http://oci-master-1.oci.oracle.local:4001 Oct 11 03:17:20 oci-master-2.oci.oracle.local etcd[1220]: recognized and used environment variable ETCD_DATA_DIR=/var/etcd/data Oct 11 03:17:20 oci-master-2.oci.oracle.local etcd[1220]: recognized and used environment variable ETCD_DISCOVERY_SRV=oci.oracle.local Oct 11 03:17:20 oci-master-2.oci.oracle.local etcd[1220]: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=http://oci-master-1.oci.oracle.local:2380 Oct 11 03:17:20 oci-master-2.oci.oracle.local etcd[1220]: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=new Oct 11 03:17:20 oci-master-2.oci.oracle.local etcd[1220]: recognized and used environment variable ETCD_INITIAL_CLUSTER_TOKEN=oci-etcd Oct 11 03:17:20 oci-master-2.oci.oracle.local etcd[1220]: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:4001 Oct 11 03:17:20 oci-master-2.oci.oracle.local etcd[1220]: recognized and used environment variable ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380 Oct 11 03:17:20 oci-master-2.oci.oracle.local etcd[1220]: recognized and used environment variable ETCD_NAME=oci Oct 11 03:17:20 oci-master-2.oci.oracle.local etcd[1220]: etcd Version: 3.2.8 Oct 11 03:17:20 oci-master-2.oci.oracle.local etcd[1220]: Git SHA: e211fb6 Oct 11 03:17:20 oci-master-2.oci.oracle.local etcd[1220]: Go Version: go1.8.3 Oct 11 03:17:20 oci-master-2.oci.oracle.local etcd[1220]: Go OS/Arch: linux/amd64 Oct 11 03:17:20 oci-master-2.oci.oracle.local etcd[1220]: setting maximum number of CPUs to 8, total number of available CPUs is 8 Oct 11 03:17:20 oci-master-2.oci.oracle.local etcd[1220]: listening for peers on http://0.0.0.0:2380 Oct 11 03:17:20 oci-master-2.oci.oracle.local etcd[1220]: listening for client requests on 0.0.0.0:4001 Oct 11 03:17:20 oci-master-2.oci.oracle.local etcd[1220]: got bootstrap from DNS for etcd-server at oci=http://oci-master-1.oci.oracle.local:2380 Oct 11 03:17:20 oci-master-2.oci.oracle.local etcd[1220]: got bootstrap from DNS for etcd-server at 0=http://oci-master-2.oci.oracle.local:2380 Oct 11 03:17:20 oci-master-2.oci.oracle.local etcd[1220]: got bootstrap from DNS for etcd-server at 1=http://oci-master-3.oci.oracle.local:2380 Oct 11 03:17:20 oci-master-2.oci.oracle.local etcd[1220]: resolving oci-master-1.oci.oracle.local:2380 to 10.0.15.5:2380 Oct 11 03:17:20 oci-master-2.oci.oracle.local etcd[1220]: resolving oci-master-1.oci.oracle.local:2380 to 10.0.15.5:2380 Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: name = oci Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: data dir = /var/etcd/data Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: member dir = /var/etcd/data/member Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: heartbeat = 100ms Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: election = 1000ms Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: snapshot count = 100000 Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: advertise client URLs = http://oci-master-1.oci.oracle.local:4001 Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: initial advertise peer URLs = http://oci-master-1.oci.oracle.local:2380 Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: initial cluster = 0=http://oci-master-2.oci.oracle.local:2380,1=http://oci-master-3.oci.oracle.local:2380,oci=http://oci-master-1.oci.oracle.local:2380 Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: starting member 20945348073f2613 in cluster 89cb34bd51d6a9f3 Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: 20945348073f2613 became follower at term 0 Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: newRaft 20945348073f2613 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0] Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: 20945348073f2613 became follower at term 1 Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: simple token is not cryptographically signed Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: starting peer 4aef4be2f926f4ff... Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: started HTTP pipelining with peer 4aef4be2f926f4ff Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: started streaming with peer 4aef4be2f926f4ff (writer) Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: started streaming with peer 4aef4be2f926f4ff (writer) Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: started peer 4aef4be2f926f4ff Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: added peer 4aef4be2f926f4ff Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: starting peer a2bafbe2ed664069... Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: started streaming with peer 4aef4be2f926f4ff (stream MsgApp v2 reader) Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: started streaming with peer 4aef4be2f926f4ff (stream Message reader) Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: started HTTP pipelining with peer a2bafbe2ed664069 Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: started streaming with peer a2bafbe2ed664069 (writer) Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: started streaming with peer a2bafbe2ed664069 (writer) Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: started streaming with peer a2bafbe2ed664069 (stream MsgApp v2 reader) Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: started peer a2bafbe2ed664069 Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: started streaming with peer a2bafbe2ed664069 (stream Message reader) Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: added peer a2bafbe2ed664069 Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: starting server... [version: 3.2.8, cluster version: to_be_decided] Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: added member 20945348073f2613 [http://oci-master-1.oci.oracle.local:2380] to cluster 89cb34bd51d6a9f3 Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: added member 4aef4be2f926f4ff [http://oci-master-2.oci.oracle.local:2380] to cluster 89cb34bd51d6a9f3 Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: started HTTP pipelining with peer 20945348073f2613 Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3 Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: added member a2bafbe2ed664069 [http://oci-master-3.oci.oracle.local:2380] to cluster 89cb34bd51d6a9f3 Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3 Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3 Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: 20945348073f2613 [term: 1] received a MsgVote message with higher term from 20945348073f2613 [term: 32] Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3 Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: 20945348073f2613 became follower at term 32 Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: 20945348073f2613 [logterm: 1, index: 3, vote: 0] cast MsgVote for 20945348073f2613 [logterm: 1, index: 3] at term 32 Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3 Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3 Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: peer 20945348073f2613 became active Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3 Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3 Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3 Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3 Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3 Oct 11 03:17:21 oci-master-2.oci.oracle.local etcd[1220]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3 Oct 11 03:17:22 oci-master-2.oci.oracle.local etcd[1220]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3

Node 3

-- Logs begin at Thu 2018-10-11 03:16:08 UTC, end at Thu 2018-10-11 03:25:09 UTC. -- Oct 11 03:16:57 oci-master-3.oci.oracle.local systemd[1]: Starting etcd... Oct 11 03:16:57 oci-master-3.oci.oracle.local etcd[1221]: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=http://oci-master-1.oci.oracle.local:4001 Oct 11 03:16:57 oci-master-3.oci.oracle.local etcd[1221]: recognized and used environment variable ETCD_DATA_DIR=/var/etcd/data Oct 11 03:16:57 oci-master-3.oci.oracle.local etcd[1221]: recognized and used environment variable ETCD_DISCOVERY_SRV=oci.oracle.local Oct 11 03:16:57 oci-master-3.oci.oracle.local etcd[1221]: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=http://oci-master-1.oci.oracle.local:2380 Oct 11 03:16:57 oci-master-3.oci.oracle.local etcd[1221]: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=new Oct 11 03:16:57 oci-master-3.oci.oracle.local etcd[1221]: recognized and used environment variable ETCD_INITIAL_CLUSTER_TOKEN=oci-etcd Oct 11 03:16:57 oci-master-3.oci.oracle.local etcd[1221]: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:4001 Oct 11 03:16:57 oci-master-3.oci.oracle.local etcd[1221]: recognized and used environment variable ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380 Oct 11 03:16:57 oci-master-3.oci.oracle.local etcd[1221]: recognized and used environment variable ETCD_NAME=oci Oct 11 03:16:57 oci-master-3.oci.oracle.local etcd[1221]: etcd Version: 3.2.8 Oct 11 03:16:57 oci-master-3.oci.oracle.local etcd[1221]: Git SHA: e211fb6 Oct 11 03:16:57 oci-master-3.oci.oracle.local etcd[1221]: Go Version: go1.8.3 Oct 11 03:16:57 oci-master-3.oci.oracle.local etcd[1221]: Go OS/Arch: linux/amd64 Oct 11 03:16:57 oci-master-3.oci.oracle.local etcd[1221]: setting maximum number of CPUs to 8, total number of available CPUs is 8 Oct 11 03:16:57 oci-master-3.oci.oracle.local etcd[1221]: listening for peers on http://0.0.0.0:2380 Oct 11 03:16:57 oci-master-3.oci.oracle.local etcd[1221]: listening for client requests on 0.0.0.0:4001 Oct 11 03:16:58 oci-master-3.oci.oracle.local etcd[1221]: got bootstrap from DNS for etcd-server at 0=http://oci-master-3.oci.oracle.local:2380 Oct 11 03:16:58 oci-master-3.oci.oracle.local etcd[1221]: got bootstrap from DNS for etcd-server at 1=http://oci-master-2.oci.oracle.local:2380 Oct 11 03:16:58 oci-master-3.oci.oracle.local etcd[1221]: got bootstrap from DNS for etcd-server at oci=http://oci-master-1.oci.oracle.local:2380 Oct 11 03:16:58 oci-master-3.oci.oracle.local etcd[1221]: resolving oci-master-1.oci.oracle.local:2380 to 10.0.15.5:2380 Oct 11 03:16:58 oci-master-3.oci.oracle.local etcd[1221]: resolving oci-master-1.oci.oracle.local:2380 to 10.0.15.5:2380 Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: name = oci Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: data dir = /var/etcd/data Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: member dir = /var/etcd/data/member Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: heartbeat = 100ms Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: election = 1000ms Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: snapshot count = 100000 Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: advertise client URLs = http://oci-master-1.oci.oracle.local:4001 Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: initial advertise peer URLs = http://oci-master-1.oci.oracle.local:2380 Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: initial cluster = 0=http://oci-master-3.oci.oracle.local:2380,1=http://oci-master-2.oci.oracle.local:2380,oci=http://oci-master-1.oci.oracle.local:2380 Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: starting member 20945348073f2613 in cluster 89cb34bd51d6a9f3 Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: 20945348073f2613 became follower at term 0 Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: newRaft 20945348073f2613 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0] Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: 20945348073f2613 became follower at term 1 Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: simple token is not cryptographically signed Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: starting peer 4aef4be2f926f4ff... Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: started HTTP pipelining with peer 4aef4be2f926f4ff Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: started streaming with peer 4aef4be2f926f4ff (writer) Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: started streaming with peer 4aef4be2f926f4ff (writer) Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: started peer 4aef4be2f926f4ff Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: added peer 4aef4be2f926f4ff Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: started streaming with peer 4aef4be2f926f4ff (stream MsgApp v2 reader) Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: started streaming with peer 4aef4be2f926f4ff (stream Message reader) Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: starting peer a2bafbe2ed664069... Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: started HTTP pipelining with peer a2bafbe2ed664069 Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: started streaming with peer a2bafbe2ed664069 (writer) Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: started streaming with peer a2bafbe2ed664069 (writer) Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: started peer a2bafbe2ed664069 Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: started streaming with peer a2bafbe2ed664069 (stream MsgApp v2 reader) Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: started streaming with peer a2bafbe2ed664069 (stream Message reader) Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: added peer a2bafbe2ed664069 Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: starting server... [version: 3.2.8, cluster version: to_be_decided] Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: added member 20945348073f2613 [http://oci-master-1.oci.oracle.local:2380] to cluster 89cb34bd51d6a9f3 Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: added member 4aef4be2f926f4ff [http://oci-master-2.oci.oracle.local:2380] to cluster 89cb34bd51d6a9f3 Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: started HTTP pipelining with peer 20945348073f2613 Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3 Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: added member a2bafbe2ed664069 [http://oci-master-3.oci.oracle.local:2380] to cluster 89cb34bd51d6a9f3 Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3 Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: 20945348073f2613 [term: 1] received a MsgVote message with higher term from 20945348073f2613 [term: 15] Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: 20945348073f2613 became follower at term 15 Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: 20945348073f2613 [logterm: 1, index: 3, vote: 0] cast MsgVote for 20945348073f2613 [logterm: 1, index: 3] at term 15 Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3 Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3 Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: peer 20945348073f2613 became active Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3 Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3 Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3 Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3 Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: 20945348073f2613 [term: 15] received a MsgVote message with higher term from 20945348073f2613 [term: 16] Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: 20945348073f2613 became follower at term 16 Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: 20945348073f2613 [logterm: 1, index: 3, vote: 0] cast MsgVote for 20945348073f2613 [logterm: 1, index: 3] at term 16 Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3 Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3 Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3 Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3 Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3 Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3 Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3 Oct 11 03:16:59 oci-master-3.oci.oracle.local etcd[1221]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3 Oct 11 03:17:00 oci-master-3.oci.oracle.local etcd[1221]: failed to find member 20945348073f2613 in cluster 89cb34bd51d6a9f3

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/crosscloudci/cross-cloud/pull/168#issuecomment-428807589, or mute the thread https://github.com/notifications/unsubscribe-auth/ACzCxzMpR-YcsQQU-dplZct7AJy-EeT8ks5ujrpogaJpZM4WiMbQ .

denverwilliams commented 5 years ago

@hbscharp thanks for looking into this. In regards to the test case you're right we can only use one node for the onap deployments, however it's only a limitation for the "worker" nodes of the cluster. Which means we can deploy a multi-master cluster without any issues - also the kubernetes e2e tests don't normally run to well if it's not a multi-master cluster.

Many Thanks, Denver.

hbscharp commented 5 years ago

Denver,

Unfortunately I am unable to reproduce your issue. I do have a change I can submit to get 3 master nodes up by default. How did you increase the number of master nodes, and have you changed any versions of Kube packages in your deployment? I do see some socket errors in etcd in my cluster, but then the cluster recovers and behaves normally. Does your etcd issue still persist?

Thanks,

-Ben

On Mon, Oct 15, 2018 at 4:43 PM Denver Williams notifications@github.com wrote:

@hbscharp https://github.com/hbscharp thanks for looking into this. In regards to the test case you're right we can only use one node for the onap deployments, however it's only a limitation for the "worker" nodes of the cluster. Which means we can deploy a multi-master cluster without any issues - also the kubernetes e2e tests don't normally run to well if it's not a multi-master cluster.

Many Thanks, Denver.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/crosscloudci/cross-cloud/pull/168#issuecomment-430005985, or mute the thread https://github.com/notifications/unsubscribe-auth/ACzCx73API4oncFrzSeeqBxqXCMthMt4ks5ulPNhgaJpZM4WiMbQ .