Deploying any Armada Chart fails with "Rendezvous of RPC that terminated with (StatusCode.UNAVAILABLE, Connect Failed"

[moffzilla]

If this is a BUG REPORT, please: I fail to deploy any of the Armada yamls found in the repo with the error:

_2017-08-03 22:18:07.200 162 ERROR armada [-] <_Rendezvous of RPC that terminated with (StatusCode.UNAVAILABLE, Connect Failed)>: _Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNAVAILABLE, Connect Failed)>_

Regardless of the Armada yamls I try I get the same error when using latest or Master or v0.6.0 Docker Armada versions, When checking Tiller by Armada command or Kubectl , it shows active here's the outcome from the simple.yaml test (I attach the file with the debug-logging enabled ) armada-simple-debug-fail.txt

_docker exec armada armada apply examples/simple.yaml found extension EntryPoint.parse('pika = oslo_messaging._drivers.impl_pika:PikaDriver') found extension EntryPoint.parse('kafka = oslo_messaging._drivers.impl_kafka:KafkaDriver') found extension EntryPoint.parse('kombu = oslo_messaging._drivers.impl_rabbit:RabbitDriver') found extension EntryPoint.parse('rabbit = oslo_messaging._drivers.impl_rabbit:RabbitDriver') found extension EntryPoint.parse('fake = oslo_messaging._drivers.impl_fake:FakeDriver') found extension EntryPoint.parse('zmq = oslo_messaging._drivers.impl_zmq:ZmqDriver') found extension EntryPoint.parse('amqp = oslo_messaging._drivers.impl_amqp1:ProtonDriver') found extension EntryPoint.parse('log = oslo_messaging.notify._impl_log:LogDriver') found extension EntryPoint.parse('messagingv2 = oslo_messaging.notify.messaging:MessagingV2Driver') found extension EntryPoint.parse('noop = oslo_messaging.notify._impl_noop:NoOpDriver') found extension EntryPoint.parse('routing = oslo_messaging.notify._impl_routing:RoutingDriver') found extension EntryPoint.parse('test = oslo_messaging.notify._impl_test:TestDriver') found extension EntryPoint.parse('messaging = oslo_messaging.notify.messaging:MessagingDriver') 2017-08-03 22:17:47.144 162 INFO armada.handlers.armada [-] Performing Pre-Flight Operations 2017-08-03 22:18:07.200 162 ERROR armada [-] <_Rendezvous of RPC that terminated with (StatusCode.UNAVAILABLE, Connect Failed)>: _Rendezvous: <Rendezvous of RPC that terminated with (StatusCode.UNAVAILABLE, Connect Failed)>

Deployment Environment?: Kubernetes: 1.7.0 Docker: 17.06.0-ce Helm v2.5.0 Ubuntu Xenial All hosts are deployed in aws/us-east

Kubernetes 1.7.0 Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.0", GitCommit:"d3ada0119e776222f11ec7945e6d860061339aad", GitTreeState:"clean", BuildDate:"2017-06-29T23:15:59Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.0", GitCommit:"d3ada0119e776222f11ec7945e6d860061339aad", GitTreeState:"clean", BuildDate:"2017-06-29T22:55:19Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Helm v2.5.0 helm version Client: &version.Version{SemVer:"v2.5.0", GitCommit:"012cb0ac1a1b2f888144ef5a67b8dab6c2d45be6", GitTreeState:"clean"} Server: &version.Version{SemVer:"v2.5.0", GitCommit:"012cb0ac1a1b2f888144ef5a67b8dab6c2d45be6", GitTreeState:"clean"}

docker version Client: Version: 17.06.0-ce API version: 1.30 Go version: go1.8.3 Git commit: 02c1d87 Built: Fri Jun 23 21:23:31 2017 OS/Arch: linux/amd64

Server: Version: 17.06.0-ce API version: 1.30 (minimum version 1.12) Go version: go1.8.3 Git commit: 02c1d87 Built: Fri Jun 23 21:19:04 2017 OS/Arch: linux/amd64 Experimental: false

Armada Status seems OK:

docker exec armada armada tiller --status found extension EntryPoint.parse('pika = oslo_messaging._drivers.impl_pika:PikaDriver') found extension EntryPoint.parse('kafka = oslo_messaging._drivers.impl_kafka:KafkaDriver') found extension EntryPoint.parse('kombu = oslo_messaging._drivers.impl_rabbit:RabbitDriver') found extension EntryPoint.parse('rabbit = oslo_messaging._drivers.impl_rabbit:RabbitDriver') found extension EntryPoint.parse('fake = oslo_messaging._drivers.impl_fake:FakeDriver') found extension EntryPoint.parse('zmq = oslo_messaging._drivers.impl_zmq:ZmqDriver') found extension EntryPoint.parse('amqp = oslo_messaging._drivers.impl_amqp1:ProtonDriver') found extension EntryPoint.parse('log = oslo_messaging.notify._impl_log:LogDriver') found extension EntryPoint.parse('messagingv2 = oslo_messaging.notify.messaging:MessagingV2Driver') found extension EntryPoint.parse('noop = oslo_messaging.notify._impl_noop:NoOpDriver') found extension EntryPoint.parse('routing = oslo_messaging.notify._impl_routing:RoutingDriver') found extension EntryPoint.parse('test = oslo_messaging.notify._impl_test:TestDriver') found extension EntryPoint.parse('messaging = oslo_messaging.notify.messaging:MessagingDriver') 2017-08-03 22:10:49.130 100 INFO armada.cli.tiller [-] Tiller is Active: True

kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE heapster-v1.4.0-2118284389-fl6kw 4/4 Running 0 3h kube-dns-3097350089-n2rcm 3/3 Running 0 3h kubernetes-dashboard-1265873680-xhbs4 1/1 Running 0 3h monitoring-influxdb-grafana-v4-8bbdt 2/2 Running 0 3h tiller-deploy-3235729489-j84j1 1/1 Running 0 3h

Release Tag or Master: Latest

Expected Behavior: Armada able to deploy the repo sample yamls

What Actually Happened: Any Armada yaml fails with the same error "_ERROR armada [-] <_Rendezvous of RPC that terminated with (StatusCode.UNAVAILABLE, Connect Failed)>: _Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNAVAILABLE, Connect Failed)>_"

How to Reproduce the Issue (as minimally as possible):

docker run -d --net host -p 8000:8000 --name armada -v /home/ubuntu/.kube/config:/armada/.kube/config -v /home/ubuntu/armada/examples:/armada/examples quay.io/attcomdev/armada:latest

docker exec armada armada apply examples/armada-aio.yaml --debug-logging or docker run -d --net host -p 8000:8000 --name armada -v /home/ubuntu/.kube/config:/armada/.kube/config -v /home/ubuntu/armada/examples:/armada/examples quay.io/attcomdev/armada:latest

Any Additional Comments: Is there any ports I need to make sure they are open among the hosts besides the 8000?

[alanmeadows]

Thanks for the report. Are there any relevant log lines from the tiller pod itself regarding the connection attempt?

kubectl get logs tiller..

[moffzilla]

Hi Alan

Thanks for your reply, FYI, My K8 deployment had originally 3 workers but I have rebuild it with just one however same behavior The Tiller logs doesn't provide much information after it is initialized I enabled log verbosity at the Docker Armada but not much information there ( I attach the output )

armada-simple-debug-fail.txt armada08042017.txt

kubectl logs -f tiller-deploy-3235729489-zz6ds -n kube-system [main] 2017/08/04 14:27:43 Starting Tiller v2.5.0 (tls=false) [main] 2017/08/04 14:27:43 GRPC listening on :44134 [main] 2017/08/04 14:27:43 Probes listening on :44135 [main] 2017/08/04 14:27:43 Storage driver is ConfigMap

Or from the Pod:

Events: FirstSeen LastSeen Count From SubObjectPath Type Reason Message

---

30s 30s 1 default-scheduler Normal Scheduled Successfully assigned tiller-deploy-3235729489-mbw4t to ip-172-31-60-242 29s 29s 1 kubelet, ip-172-31-60-242 Normal SuccessfulMountVolume MountVolume.SetUp succeeded for volume "default-token-3shbz" 29s 29s 1 kubelet, ip-172-31-60-242 spec.containers{tiller} Normal Pulled Container image "gcr.io/kubernetes-helm/tiller:v2.5.0" already present on machine 29s 29s 1 kubelet, ip-172-31-60-242 spec.containers{tiller} Normal Created Created container 29s 29s 1 kubelet, ip-172-31-60-242 spec.containers{tiller} Normal Started Started container

[SharpEdgeMarshall]

Same issue here, there is no solution?