timeout on deployment for gke 1.6.7 and 1.7.2

[ambition-consulting]

Got:

Loading fabric8 releases from maven repository:https://repo1.maven.org/maven2/ Deploying package: platform version: 2.4.24

Then nothing, for at least 15 minutes, gave up after that.

[ambition-consulting]

tested for both us-central1-a and europe-west3-c.

[jstrachan]

@rawlingsj I wonder if this is a recent regression in gofabric8 for the 4.x fabric8 stuff?

@ambition-consulting I wonder if you tried using an older gofabric8 things are OK? FWIW we're very close to releasing fabric8 4.x which is a little different: https://github.com/fabric8io/fabric8-platform#fabric8-platform-overview - we're just trying to nail down automated testing across different kubernetes/openshift clusters before we release but its hopefully in the next week or two.

Incidentally you could give the 4.x a try via these instructions - I wonder if that works any better for you? https://github.com/fabric8io/fabric8-platform/blob/master/INSTALL.md we've not yet updated the website until its ready

[rawlingsj]

@ambition-consulting @jstrachan dam - yeah sounds like a change for the new 4.x deployment has affected the current approach. I'll take a look.

[jstrachan]

@rawlingsj we may need to wrap any 4.x specific code in gofabric8 deploy with the code if !legacy {...}

[soloman747]

@ambition-consulting yes if I try 0.4.141, gofabric deploys with no problem. 0.4.142 hangs on deployment of ingress, and 0.4.145 doesn't even get past "Deploying package: platform version: 2.4.24." I'm currently running 0.4.141 on kubernetes stable. I should point out that the flags I'm using are:

"gofabric8 deploy -y --namespace fabric8 --open-console=false --loadbalancer=true"

[soloman747]

This is also on Microsoft Azure.

[soloman747]

I just tried release 4. It freezes as well, but this time on system.

`Loading fabric8 releases from maven repository:https://repo1.maven.org/maven2/ Continue? [Y/n] y Deploying package: system version: 4.0.147

Now about to install package https://repo1.maven.org/maven2/io/fabric8/platform/packages/fabric8-system/4.0.147/fabric8-system-4.0.147-k8s-template.yml Template APISERVER_HOSTPORT = REDACTED Template NODE_IP = Template EXPOSER = Ingress Template DOMAIN = .nip.io Template GITHUB_OAUTH_CLIENT_SECRET = REDACTED Template GITHUB_OAUTH_CLIENT_ID = REDACTED Template HTTP = Template TLS_ACME = Template OAUTH_AUTHORIZE_PORT = Template APISERVER = Template NAMESPACE = fabric8 Creating system template resources in namespace fabric8 from 45 objects Processing resource kind: Secret in namespace fabric8 name f8tenant Processing resource kind: Secret in namespace fabric8 name keycloak Processing resource kind: Secret in namespace fabric8 name wit Processing resource kind: ServiceAccount in namespace fabric8 name configmapcontroller Processing resource kind: ServiceAccount in namespace fabric8 name exposecontroller Processing resource kind: ServiceAccount in namespace fabric8 name init-tenant Processing resource kind: Service in namespace fabric8 name che-starter Processing resource kind: Service in namespace fabric8 name f8tenant Processing resource kind: Service in namespace fabric8 name fabric8 Processing resource kind: Service in namespace fabric8 name fabric8-docker-registry Processing resource kind: Service in namespace fabric8 name forge Processing resource kind: Service in namespace fabric8 name init-tenant-db Processing resource kind: Service in namespace fabric8 name keycloak Processing resource kind: Service in namespace fabric8 name keycloak-db Processing resource kind: Service in namespace fabric8 name sso Processing resource kind: Service in namespace fabric8 name wit Processing resource kind: Service in namespace fabric8 name wit-db Processing resource kind: PersistentVolumeClaim in namespace fabric8 name fabric8-docker-registry-storage Processing resource kind: PersistentVolumeClaim in namespace fabric8 name init-tenant-postgresql-data Processing resource kind: PersistentVolumeClaim in namespace fabric8 name keycloak-db-postgresql-data Processing resource kind: PersistentVolumeClaim in namespace fabric8 name wit-postgresql-data Processing resource kind: ConfigMap in namespace fabric8 name che-starter Processing resource kind: ConfigMap in namespace fabric8 name exposecontroller Processing resource kind: ConfigMap in namespace fabric8 name f8tenant Processing resource kind: ConfigMap in namespace fabric8 name fabric8 Processing resource kind: ConfigMap in namespace fabric8 name fabric8-system Processing resource kind: ConfigMap in namespace fabric8 name forge Processing resource kind: ConfigMap in namespace fabric8 name keycloak Processing resource kind: ConfigMap in namespace fabric8 name sso Processing resource kind: ConfigMap in namespace fabric8 name wit Processing resource kind: CustomResourceDefinition in namespace fabric8 name buildconfigs.build.openshift.io Processing resource kind: CustomResourceDefinition in namespace fabric8 name builds.build.openshift.io Processing resource kind: CustomResourceDefinition in namespace fabric8 name templates.template.openshift.io Processing resource kind: Deployment in namespace fabric8 name che-starter Processing resource kind: Deployment in namespace fabric8 name configmapcontroller Processing resource kind: Deployment in namespace fabric8 name docker-registry Processing resource kind: Deployment in namespace fabric8 name exposecontroller Processing resource kind: Deployment in namespace fabric8 name fabric8 Processing resource kind: Deployment in namespace fabric8 name forge Processing resource kind: Deployment in namespace fabric8 name init-tenant Processing resource kind: Deployment in namespace fabric8 name init-tenant-db Processing resource kind: Deployment in namespace fabric8 name keycloak Processing resource kind: Deployment in namespace fabric8 name keycloak-db Processing resource kind: Deployment in namespace fabric8 name wit Processing resource kind: Deployment in namespace fabric8 name wit-db system........................................................................✔ ..................... `

[rawlingsj]

I just merged a commit which will hopefully fix the first issue. The release has started so should be ready in ~7 mins

[rawlingsj]

As for the 4.x; are you using your own ingress controller? It sounds like maybe services haven't started or ingress rules have not been generate. Can you check if all the pods are running? I tend to use kubectl get pods --all-namespaces --show-all just in case.

[soloman747]

I'm exposing using the default nginx ingress. I'm deploying on a kubernetes cluster in Azure using a loadbalancer. So all my services are exposed with a public IP.

NAMESPACE NAME READY STATUS RESTARTS AGE fabric8 che-starter-1417136367-2r5vf 1/1 Running 0 1h fabric8 configmapcontroller-4273343753-81mrs 1/1 Running 0 17h fabric8 docker-registry-3403729758-41pkp 0/1 Init:0/1 0 1h fabric8 exposecontroller-1070912791-04gd4 1/1 Running 0 17h fabric8 fabric8-3873669821-08vpb 2/2 Running 0 17h fabric8 fabric8-docker-registry-125311296-d7p0m 1/1 Running 0 17h fabric8 fabric8-forge-1088523184-p0mbq 1/1 Running 0 17h fabric8 forge-432422340-zjqq6 1/1 Running 0 1h fabric8 gogs-2621620945-mxn6j 1/1 Running 0 17h fabric8 init-tenant-1561764138-0gl9n 1/1 Running 6 1h fabric8 init-tenant-db-1370173110-3m23s 1/1 Running 0 1h fabric8 jenkins-56914896-719c5 1/1 Running 0 17h fabric8 keycloak-3874412563-mnn16 1/1 Running 0 1h fabric8 keycloak-db-2585062680-749p3 1/1 Running 0 1h fabric8 nexus-2230784709-64p5x 1/1 Running 0 17h fabric8 wit-3251094805-w881p 1/1 Running 4 1h fabric8 wit-db-159381727-0g1h8 1/1 Running 0 1h fabric8-production testapp2-2816705835-1jn8v 1/1 Running 0 17h fabric8-staging testapp2-2816705835-0s4cx 1/1 Running 0 17h fabric8-system ingress-nginx-775506543-nwz05 1/1 Running 0 17h kube-system heapster-2708163903-kp7gn 2/2 Running 0 18s kube-system heapster-2708163903-mmkg6 0/2 Terminating 0 17h kube-system kube-addon-manager-k8s-master-612bd988-0 1/1 Running 0 18h kube-system kube-apiserver-k8s-master-612bd988-0 1/1 Running 0 18h kube-system kube-controller-manager-k8s-master-612bd988-0 1/1 Running 0 18h kube-system kube-dns-v20-g5kz5 3/3 Running 4 18h kube-system kube-dns-v20-qctx7 3/3 Running 4 18h kube-system kube-proxy-16b79 1/1 Running 0 18h kube-system kube-proxy-9cvr7 1/1 Running 0 18h kube-system kube-proxy-9hbm5 1/1 Running 0 18h kube-system kube-proxy-mgznq 1/1 Running 0 18h kube-system kube-scheduler-k8s-master-612bd988-0 1/1 Running 0 18h kube-system kubernetes-dashboard-3995387264-m19f5 0/1 CrashLoopBackOff 10 18h kube-system tiller-deploy-3019006398-j7gmq 1/1 Running 3 18h nginx-ingress default-backend-1365105359-2sxzq 1/1 Running 1 17h nginx-ingress nginx-ingress-4004635195-gfwzc 1/1 Running 6 17h

[rawlingsj]

I've not tried this on Azure yet I'm afraid. On GKE and AWS they give the nginx-ingress service an external IP which gofabric8 waits for. My guess is this isn't happening or is different to how GKE does it which is where I've done my testing.

I'll try and get an Azure account set up this evening to test and hopefully get this working real soon!

[ambition-consulting]

@rawlingsj

λ kubectl get pods --all-namespaces --show-all
NAMESPACE     NAME                                                READY     STATUS    RESTARTS   AGE
kube-system   event-exporter-v0.1.4-4272745813-gph5m              2/2       Running   0          22h
kube-system   fluentd-gcp-v2.0-5l27q                              2/2       Running   0          22h
kube-system   fluentd-gcp-v2.0-hmhwt                              2/2       Running   0          22h
kube-system   heapster-v1.4.0-784784086-d8rx6                     2/2       Running   0          22h
kube-system   kube-dns-3557498097-0pm6q                           3/3       Running   0          22h
kube-system   kube-dns-3557498097-5dhph                           3/3       Running   0          22h
kube-system   kube-dns-autoscaler-97162954-np82n                  1/1       Running   0          22h
kube-system   kube-proxy-gke-fabric8-default-pool-4a6c4211-8jc9   1/1       Running   0          22h
kube-system   kube-proxy-gke-fabric8-default-pool-4a6c4211-w1lr   1/1       Running   0          22h
kube-system   kubernetes-dashboard-1914926742-3qzvx               1/1       Running   0          22h
kube-system   l7-default-backend-1798834265-k37c5                 1/1       Running   0          22h

@jstrachan I have followed https://github.com/fabric8io/fabric8-platform/blob/master/INSTALL.md closely - resultign in the above-mentioned timeout:

When using

gofabric8 deploy --package system

I get the timeout after:`

Loading fabric8 releases from maven repository:https://repo1.maven.org/maven2/ Continue? [Y/n] Y Deploying package: system version: 4.0.147

When using

gofabric8 deploy --package platform

it will deploy 3.0 succesfully.

Any plans on when 4.0 might work on gke? The new features are amazing! 👍

[rawlingsj]

We're almost there with 4.x on GKE, fingers crossed we'll have a release next week and hopefully we have it ready for Azure then too.

BTW I'm not familiar with Azure, I just signed up and created a cluster but it defaults to Kubernetes v1.6.6. In this issue you mentioned you'd tried gofabric8 with v1.7.2 which fabric8 4.x makes use of the new CustomDefinedResource type. I'll keep googling but so far I've not figured out how to create a v1.7.2 cluster.. any pointers would be appreciated :) Once I get a cluster going hopefully it wont be too much to figure out why gofabric8 hangs as you describe.

[soloman747]

@rawlingsj there may have been some misunderstanding. The Azure kubernetes version I'm running gofabric on is 1.6.6. I think the 1.7.2 version is from @ambition-consulting on GKE.