fabric8io / fabric8

fabric8 is an open source microservices platform based on Docker, Kubernetes and Jenkins
http://fabric8.io/
1.76k stars 504 forks source link

Vanilla kubernetes on AWS #5571

Open kastork opened 8 years ago

kastork commented 8 years ago

There's some IRC chat history behind this. jrawlings (rawlingsj) suggested I raise this here.

This is vanilla kubernetes deployed on AWS with console-kubernetes 2.2.102 . I know your guides talk about Google Container Engine, so there might be tweaks to the k8 setup that are needed before this can work.

However please note that I've followed this approach before and gotten it to work without any contortions -- that effort took place back in Oct or Nov of 2015, and it used a console-kubernetes version that was close to (but not equal) 2.2.28 (AFAIK, teh 2.2.28 version was never actually in maven central.) It was also k8 1.1.0, not 1.1.3

My hope here is to get this type of thing working and then to contribute a guide for vanilla k8 on AWS. If fabric8 is "supposed to work" on vanilla kubernetes, then I think this use case should be considered a valid one.

My setup steps:

  1. On AWS, build a k8 cluster using the turn key get-kube procedure provided here: http://kubernetes.io/v1.1/docs/getting-started-guides/aws.html I set up a master and two minion nodes.
  2. Work through the guest book tutorial, be sure can get a simple example working properly. http://kubernetes.io/v1.1/examples/guestbook-go/README.html
  3. Guided by http://fabric8.io/guide/getStarted/gke.html, deploy the 2.2.102 version of console-kubernetes
  4. Wait a while. Observe the following
[ec2-user@ip-10-0-0-165 ~]$ kc get pods
NAME              READY     STATUS    RESTARTS   AGE
fabric8-6cnab     1/1       Running   0          47m
templates-76m5z   1/1       Running   0          47m

[ec2-user@ip-10-0-0-165 ~]$ kc get services
NAME         CLUSTER_IP     EXTERNAL_IP   PORT(S)   SELECTOR                                                   AGE
fabric8      10.0.5.191                   80/TCP    group=io.fabric8.apps,project=console,provider=fabric8     48m
kubernetes   10.0.0.1       <none>        443/TCP   <none>                                                     1h
templates    10.0.193.248   <none>        80/TCP    group=io.fabric8.apps,project=templates,provider=fabric8   48m
  1. Attempt to access fabric8 via the kubernetes api server.

https://52.11.245.16/api/v1/proxy/namespaces/default/services/fabric8 Observe browser console:

[hawtio-oauth] All oauth plugins have executed
libs.js:8686XHR finished loading: GET "https://52.11.245.16/api/v1/proxy/namespaces/default/services/fabric8/osconsole/config.js?_=1452192861010".Y.cors.e.crossDomain.send @ libs.js:8686Z.extend.ajax @ libs.js:8222Z.(anonymous function) @ libs.js:8367Z.extend.getScript @ libs.js:8354hawtioPluginLoader.registerPreBootstrapTask.task @ libs.js:152556a @ libs.js:39760
libs.js:8686GET https://52.11.245.16/oapi 404 (Not Found)Y.cors.e.crossDomain.send @ libs.js:8686Z.extend.ajax @ libs.js:8222r @ libs.js:145188hawtioPluginLoader.registerPreBootstrapTask.task @ libs.js:169179a @ libs.js:39760a @ libs.js:39752a @ libs.js:39752
libs.js:8686XHR finished loading: GET "https://52.11.245.16/oapi".Y.cors.e.crossDomain.send @ libs.js:8686Z.extend.ajax @ libs.js:8222r @ libs.js:145188hawtioPluginLoader.registerPreBootstrapTask.task @ libs.js:169179a @ libs.js:39760a @ libs.js:39752a @ libs.js:39752
libs.js:39375[kubernetes-watcher] Backend is a vanilla Kubernetes instance
libs.js:8686XHR finished loading: GET "https://52.11.245.16/api/v1/proxy/namespaces/default/services/fabric8/jolokia".Y.cors.e.crossDomain.send @ libs.js:8686Z.extend.ajax @ libs.js:8222o @ libs.js:240673(anonymous function) @ libs.js:240702a @ libs.js:39760a @ libs.js:39752
libs.js:8686XHR finished loading: GET "https://52.11.245.16/api/v1/namespaces".Y.cors.e.crossDomain.send @ libs.js:8686Z.extend.ajax @ libs.js:8222t.connect @ libs.js:152967i.connect @ libs.js:153077c @ libs.js:153487hawtioPluginLoader.registerPreBootstrapTask.task @ libs.js:169123a @ libs.js:39760
libs.js:8686GET https://52.11.245.16/jolokia 404 (Not Found)Y.cors.e.crossDomain.send @ libs.js:8686Z.extend.ajax @ libs.js:8222o @ libs.js:240673r @ libs.js:240668(anonymous function) @ libs.js:240688u @ libs.js:3155d.fireWith @ libs.js:3267n @ libs.js:8320(anonymous function) @ libs.js:8661
libs.js:8686XHR finished loading: GET "https://52.11.245.16/jolokia".Y.cors.e.crossDomain.send @ libs.js:8686Z.extend.ajax @ libs.js:8222o @ libs.js:240673r @ libs.js:240668(anonymous function) @ libs.js:240688u @ libs.js:3155d.fireWith @ libs.js:3267n @ libs.js:8320(anonymous function) @ libs.js:8661
libs.js:8686GET https://52.11.245.16/hawtio/jolokia 404 (Not Found)Y.cors.e.crossDomain.send @ libs.js:8686Z.extend.ajax @ libs.js:8222o @ libs.js:240673r @ libs.js:240668(anonymous function) @ libs.js:240698u @ libs.js:3155d.fireWith @ libs.js:3267n @ libs.js:8322(anonymous function) @ libs.js:8661
libs.js:8686XHR finished loading: GET "https://52.11.245.16/hawtio/jolokia".Y.cors.e.crossDomain.send @ libs.js:8686Z.extend.ajax @ libs.js:8222o @ libs.js:240673r @ libs.js:240668(anonymous function) @ libs.js:240698u @ libs.js:3155d.fireWith @ libs.js:3267n @ libs.js:8322(anonymous function) @ libs.js:8661
forgePlugin.ts:12 Listening on: /workspaces/:namespace/forge/createProject
libs.js:39375[fabric8-console] started, version:  undefined
libs.js:39375[fabric8-console] commit ID:  undefined
libs.js:8686XHR finished loading: GET "https://52.11.245.16/api/v1/namespaces/default/events".Y.cors.e.crossDomain.send @ libs.js:8686Z.extend.ajax @ libs.js:8222t.connect @ libs.js:152967i.connect @ libs.js:153077c @ libs.js:153487(anonymous function) @ libs.js:169245Qe @ libs.js:41392Ii @ libs.js:46046o.setNamespace @ libs.js:169239l.watch.KubernetesAPI.watch.success @ libs.js:169129e.emit @ libs.js:151987triggerChangedEvent._.debounce.trailing @ libs.js:152628r @ libs.js:47283
libs.js:8686XHR finished loading: GET "https://52.11.245.16/api/v1/nodes".Y.cors.e.crossDomain.send @ libs.js:8686Z.extend.ajax @ libs.js:8222t.connect @ libs.js:152967i.connect @ libs.js:153077c @ libs.js:153487(anonymous function) @ libs.js:169245Qe @ libs.js:41392Ii @ libs.js:46046o.setNamespace @ libs.js:169239l.watch.KubernetesAPI.watch.success @ libs.js:169129e.emit @ libs.js:151987triggerChangedEvent._.debounce.trailing @ libs.js:152628r @ libs.js:47283
libs.js:8686XHR finished loading: GET "https://52.11.245.16/api/v1/namespaces/default/pods".Y.cors.e.crossDomain.send @ libs.js:8686Z.extend.ajax @ libs.js:8222t.connect @ libs.js:152967i.connect @ libs.js:153077c @ libs.js:153487(anonymous function) @ libs.js:169245Qe @ libs.js:41392Ii @ libs.js:46046o.setNamespace @ libs.js:169239l.watch.KubernetesAPI.watch.success @ libs.js:169129e.emit @ libs.js:151987triggerChangedEvent._.debounce.trailing @ libs.js:152628r @ libs.js:47283
libs.js:8686XHR finished loading: GET "https://52.11.245.16/api/v1/namespaces/default/replicationcontrollers".Y.cors.e.crossDomain.send @ libs.js:8686Z.extend.ajax @ libs.js:8222t.connect @ libs.js:152967i.connect @ libs.js:153077c @ libs.js:153487(anonymous function) @ libs.js:169245Qe @ libs.js:41392Ii @ libs.js:46046o.setNamespace @ libs.js:169239l.watch.KubernetesAPI.watch.success @ libs.js:169129e.emit @ libs.js:151987triggerChangedEvent._.debounce.trailing @ libs.js:152628r @ libs.js:47283
libs.js:8686XHR finished loading: GET "https://52.11.245.16/api/v1/namespaces/default/secrets".Y.cors.e.crossDomain.send @ libs.js:8686Z.extend.ajax @ libs.js:8222t.connect @ libs.js:152967i.connect @ libs.js:153077c @ libs.js:153487(anonymous function) @ libs.js:169245Qe @ libs.js:41392Ii @ libs.js:46046o.setNamespace @ libs.js:169239l.watch.KubernetesAPI.watch.success @ libs.js:169129e.emit @ libs.js:151987triggerChangedEvent._.debounce.trailing @ libs.js:152628r @ libs.js:47283
libs.js:8686XHR finished loading: GET "https://52.11.245.16/api/v1/namespaces/default/services".Y.cors.e.crossDomain.send @ libs.js:8686Z.extend.ajax @ libs.js:8222t.connect @ libs.js:152967i.connect @ libs.js:153077c @ libs.js:153487(anonymous function) @ libs.js:169245Qe @ libs.js:41392Ii @ libs.js:46046o.setNamespace @ libs.js:169239l.watch.KubernetesAPI.watch.success @ libs.js:169129e.emit @ libs.js:151987triggerChangedEvent._.debounce.trailing @ libs.js:152628r @ libs.js:47283
libs.js:8686XHR finished loading: GET "https://52.11.245.16/api/v1/proxy/namespaces/default/services/fabric8/version.json?rev=1452192862392".Y.cors.e.crossDomain.send @ libs.js:8686Z.extend.ajax @ libs.js:8222(anonymous function) @ mainPlugin.ts:132a @ libs.js:39760
libs.js:177368XHR finished loading: GET "https://52.11.245.16/api/v1/proxy/namespaces/default/services/fabric8/resources/sounds/podlek_laser.ogg".p @ libs.js:177368d.load @ libs.js:176448d @ libs.js:176392(anonymous function) @ libs.js:223644r @ libs.js:16185(anonymous function) @ libs.js:16007o @ libs.js:12292He @ libs.js:16007Z.s @ libs.js:13421Z @ libs.js:13442(anonymous function) @ libs.js:40091
libs.js:177368XHR finished loading: GET "https://52.11.245.16/api/v1/proxy/namespaces/default/services/fabric8/resources/sounds/hit.ogg".p @ libs.js:177368d.load @ libs.js:176448d @ libs.js:176392(anonymous function) @ libs.js:223648r @ libs.js:16185(anonymous function) @ libs.js:16007o @ libs.js:12292He @ libs.js:16007Z.s @ libs.js:13421Z @ libs.js:13442(anonymous function) @ libs.js:40091
libs.js:177368XHR finished loading: GET "https://52.11.245.16/api/v1/proxy/namespaces/default/services/fabric8/resources/sounds/player_laser.ogg".p @ libs.js:177368d.load @ libs.js:176448d @ libs.js:176392(anonymous function) @ libs.js:223640r @ libs.js:16185(anonymous function) @ libs.js:16007o @ libs.js:12292He @ libs.js:16007Z.s @ libs.js:13421Z @ libs.js:13442(anonymous function) @ libs.js:40091
libs.js:177368XHR finished loading: GET "https://52.11.245.16/api/v1/proxy/namespaces/default/services/fabric8/resources/sounds/player_spawn.ogg".p @ libs.js:177368d.load @ libs.js:176448d @ libs.js:176392(anonymous function) @ libs.js:223660r @ libs.js:16185(anonymous function) @ libs.js:16007o @ libs.js:12292He @ libs.js:16007Z.s @ libs.js:13421Z @ libs.js:13442(anonymous function) @ libs.js:40091
libs.js:177368XHR finished loading: GET "https://52.11.245.16/api/v1/proxy/namespaces/default/services/fabric8/resources/sounds/player_explosion.ogg".p @ libs.js:177368d.load @ libs.js:176448d @ libs.js:176392(anonymous function) @ libs.js:223656r @ libs.js:16185(anonymous function) @ libs.js:16007o @ libs.js:12292He @ libs.js:16007Z.s @ libs.js:13421Z @ libs.js:13442(anonymous function) @ libs.js:40091
libs.js:177368XHR finished loading: GET "https://52.11.245.16/api/v1/proxy/namespaces/default/services/fabric8/resources/sounds/explosion.ogg".p @ libs.js:177368d.load @ libs.js:176448d @ libs.js:176392(anonymous function) @ libs.js:223652r @ libs.js:16185(anonymous function) @ libs.js:16007o @ libs.js:12292He @ libs.js:16007Z.s @ libs.js:13421Z @ libs.js:13442(anonymous function) @ libs.js:40091
libs.js:177368 XHR finished loading: GET "https://52.11.245.16/api/v1/proxy/namespaces/default/services/fabric8/resources/bg_music/background1.ogg".p @ libs.js:177368d.load @ libs.js:176448d @ libs.js:176392(anonymous function) @ libs.js:223666Qe @ libs.js:41392Ii @ libs.js:46046(anonymous function) @ libs.js:223665r @ libs.js:16185(anonymous function) @ libs.js:16007o @ libs.js:12292He @ libs.js:16007Z.s @ libs.js:13421Z @ libs.js:13442(anonymous function) @ libs.js:40091
libs.js:177368 XHR finished loading: GET "https://52.11.245.16/api/v1/proxy/namespaces/default/services/fabric8/resources/bg_music/background3.ogg".p @ libs.js:177368d.load @ libs.js:176448d @ libs.js:176392(anonymous function) @ libs.js:223666Qe @ libs.js:41392Ii @ libs.js:46046(anonymous function) @ libs.js:223665r @ libs.js:16185(anonymous function) @ libs.js:16007o @ libs.js:12292He @ libs.js:16007Z.s @ libs.js:13421Z @ libs.js:13442(anonymous function) @ libs.js:40091
libs.js:177368 XHR finished loading: GET "https://52.11.245.16/api/v1/proxy/namespaces/default/services/fabric8/resources/bg_music/background2.ogg".p @ libs.js:177368d.load @ libs.js:176448d @ libs.js:176392(anonymous function) @ libs.js:223666Qe @ libs.js:41392Ii @ libs.js:46046(anonymous function) @ libs.js:223665r @ libs.js:16185(anonymous function) @ libs.js:16007o @ libs.js:12292He @ libs.js:16007Z.s @ libs.js:13421Z @ libs.js:13442(anonymous function) @ libs.js:40091

Also observe network activity in browser:

namespaces?watch=true&access_token  GET 101 websocket   Other   0 B   Pending 
events?watch=true&access_token  GET 101 websocket   Other   0 B   Pending 
nodes?watch=true&access_token   GET 101 websocket   Other   0 B   Pending 
pods?watch=true&access_token    GET 101 websocket   Other   0 B   Pending 
replicationcontrollers?watch=true&access_token  GET 101 websocket   Other   0 B   Pending 
secrets?watch=true&access_token GET 101 websocket   Other   0 B   Pending 

The pending status never changes.

  1. Notice that the fabric8 console was given an AWS Elastic Load Balancer endpoint.
LoadBalancer Ingress    aca0007ceb56611e5b1d1029238c4679-1323732677.us-west-2.elb.amazonaws.com

Try accessing that way:

http://aca0007ceb56611e5b1d1029238c4679-1323732677.us-west-2.elb.amazonaws.com/workspaces?q=#%2Fdashboard%2Fservices%2Ffabric8

Web browser console:

WebSocket connection to 'ws://api/v1/namespaces?watch=true&access_token' failed: Error in connection establishment: net::ERR_NAME_NOT_RESOLVED
libs-f1798ed4.js:10 [k8s-objects] Using polling for kind:  projects
libs-f1798ed4.js:10 [k8s-objects] Using polling for kind:  imagestreamtags
libs-f1798ed4.js:59 Uncaught SyntaxError: Failed to construct 'WebSocket': The URL contains a fragment identifier ('/dashboard/services/fabric8/api/v1/namespaces/default/events'). Fragment identifiers are not allowed in WebSocket URLs.
libs-f1798ed4.js:59 Uncaught SyntaxError: Failed to construct 'WebSocket': The URL contains a fragment identifier ('/dashboard/services/fabric8/api/v1/nodes'). Fragment identifiers are not allowed in WebSocket URLs.
app-c22e1c22.js:1 Listening on: /workspaces/:namespace/forge/createProject
libs-f1798ed4.js:10 [fabric8-console] started, version:  1.0.73
rawlingsj commented 8 years ago

It looks like the console doesn't know about the kubernetes master. From the log above the line..

I'm pretty sure ws://api/v1/namespaces?watch=true&access_token should have the ip of kubernetes api server

@gashcrumb is there anything else that you can see that explains what might be going on?

kastork commented 8 years ago

Yes, that last attempt, using the load balancer endpoint is wrong in different ways than the attempt using the api server. I wouldn't think the load balancer would be configured at all if that wasn't the intended way to access it. This might be a point of difference between AWS and Google k8.

didn't provide a log of the ws connection when using the api server endpoint, but if memory serves, the websocket request is a full url in that case (with the ip address)

kastork commented 8 years ago

Spun it all up again. This time I removed the LoadBalancer from the config. Then attempted access via the api server:

https://52.11.245.16/api/v1/proxy/namespaces/default/services/fabric8/workspaces?q=

and the websocket requests look okay

wss://52.11.245.16/api/v1/namespaces?watch=true&access_token

Now I'm noticing something else -- the fabric8 service (both now and in the previous iteration where I used your json file unmodified), the IP field in the Kubernetes service dashboard shows blank for this service. It is also listed as being on port 80 only. Does it need to be on 443 as well?

kastork commented 8 years ago

I've managed to find the version that works on the AWS setup as I've described. It is console-kubernetes-2.2.23.1-kubernetes.json

canthefason commented 8 years ago

I also got a similar browser console log with you @kastork: as the solution, first set KUBERNETES_MASTER_URI environment variable for fabric8 replication controller, and added my domain as cors-allowed-origins flag variable for my kube-apiserver.

As the one remaning error, I am getting https://KUBERNETES_MASTER/oapi 404 (Not Found) error. Any ideas about that?

timbunce commented 8 years ago

@kastork, just a note to say that I'd also love to see a Getting Started Guide for fabric8 on Kubernetes at AWS.

lattwood commented 8 years ago

@kastork @timbunce seconded

jstrachan commented 8 years ago

@timbunce @kastork @lattwood great idea! Lets try do it...