Test OpenStack integration code

altvnk commented 9 years ago

Need to test OpenStack integration in k8s, at least find out does is able to create LB with external connectivity.

[x] Provide cloud.cfg via Ansible and add relevant options to kube-apiserver, kube-scheduler
[ ] Decide the way how to provide cred's for OpenStack access:
- read user ENV's with Ansible
- fill out somewhere in Ansible variables
[x] Check if k8s components can reach OpenStack API
[ ] Check service creation with external loadbalancer option enabled
[ ] Check OpenStack Cinder persistant volumes plugin

kenjones-cisco commented 9 years ago

When referring to the LB with external connectivity, are you referring to this? https://github.com/kubernetes/kubernetes/blob/master/docs/user-guide/services.md#type-loadbalancer

altvnk commented 9 years ago

Yes, but with small improvement: https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/openstack/openstack.go

kenjones-cisco commented 9 years ago

On the Openstack Access, I prefer ENVs so within the terraform files I remove all the variables with the openstack provider blocks so everything is picked up from ENV.

But if some have generic users or custom users with limited role then via variables works great. But not sure that will be the most common case.

kenjones-cisco commented 9 years ago

This PR could be of value here as well: https://github.com/kubernetes/kubernetes/pull/12589

altvnk commented 9 years ago

Started with manual config, but no luck. kube-apiserver fail immediately after start.

Aug 13 11:07:22 k8s-master-01 systemd[1]: Starting Kubernetes API Server...
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: panic: runtime error: invalid memory address or nil pointer dereference
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: [signal 0xb code=0x1 addr=0x0 pc=0x53443d]
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: goroutine 1 [running]:
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: k8s.io/kubernetes/pkg/cloudprovider/openstack.(*OpenStack).Instances(0x0, 0x0, 0x0, 0x6)
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: /builddir/build/BUILD/kubernetes-2bfa9a1f98147cfdc2e9f4cf50e2c430518d91eb/_output/local/go/src/k8s.io/kubernetes/pkg/cloudprovider/openstack/openstack.go:163 +0x8ed
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: k8s.io/kubernetes/cmd/kube-apiserver/app.(*APIServer).Run(0xc820275180, 0xc820190f00, 0x0, 0x10, 0x0, 0x0)
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: /builddir/build/BUILD/kubernetes-2bfa9a1f98147cfdc2e9f4cf50e2c430518d91eb/_output/local/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:376 +0x1477
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: main.main()
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: /builddir/build/BUILD/kubernetes-2bfa9a1f98147cfdc2e9f4cf50e2c430518d91eb/_output/local/go/src/k8s.io/kubernetes/cmd/kube-apiserver/apiserver.go:48 +0x171
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: goroutine 5 [chan receive]:
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: github.com/golang/glog.(*loggingT).flushDaemon(0x1f763e0)
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: /builddir/build/BUILD/kubernetes-2bfa9a1f98147cfdc2e9f4cf50e2c430518d91eb/Godeps/_workspace/src/github.com/golang/glog/glog.go:879 +0x67
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: created by github.com/golang/glog.init.1
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: /builddir/build/BUILD/kubernetes-2bfa9a1f98147cfdc2e9f4cf50e2c430518d91eb/Godeps/_workspace/src/github.com/golang/glog/glog.go:410 +0x297
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: goroutine 12 [syscall]:
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: os/signal.loop()
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: /usr/lib/golang/src/os/signal/signal_unix.go:22 +0x18
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: created by os/signal.init.1
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: /usr/lib/golang/src/os/signal/signal_unix.go:28 +0x37
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: goroutine 18 [runnable]:
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: github.com/prometheus/client_golang/prometheus.(*registry).Register.func1(0x7f7f9f186990, 0xc8200e1ce0, 0xc8200e40c0)
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: /builddir/build/BUILD/kubernetes-2bfa9a1f98147cfdc2e9f4cf50e2c430518d91eb/Godeps/_workspace/src/github.com/prometheus/client_golang/prometheus/registry.go:216
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: created by github.com/prometheus/client_golang/prometheus.(*registry).Register
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: /builddir/build/BUILD/kubernetes-2bfa9a1f98147cfdc2e9f4cf50e2c430518d91eb/Godeps/_workspace/src/github.com/prometheus/client_golang/prometheus/registry.go:219 +0xb2
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: goroutine 40 [sleep]:
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: time.Sleep(0x12a05f200)
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: /usr/lib/golang/src/runtime/time.go:59 +0xf9
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: k8s.io/kubernetes/pkg/util.Until(0x1722340, 0x12a05f200, 0x0)
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: /builddir/build/BUILD/kubernetes-2bfa9a1f98147cfdc2e9f4cf50e2c430518d91eb/_output/local/go/src/k8s.io/kubernetes/pkg/util/util.go:113 +0x61
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: k8s.io/kubernetes/pkg/util.Forever(0x1722340, 0x12a05f200)
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: /builddir/build/BUILD/kubernetes-2bfa9a1f98147cfdc2e9f4cf50e2c430518d91eb/_output/local/go/src/k8s.io/kubernetes/pkg/util/util.go:93 +0x34
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: created by k8s.io/kubernetes/pkg/util.InitLogs
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: /builddir/build/BUILD/kubernetes-2bfa9a1f98147cfdc2e9f4cf50e2c430518d91eb/_output/local/go/src/k8s.io/kubernetes/pkg/util/logs.go:50 +0xae
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: goroutine 41 [runnable]:
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: k8s.io/kubernetes/pkg/util.Forever(0xc8202b7aa0, 0x3b9aca00)
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: /builddir/build/BUILD/kubernetes-2bfa9a1f98147cfdc2e9f4cf50e2c430518d91eb/_output/local/go/src/k8s.io/kubernetes/pkg/util/util.go:92
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: created by k8s.io/kubernetes/pkg/client/cache.(*Reflector).Run
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: /builddir/build/BUILD/kubernetes-2bfa9a1f98147cfdc2e9f4cf50e2c430518d91eb/_output/local/go/src/k8s.io/kubernetes/pkg/client/cache/reflector.go:138 +0x78
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: goroutine 42 [runnable]:
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: k8s.io/kubernetes/pkg/util.Forever(0xc8202b7af0, 0x3b9aca00)
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: /builddir/build/BUILD/kubernetes-2bfa9a1f98147cfdc2e9f4cf50e2c430518d91eb/_output/local/go/src/k8s.io/kubernetes/pkg/util/util.go:92
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: created by k8s.io/kubernetes/pkg/client/cache.(*Reflector).Run
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: /builddir/build/BUILD/kubernetes-2bfa9a1f98147cfdc2e9f4cf50e2c430518d91eb/_output/local/go/src/k8s.io/kubernetes/pkg/client/cache/reflector.go:138 +0x78
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: goroutine 43 [chan receive]:
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: github.com/prometheus/client_golang/prometheus.(*registry).Register(0xc82001c8a0, 0x7f7f9f186990, 0xc8200e1ce0, 0x0, 0x0, 0x0, 0x0)
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: /builddir/build/BUILD/kubernetes-2bfa9a1f98147cfdc2e9f4cf50e2c430518d91eb/Godeps/_workspace/src/github.com/prometheus/client_golang/prometheus/registry.go:229 +0x1c4
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: github.com/prometheus/client_golang/prometheus.Register(0x7f7f9f186990, 0xc8200e1ce0, 0x0, 0x0)
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: /builddir/build/BUILD/kubernetes-2bfa9a1f98147cfdc2e9f4cf50e2c430518d91eb/Godeps/_workspace/src/github.com/prometheus/client_golang/prometheus/registry.go:114 +0x43
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: github.com/prometheus/client_golang/prometheus.MustRegister(0x7f7f9f186990, 0xc8200e1ce0)
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: /builddir/build/BUILD/kubernetes-2bfa9a1f98147cfdc2e9f4cf50e2c430518d91eb/Godeps/_workspace/src/github.com/prometheus/client_golang/prometheus/registry.go:121 +0x2b
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: k8s.io/kubernetes/pkg/client/metrics.Register.func1()
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: /builddir/build/BUILD/kubernetes-2bfa9a1f98147cfdc2e9f4cf50e2c430518d91eb/_output/local/go/src/k8s.io/kubernetes/pkg/client/metrics/metrics.go:46 +0x40
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: sync.(*Once).Do(0x1f9f850, 0x1724c90)
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: /usr/lib/golang/src/sync/once.go:44 +0xe4
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: k8s.io/kubernetes/pkg/client/metrics.Register()
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: /builddir/build/BUILD/kubernetes-2bfa9a1f98147cfdc2e9f4cf50e2c430518d91eb/_output/local/go/src/k8s.io/kubernetes/pkg/client/metrics/metrics.go:47 +0x2f
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: k8s.io/kubernetes/pkg/client.NewRequest(0x7f7f9f1873d0, 0xc8202b43c0, 0x14016a8, 0x3, 0xc8202b2a80, 0x140b020, 0x2, 0x7f7f9f1810e0, 0xc82010ba00, 0x7f7f9f13c050)
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: /builddir/build/BUILD/kubernetes-2bfa9a1f98147cfdc2e9f4cf50e2c430518d91eb/_output/local/go/src/k8s.io/kubernetes/pkg/client/request.go:104 +0x1c
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: k8s.io/kubernetes/pkg/client.(*RESTClient).Verb(0xc82014d900, 0x14016a8, 0x3, 0xc8200e4060)
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: /builddir/build/BUILD/kubernetes-2bfa9a1f98147cfdc2e9f4cf50e2c430518d91eb/_output/local/go/src/k8s.io/kubernetes/pkg/client/restclient.go:94 +0xb4
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: k8s.io/kubernetes/pkg/client.(*RESTClient).Get(0xc82014d900, 0xc8200e4060)
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: /builddir/build/BUILD/kubernetes-2bfa9a1f98147cfdc2e9f4cf50e2c430518d91eb/_output/local/go/src/k8s.io/kubernetes/pkg/client/restclient.go:114 +0x36
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: k8s.io/kubernetes/pkg/client.(*limitRanges).List(0xc8200e8020, 0x7f7f9f13c050, 0xc8200e8040, 0xc8200e4060, 0x0, 0x0)
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: /builddir/build/BUILD/kubernetes-2bfa9a1f98147cfdc2e9f4cf50e2c430518d91eb/_output/local/go/src/k8s.io/kubernetes/pkg/client/limit_ranges.go:60 +0x7f
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: k8s.io/kubernetes/plugin/pkg/admission/limitranger.NewLimitRanger.func1(0x0, 0x0, 0x0, 0x0)
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: /builddir/build/BUILD/kubernetes-2bfa9a1f98147cfdc2e9f4cf50e2c430518d91eb/_output/local/go/src/k8s.io/kubernetes/plugin/pkg/admission/limitranger/admission.go:96 +0x143
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: k8s.io/kubernetes/pkg/client/cache.(*ListWatch).List(0xc8202b7b00, 0x0, 0x0, 0x0, 0x0)
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: /builddir/build/BUILD/kubernetes-2bfa9a1f98147cfdc2e9f4cf50e2c430518d91eb/_output/local/go/src/k8s.io/kubernetes/pkg/client/cache/listwatch.go:63 +0x39
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: k8s.io/kubernetes/pkg/client/cache.(*Reflector).ListAndWatch(0xc8202b2d80, 0xc82001cc60)
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: /builddir/build/BUILD/kubernetes-2bfa9a1f98147cfdc2e9f4cf50e2c430518d91eb/_output/local/go/src/k8s.io/kubernetes/pkg/client/cache/reflector.go:178 +0xab
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: k8s.io/kubernetes/pkg/client/cache.(*Reflector).Run.func1()
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: /builddir/build/BUILD/kubernetes-2bfa9a1f98147cfdc2e9f4cf50e2c430518d91eb/_output/local/go/src/k8s.io/kubernetes/pkg/client/cache/reflector.go:138 +0x2c
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: k8s.io/kubernetes/pkg/util.Until.func1(0xc8202b7b30)
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: /builddir/build/BUILD/kubernetes-2bfa9a1f98147cfdc2e9f4cf50e2c430518d91eb/_output/local/go/src/k8s.io/kubernetes/pkg/util/util.go:111 +0x3a
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: k8s.io/kubernetes/pkg/util.Until(0xc8202b7b30, 0x3b9aca00, 0x0)
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: /builddir/build/BUILD/kubernetes-2bfa9a1f98147cfdc2e9f4cf50e2c430518d91eb/_output/local/go/src/k8s.io/kubernetes/pkg/util/util.go:112 +0x53
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: k8s.io/kubernetes/pkg/util.Forever(0xc8202b7b30, 0x3b9aca00)
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: /builddir/build/BUILD/kubernetes-2bfa9a1f98147cfdc2e9f4cf50e2c430518d91eb/_output/local/go/src/k8s.io/kubernetes/pkg/util/util.go:93 +0x34
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: created by k8s.io/kubernetes/pkg/client/cache.(*Reflector).Run
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: /builddir/build/BUILD/kubernetes-2bfa9a1f98147cfdc2e9f4cf50e2c430518d91eb/_output/local/go/src/k8s.io/kubernetes/pkg/client/cache/reflector.go:138 +0x78
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: goroutine 44 [runnable]:
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: k8s.io/kubernetes/pkg/util.Until(0xc8202cb1c0, 0x3b9aca00, 0xc82016db00)
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: /builddir/build/BUILD/kubernetes-2bfa9a1f98147cfdc2e9f4cf50e2c430518d91eb/_output/local/go/src/k8s.io/kubernetes/pkg/util/util.go:102
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: created by k8s.io/kubernetes/pkg/client/cache.(*Reflector).RunUntil
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: /builddir/build/BUILD/kubernetes-2bfa9a1f98147cfdc2e9f4cf50e2c430518d91eb/_output/local/go/src/k8s.io/kubernetes/pkg/client/cache/reflector.go:144 +0x9a
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: goroutine 45 [runnable]:
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: k8s.io/kubernetes/pkg/util.Until(0xc8202cb1e0, 0x3b9aca00, 0xc82016db00)
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: /builddir/build/BUILD/kubernetes-2bfa9a1f98147cfdc2e9f4cf50e2c430518d91eb/_output/local/go/src/k8s.io/kubernetes/pkg/util/util.go:102
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: created by k8s.io/kubernetes/pkg/client/cache.(*Reflector).RunUntil
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: /builddir/build/BUILD/kubernetes-2bfa9a1f98147cfdc2e9f4cf50e2c430518d91eb/_output/local/go/src/k8s.io/kubernetes/pkg/client/cache/reflector.go:144 +0x9a
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: goroutine 46 [runnable]:
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: k8s.io/kubernetes/pkg/util.Forever(0xc8202b7be0, 0x3b9aca00)
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: /builddir/build/BUILD/kubernetes-2bfa9a1f98147cfdc2e9f4cf50e2c430518d91eb/_output/local/go/src/k8s.io/kubernetes/pkg/util/util.go:92
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: created by k8s.io/kubernetes/pkg/client/cache.(*Reflector).Run
Aug 13 11:07:22 k8s-master-01 kube-apiserver[6878]: /builddir/build/BUILD/kubernetes-2bfa9a1f98147cfdc2e9f4cf50e2c430518d91eb/_output/local/go/src/k8s.io/kubernetes/pkg/client/cache/reflector.go:138 +0x78
Aug 13 11:07:22 k8s-master-01 systemd[1]: kube-apiserver.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Aug 13 11:07:22 k8s-master-01 systemd[1]: Failed to start Kubernetes API Server.

altvnk commented 9 years ago

Finally i get working integration, k8s can login into OpenStack API, trys to create LB and fails. Continuing my investigation...

altvnk commented 9 years ago

I'm blocked here because gophercloud Golang library used in OpenStack integration doesn't supports compute API 1.x, only 2.0 and higher. Our OpenStack uses old (deprecated) compute API 1.1.

CiscoCloud / kubernetes-ansible

Test OpenStack integration code #36