search

kubernetes / cloud-provider-openstack

Apache License 2.0
619 stars 610 forks source link

Hardcoded wrong lb-provider name in octavia-ingress-controller #610

Closed horakmar closed 5 years ago

horakmar commented 5 years ago

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug

What happened: When trying to deploy and test octavia-ingrass-controller it fails to create loadbalancer with following error:

E0510 14:12:04.621929       1 controller.go:449] failed to create openstack resources for ingress default/test-octavia-ingress: error creating loadbalancer {k8s_ing_kubeo1_default_test-octavia-ingress Kubernetes ingress test-octavia-ingress in namespace default from cluster kubeo1  a171b919-2572-4ca0-ab9c-ddad6253305f    <nil>  octavia []}: Bad request with: [POST https://172.28.115.10:9876/v2.0/lbaas/loadbalancers], error message: {"faultcode": "Client", "faultstring": "Provider 'octavia' is not enabled.", "debuginfo": null}

What you expected to happen: Create LB for octavia-ingress-controller. Provider name should be 'amphora'.

How to reproduce it (as minimally and precisely as possible): Just follow guide: https://github.com/kubernetes/cloud-provider-openstack/raw/master/docs/using-octavia-ingress-controller.md

Anything else we need to know?: I found hardcoded name here: https://github.com/kubernetes/cloud-provider-openstack/blob/20b7744ee505a1da05550a866051bf5323e4feba/pkg/ingress/controller/openstack/octavia.go#L319 The name should be configurable.

Environment:

lingxiankong commented 5 years ago

Hi, what's the provider configured in your Octavia config file?

horakmar commented 5 years ago

Hi, in octavia config I found only this line related to provider:

[api_settings]
enabled_provider_drivers = amphora:Octavia Amphora driver

(I am not an admin of Openstack)

Ingress controller works, when I change hardcoded string "octavia" into "amphora". But I suppose it should be configurable via configmap.

lingxiankong commented 5 years ago

hi @horakmar, could you please help to test if https://github.com/kubernetes/cloud-provider-openstack/pull/614 works for you?

horakmar commented 5 years ago

Hi @lingxiankong. It doesn't work. LoadBalancer is created, but Floating IP is not assigned and LB stays in Pending Update state. Here is the log of octavia-ingress-controller pod:

W0515 14:20:55.719207       1 client_config.go:549] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
time="2019-05-15T14:20:55Z" level=info msg="Using config file" file=/etc/config/octavia-ingress-controller-config.yaml
time="2019-05-15T14:20:56Z" level=info msg="ingress controller synced and ready"
I0515 14:29:50.458575       1 event.go:209] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"test-octavia-ingress", UID:"e5cac877-771d-11e9-b5bd-fa163ea0fb68", APIVersion:"extensions/v1beta1", ResourceVersion:"1036663", FieldPath:""}): type: 'Normal' reason: 'Creating' Ingress default/test-octavia-ingress
time="2019-05-15T14:29:50Z" level=info msg="ingress created, will create openstack resources" ingress=default/test-octavia-ingress
time="2019-05-15T14:29:52Z" level=info msg="loadbalancer created" ID=c3a9bdef-22d8-4766-9d21-c1a196e29ef2 name=kube_ingress_kubeo1_default_test-octavia-ingress
time="2019-05-15T14:30:30Z" level=info msg="listener created" lb=c3a9bdef-22d8-4766-9d21-c1a196e29ef2 listenerName=kube_ingress_kubeo1_default_test-octavia-ingress
time="2019-05-15T14:30:33Z" level=info msg="pool created" lb=c3a9bdef-22d8-4766-9d21-c1a196e29ef2 listenerID= pooID=843e25cb-d0da-4eb3-90de-407117518281 poolName=38df6e716b70f28ecfa4508c7e5c21dc18b1b8cc97c610ab4111c3237507fbfb
E0515 14:42:58.406703       1 controller.go:449] failed to create openstack resources for ingress default/test-octavia-ingress: error waiting for loadbalancer c3a9bdef-22d8-4766-9d21-c1a196e29ef2 to be active: loadbalancer failed to go into ACTIVE provisioning status within alloted time
I0515 14:42:58.407505       1 event.go:209] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"test-octavia-ingress", UID:"e5cac877-771d-11e9-b5bd-fa163ea0fb68", APIVersion:"extensions/v1beta1", ResourceVersion:"1036663", FieldPath:""}): type: 'Warning' reason: 'Failed' Failed to create openstack resources for ingress default/test-octavia-ingress: error waiting for loadbalancer c3a9bdef-22d8-4766-9d21-c1a196e29ef2 to be active: loadbalancer failed to go into ACTIVE provisioning status within alloted time

Again, with hardcoded provider 'amphora' it works. Updated, see below...

horakmar commented 5 years ago

I am sorry, I did new fresh build of octavia-ingress-controller image and it now works as expected:

time="2019-05-15T15:26:01Z" level=info msg="Using config file" file=/etc/config/octavia-ingress-controller-config.yaml
W0515 15:26:01.796178       1 client_config.go:549] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
time="2019-05-15T15:26:02Z" level=info msg="ingress controller synced and ready"
I0515 15:26:55.575903       1 event.go:209] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"test-oing", UID:"df520a26-7725-11e9-b5bd-fa163ea0fb68", APIVersion:"extensions/v1beta1", ResourceVersion:"1043336", FieldPath:""}): type: 'Normal' reason: 'Creating' Ingress default/test-oing
time="2019-05-15T15:26:55Z" level=info msg="ingress created, will create openstack resources" ingress=default/test-oing
time="2019-05-15T15:26:57Z" level=info msg="loadbalancer created" ID=56c5f8d6-385b-4b5d-9294-f077a438440d name=kube_ingress_kubeo1_default_test-oing
time="2019-05-15T15:27:37Z" level=info msg="listener created" lb=56c5f8d6-385b-4b5d-9294-f077a438440d listenerName=kube_ingress_kubeo1_default_test-oing
time="2019-05-15T15:27:41Z" level=info msg="pool created" lb=56c5f8d6-385b-4b5d-9294-f077a438440d listenerID= pooID=44a9df2b-2a7a-4ac5-814f-c0e6c9b0c091 poolName=38df6e716b70f28ecfa4508c7e5c21dc18b1b8cc97c610ab4111c3237507fbfb
time="2019-05-15T15:27:48Z" level=info msg="pool members updated" lb=56c5f8d6-385b-4b5d-9294-f077a438440d listenerID= pooID=44a9df2b-2a7a-4ac5-814f-c0e6c9b0c091 poolName=38df6e716b70f28ecfa4508c7e5c21dc18b1b8cc97c610ab4111c3237507fbfb
time="2019-05-15T15:27:51Z" level=info msg="policy created" lb=56c5f8d6-385b-4b5d-9294-f077a438440d listenerID=cad82aa4-2266-4d26-84d9-bbb147f201c8 policyID=2dee3547-5e38-4653-ac55-4c0e431b3c4d
time="2019-05-15T15:27:54Z" level=info msg="policy rule created" host=test.cl.corp listenerID=cad82aa4-2266-4d26-84d9-bbb147f201c8 policyID=2dee3547-5e38-4653-ac55-4c0e431b3c4d type=HOST_NAME
time="2019-05-15T15:27:58Z" level=info msg="policy rule created" listenerID=cad82aa4-2266-4d26-84d9-bbb147f201c8 path=/ping policyID=2dee3547-5e38-4653-ac55-4c0e431b3c4d type=PATH
time="2019-05-15T15:27:58Z" level=info msg="creating floating IP" ingress=default/test-oing
time="2019-05-15T15:28:00Z" level=info msg="floating IP created" fip=172.28.64.169 ingress=default/test-oing
I0515 15:28:00.174811       1 event.go:209] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"test-oing", UID:"df520a26-7725-11e9-b5bd-fa163ea0fb68", APIVersion:"extensions/v1beta1", ResourceVersion:"1043336", FieldPath:""}): type: 'Normal' reason: 'Updated' Successfully associated IP address 172.28.64.169 to ingress default/test-oing
time="2019-05-15T15:28:00Z" level=info msg="openstack resources for ingress created" ingress=default/test-oing lbID=56c5f8d6-385b-4b5d-9294-f077a438440d
I0515 15:28:00.377486       1 event.go:209] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"test-oing", UID:"df520a26-7725-11e9-b5bd-fa163ea0fb68", APIVersion:"extensions/v1beta1", ResourceVersion:"1043336", FieldPath:""}): type: 'Normal' reason: 'Created' Ingress default/test-oing
I0515 16:27:36.132594       1 event.go:209] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"test-oing", UID:"df520a26-7725-11e9-b5bd-fa163ea0fb68", APIVersion:"extensions/v1beta1", ResourceVersion:"1050457", FieldPath:""}): type: 'Normal' reason: 'Deleting' Ingress default/test-oing
time="2019-05-15T16:27:36Z" level=info msg="ingress has been deleted, will delete openstack resources" ingress=default/test-oing
time="2019-05-15T16:27:36Z" level=info msg="deleting floating IP" ingress=default/test-oing
time="2019-05-15T16:27:37Z" level=info msg="floating IP deleted" ingress=default/test-oing
time="2019-05-15T16:27:37Z" level=info msg="loadbalancer deleted" lbID=56c5f8d6-385b-4b5d-9294-f077a438440d
I0515 16:27:37.261021       1 event.go:209] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"test-oing", UID:"df520a26-7725-11e9-b5bd-fa163ea0fb68", APIVersion:"extensions/v1beta1", ResourceVersion:"1050457", FieldPath:""}): type: 'Normal' reason: 'Deleted' Ingress default/test-oing
lingxiankong commented 5 years ago

@horakmar nice, thanks for the feedback! I will try to push to merge that PR asap so you can get a daily built official image.

horakmar commented 5 years ago

Perfect, thank you for the patch!

ReggieCarey commented 2 years ago

Several years later.... I'm seeing the same inability to assign a floating IP address.

INFO   [2022-04-11T22:48:30Z] l7 rules created                              ingress=default/test-octavia-ingress lbID=4c5ed8c2-4c64-4939-9aca-1a1203840e14 listenerID=57c0dc95-b060-42c0-a8f2-d6cef028c6a2 policyID=d27d30ad-c5c0-4058-b674-cf944f15d92e
INFO   [2022-04-11T22:48:30Z] ensuring security group rules                 ingress=default/test-octavia-ingress lbID=4c5ed8c2-4c64-4939-9aca-1a1203840e14 sgID=dde17573-d287-4128-a906-8ea4cc3562db
INFO   [2022-04-11T22:48:32Z] ensured security group rules                  ingress=default/test-octavia-ingress lbID=4c5ed8c2-4c64-4939-9aca-1a1203840e14 sgID=dde17573-d287-4128-a906-8ea4cc3562db
INFO   [2022-04-11T22:48:32Z] creating floating IP                          ingress=default/test-octavia-ingress lbID=4c5ed8c2-4c64-4939-9aca-1a1203840e14
E0411 22:48:32.778774       1 controller.go:504] failed to create openstack resources for ingress default/test-octavia-ingress: failed to create floating IP: Resource not found
I0411 22:48:32.778918       1 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"test-octavia-ingress", UID:"cad4e457-ac93-4119-86ce-f9df56795b66", APIVersion:"networking.k8s.io/v1", ResourceVersion:"8279384", FieldPath:""}): type: 'Warning' reason: 'Failed' Failed to create openstack resources for ingress default/test-octavia-ingress: failed to create floating IP: Resource not found

I'm unsure how to resolve. I've followed the same set of instructions, adding in "provider: amphora" to get past the same initial problem.

ReggieCarey commented 2 years ago

Manually assigning an floating IP to the loadbalancer in the horizon dashboard, does not result in that IP address being configured for the ingress

$ kubectl get ing
NAME                   CLASS    HOSTS         ADDRESS   PORTS   AGE
test-octavia-ingress   <none>   foo.bar.com             80      25m

Even though I can see the attached floating ip in the horizon dashboard Screen Shot 2022-04-11 at 19 14 59

I should point out that when I kill the pod octavia-ingress-controller-0 and let the stateful set regenerate it, it does recognize the IP address of the load balancer. I don't think the problem is resolved if I have to manually kill the controller and manually add a floating ip address to the load balancer

lingxiankong commented 2 years ago

failed to create openstack resources for ingress default/test-octavia-ingress: failed to create floating IP: Resource not found

Please check the config floating-network-id, and if there are still available IPs in that network.

ReggieCarey commented 2 years ago

Thanks for the pointer. In the end the combination that works is to set subnet_id to the internal subnet and the floating_network_id to the external network.

This allows the LB to be created in the internal network but allocate FIP on the external network.

Also it appears that changes to the configmap are not picked up until the octavia-ingress-controller is reloaded/restarted. It might me a nice enhancement to watch for changes in the configmap.

I've also had to add in a volumeMount to mount an updated ca-certificates.crt that includes the ca of the keystone endpoint. That might be handy to incorporate into the configmap or point to existing configs that would normally contain the keystone ca.