[octavia-ingress-controller] 503 Service Unavailable - Error when accessing loadbalancer Floating IP

gpugal commented 3 years ago

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug

What happened: I followed the instructions at: https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/octavia-ingress-controller/using-octavia-ingress-controller.md

(I am running a small k8s cluster on top of Openstack, which is Victoria, and has Octavia) Loadbalancer is created and floating IP gets assigned, but when accessing the IP it returns 503 Service Unavailable

ubuntu@ubuntu-1:~$ kubectl -n kube-system logs octavia-ingress-controller-0 time="2021-03-02T15:23:32Z" level=info msg="Using config file" file=/etc/config/octavia-ingress-controller-config.yaml W0302 15:23:32.138661 1 client_config.go:614] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. I0302 15:23:32.187049 1 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"test-octavia-ingress", UID:"0700306e-b4b1-467c-bcd4-a1426821bcae", APIVersion:"networking.k8s.io/v1", ResourceVersion:"9414", FieldPath:""}): type: 'Normal' reason: 'Creating' Ingress default/test-octavia-ingress time="2021-03-02T15:23:32Z" level=info msg="ingress controller synced and ready" time="2021-03-02T15:23:32Z" level=info msg="ingress created, will create openstack resources" ingress=default/test-octavia-ingress time="2021-03-02T15:23:33Z" level=info msg="creating loadbalancer" ID=4d69cc29-8b30-4c4e-8fa6-ac9d58768525 name=kube_ingress_k8sing1_default_test-octavia-ingress time="2021-03-02T15:24:22Z" level=info msg="listener created" lb=4d69cc29-8b30-4c4e-8fa6-ac9d58768525 listenerName=kube_ingress_k8sing1_default_test-octavia-ingress time="2021-03-02T15:24:25Z" level=info msg="pool created" lb=4d69cc29-8b30-4c4e-8fa6-ac9d58768525 listenerID= pooID=cb8545de-e127-4c7e-8da5-4c9a26e012fd poolName=8b579a9997fa22d64b274eeb9714eb26f12412dce7931685347b1e42b4342a1c time="2021-03-02T15:24:31Z" level=info msg="pool members updated" lb=4d69cc29-8b30-4c4e-8fa6-ac9d58768525 listenerID= pooID=cb8545de-e127-4c7e-8da5-4c9a26e012fd poolName=8b579a9997fa22d64b274eeb9714eb26f12412dce7931685347b1e42b4342a1c time="2021-03-02T15:24:34Z" level=info msg="policy created" lb=4d69cc29-8b30-4c4e-8fa6-ac9d58768525 listenerID=26a713cd-69a4-4b32-bc06-2eb3b4d89a26 policyID=2981b268-0bf7-4c1a-a842-5f86742e1cb9 time="2021-03-02T15:24:38Z" level=info msg="policy rule created" listenerID=26a713cd-69a4-4b32-bc06-2eb3b4d89a26 path=/ping policyID=2981b268-0bf7-4c1a-a842-5f86742e1cb9 type=PATH time="2021-03-02T15:24:38Z" level=info msg="creating floating IP" ingress=default/test-octavia-ingress time="2021-03-02T15:24:40Z" level=info msg="floating IP created" fip=10.221.43.238 ingress=default/test-octavia-ingress I0302 15:24:40.333566 1 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"test-octavia-ingress", UID:"0700306e-b4b1-467c-bcd4-a1426821bcae", APIVersion:"networking.k8s.io/v1", ResourceVersion:"9414", FieldPath:""}): type: 'Normal' reason: 'Updated' Successfully associated IP address 10.221.43.238 to ingress default/test-octavia-ingress time="2021-03-02T15:24:40Z" level=info msg="openstack resources for ingress created" ingress=default/test-octavia-ingress lbID=4d69cc29-8b30-4c4e-8fa6-ac9d58768525 I0302 15:24:40.486854 1 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"test-octavia-ingress", UID:"0700306e-b4b1-467c-bcd4-a1426821bcae", APIVersion:"networking.k8s.io/v1", ResourceVersion:"9414", FieldPath:""}): type: 'Normal' reason: 'Created' Ingress default/test-octavia-ingress What you expected to happen: Correct HTTP response should be returned when accessing the assigned floating IP.

How to reproduce it:

Install multinode Openstack using devstack,
Create 3 nova instance of bionic-server-cloudimg-amd64-disk.img.
Install kuberenetes on nova VMs - (https://phoenixnap.com/kb/install-kubernetes-on-ubuntu)
Then follow https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/using-octavia-ingress-controller.md

Anything else we need to know?:

Openstack Setup : 3 Ubuntu (Xenial) Machines with single NIC for network connection.

Environment:

openstack-cloud-controller-manager version: OpenStack Victoria
OS (e.g. from /etc/os-release): Ubuntu (Xenial)
Kernel (e.g. uname -a): 4.4.0-131-generic
Install tools: k8s was installed on Nova VMs manually.

lingxiankong commented 3 years ago

Please provide your octavia-ingress-controller config.
You can try to access the fixed VIP of the load balancer from within the subnet of the VIP, see if that works.
In the future, please paste the command and output using correct markdown format

gpugal commented 3 years ago

Thanks for your reply, Please find the requested details.

octavia-ingress-controller config.

---
kind: ConfigMap
apiVersion: v1
metadata:
name: octavia-ingress-controller-config
namespace: kube-system
data:
config: |
cluster-name: k8sing1
openstack:
  auth-url: http://<IP>/identity
  domain-name: Default
  username: demo
  # user-id:
  password: labstack
  project-id: dcf80a9ea2774625a530ea68d951ebd2
  region: RegionOne
octavia:
  subnet-id: 2098b61f-c4e1-43a7-a567-2ed7ceb5c08b
  floating-network-id: 8a6b772f-12c9-46d7-8483-ff530767b4b3

I have hardcoded config values, instead of allowing it to get injected.

You can try to access the fixed VIP of the load balancer from within the subnet of the VIP, see if that works.

I tried to access the VIP of the load balancer, I'm getting the same error.

lingxiankong commented 3 years ago

I tried to access the VIP of the load balancer, I'm getting the same error.

Check if you can access the service via NodePort (on the load balancer member IP). The ingress-controller has already successfully set up the load balancer, but seems there is network firewall blocks the traffic.

gpugal commented 3 years ago

Thanks for your reply, HTTP is working now, The fix was I added the NodePort in the security groups. But I got another issue with the HTTPS.

Secret(s) retrieval attempt not allowed - please review your user/project privileges

Logs

INFO   [2021-03-08T07:09:48Z] floating IP created                           fip=10.221.43.237 ingress=default/ingress-srv lbID=802d04ca-ea71-44ed-98bf-9ff2b44c1839
I0308 07:09:49.024514       1 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"ingress-srv", UID:"bc83a908-beba-4d3c-9928-4b3d062180e7", APIVersion:"networking.k8s.io/v1", ResourceVersion:"26931", FieldPath:""}): type: 'Normal' reason: 'Updated' Successfully associated IP address 10.221.43.237 to ingress default/ingress-srv
INFO   [2021-03-08T07:09:49Z] openstack resources for ingress created       ingress=default/ingress-srv lbID=802d04ca-ea71-44ed-98bf-9ff2b44c1839
I0308 07:09:49.216261       1 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"ingress-srv", UID:"bc83a908-beba-4d3c-9928-4b3d062180e7", APIVersion:"networking.k8s.io/v1", ResourceVersion:"26931", FieldPath:""}): type: 'Normal' reason: 'Created' Ingress default/ingress-srv
I0308 07:32:01.045540       1 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"ingress-srv", UID:"bc83a908-beba-4d3c-9928-4b3d062180e7", APIVersion:"networking.k8s.io/v1", ResourceVersion:"28765", FieldPath:""}): type: 'Normal' reason: 'Updating' Ingress default/ingress-srv
INFO   [2021-03-08T07:32:01Z] updating ingress                              ingress=default/ingress-srv
E0308 07:32:01.283858       1 controller.go:513] failed to update openstack resources for ingress default/ingress-srv: failed to create Barbican secret: Request forbidden: [GET http://10.221.43.28/key-manager/v1/secrets?name=kube_ingress_k8slocal_default_ingress-srv_tls-secret], error message: {"code": 403, "title": "Forbidden", "description": Secret(s) retrieval attempt not allowed - please review your user/project privileges"}
I0308 07:32:01.284105       1 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"ingress-srv", UID:"bc83a908-beba-4d3c-9928-4b3d062180e7", APIVersion:"networking.k8s.io/v1", ResourceVersion:"28765", FieldPath:""}): type: 'Warning' reason: 'Failed' Failed to update openstack resources for ingress default/ingress-srv: failed to create Barbican secret: Request forbidden: [GET http://10.221.43.28/key-manager/v1/secrets?name=kube_ingress_k8slocal_default_ingress-srv_tls-secret], error message: {"code": 403, "title": "Forbidden", "description": "Secret(s) retrieval attempt not allowed - please review your user/project privileges"}

This is my ingress file

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-srv
  annotations:
    kubernetes.io/ingress.class: "openstack"
    octavia.ingress.kubernetes.io/internal: "false"
    octavia.ingress.kubernetes.io/use-regex: 'true'
spec:
  defaultBackend:
    service:
      name: default-http-backend
      port:
        number: 80
  tls:
    - secretName: tls-secret
  rules:
    - host: abc.com
      http:
        paths:
          - path: /abc/?(.*)
            pathType: Prefix
            backend:
              service:
                name: data-clusterip-svc
                port:
                  number: 5000
          - path: /?(.*)
            pathType: Prefix
            backend:
              service:
                name: abcui-clusterip-svc
                port:
                  number: 8080

gpugal commented 3 years ago

@lingxiankong This ticket can be closed, the https issue is the same as "https://bugs.launchpad.net/barbican/+bug/1627391".

Thanks for your help.

kubernetes / cloud-provider-openstack

[octavia-ingress-controller] 503 Service Unavailable - Error when accessing loadbalancer Floating IP #1423