search

kubernetes / cloud-provider-openstack

Apache License 2.0
616 stars 601 forks source link

Dual Stack Octavia - Buggy Behavior / Not Working Correctly - Can Not Make An IPv6 Load Balancer nor Dual Stack #2333

Closed noelashford closed 5 months ago

noelashford commented 1 year ago

Why is this requesting an IPv4 VIP ?! Cam i ONLY have Single Stack in the cloud.yaml !?

What do i need in "address-sort-order" under network to correct this behavior.... ? I have two default routes (ipv6 and IPv4) 

[Router]
router-id=66edcd05-f9c3-45ca-9356-3779840d9185  <- Dual Stack Router with 0::/0 and 0.0.0.0/0 routes

[LoadBalancer]
use-octavia=true
network-id=f98de447-6357-41bb-971e-fa21b1721a0c <- Dua; Stck Subnet
floating-network-id=a027ee56-79ab-4dd9-ada3-4d56eece242d <- Dual Stack Subnet

apiVersion: v1
kind: Service
metadata:
  name: hostname-server
  namespace: default
  labels:
    app.kubernetes.io/name: hostname-server
spec:
  ipFamilyPolicy: SingleStack
  ipFamilies:
  - IPv6
  type: LoadBalancer
  selector:
    app.kubernetes.io/name: MyApp
  ports:
  - port: 80
    protocol: TCP
    targetPort: 8080

root@5net-k8s-master-1:~/kubernetes# kubectl describe svc hostname-server
Name:                     hostname-server
Namespace:                default
Labels:                   app.kubernetes.io/name=hostname-server
Annotations:              <none>
Selector:                 app.kubernetes.io/name=MyApp
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv6
IP:                       fc00:db8:42:1::a362
IPs:                      fc00:db8:42:1::a362
Port:                     <unset>  80/TCP
TargetPort:               8080/TCP
NodePort:                 <unset>  31366/TCP
Endpoints:                <none>
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type     Reason                  Age              From                Message
  ----     ------                  ----             ----                -------
  Warning  SyncLoadBalancerFailed  6s               service-controller  Error syncing load balancer: failed to ensure load balancer: error creating loadbalancer kube_service_kubernetes_default_hostname-server: error creating loadbalancer {"name":"kube_service_kubernetes_default_hostname-server","description":"Kubernetes external service default/hostname-server from cluster kubernetes","vip_network_id":"f98de447-6357-41bb-971e-fa21b1721a0c","provider":"amphora","listeners":[{"protocol":"TCP","protocol_port":80,"name":"listener_0_kube_service_kubernetes_default_hostname-server","default_pool":{"lb_algorithm":"ROUND_ROBIN","protocol":"TCP","name":"pool_0_kube_service_kubernetes_default_hostname-server","members":[{"address":"2606:8180:5001:4004::211","protocol_port":31366,"name":"5net-k8s-worker-1"},{"address":"2606:8180:5001:4004::c1","protocol_port":31366,"name":"5net-k8s-worker-2"},{"address":"2606:8180:5001:4004::168","protocol_port":31366,"name":"5net-k8s-worker-3"}]},"connection_limit":-1,"timeout_client_data":50000,"timeout_member_data":50000,"timeout_member_connect":5000,"timeout_tcp_inspect":0,"allowed_cidrs":["::/0"],"tags":["kube_service_kubernetes_default_hostname-server"]}],"tags":["kube_service_kubernetes_default_hostname-server"]}: Bad request with: [POST https://192.168.5.99:9876/v2.0/lbaas/loadbalancers], error message: {"faultcode": "Client", "faultstring": "Validation failure: CIDR ::/0 IP version incompatible with VIP 172.16.5.39 IP version.", "debuginfo": null}
  Normal   EnsuringLoadBalancer    1s (x2 over 8s)  service-controller  Ensuring load balancer
MaysaMacedo commented 1 year ago

Given it's not possible to have FIPs with IPv6, could you try again without specifying any network-id? Also, it seems the members subnet can be identified based on the Node addresses and IPFamily.

noelashford commented 1 year ago

That did not work seemingly.

[LoadBalancer]
use-octavia=true
member-subnet-id=084267f1-ab2b-4827-8e53-229d63d448ad
root@5net-k8s-master-1:~/kubernetes# kubectl describe svc hostname-server
Name:                     hostname-server
Namespace:                default
Labels:                   app.kubernetes.io/name=hostname-server
Annotations:              <none>
Selector:                 app.kubernetes.io/name=MyApp
Type:                     LoadBalancer
IP Family Policy:         PreferDualStack
IP Families:              IPv4,IPv6
IP:                       10.96.7.81
IPs:                      10.96.7.81,fc00:db8:42:1::79e7
Port:                     <unset>  80/TCP
TargetPort:               8080/TCP
NodePort:                 <unset>  30486/TCP
Endpoints:                <none>
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type     Reason                  Age              From                Message
  ----     ------                  ----             ----                -------
  Normal   EnsuringLoadBalancer    0s (x2 over 5s)  service-controller  Ensuring load balancer
  Warning  SyncLoadBalancerFailed  0s (x2 over 5s)  service-controller  Error syncing load balancer: failed to ensure load balancer: failed to get subnet to create load balancer for service default/hostname-server: Resource not found: [GET https://192.168.5.99:8774/v2.1/servers//os-interface], error message: {"itemNotFound": {"code": 404, "message": "Instance os-interface could not be found."}}
dulek commented 1 year ago

Are you running a dual stack environment? In order for IPv6 LBs to work you need your nodes to have IPv6 addresses too. This is how LB Services work.

It seems like you're using some older version of CPO. Please tell us which one.

Doing some guesses based on the error message I think you might have something weird as providerID in the node [1]. The call to Nova API seems to be done with empty ID of the server. Please tell us more about your setup. Please note that if your nodes are baremetal, then CPO will not work properly.

[1] https://github.com/kubernetes/cloud-provider-openstack/blob/fe71c9f0cfe57c9b39df5ff2c5a961caefe1f6bf/pkg/openstack/loadbalancer.go#L716-L719

noelashford commented 1 year ago

Nodes are VM - Dual stack of course as is K8S. I am trying IPv6 only as Dual stack seems to not be supported with Octavia as of yet. K8s = current., OS = 2023.1, CPO = current.

Ref to provider ID, I am using CPO & the following cloud.yaml (I am hiding the Auth part)

[Router] router-id=66edcd05-f9c3-45ca-9356-3779840d9185

[LoadBalancer] use-octavia=true member-subnet-id=084267f1-ab2b-4827-8e53-229d63d448ad

MaysaMacedo commented 1 year ago

@noelashford Can you try removing the member-subnet-id setting? This way I believe CPO will try to discover the IPv6 subnet based on the Node addresses. I tried creating a IPv6 loadbalancer Service without specifying any subnet/network setting and it got created.

k8s-triage-robot commented 7 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

noelashford commented 7 months ago

Is this by chance fixed ? does it now support multi vip dual stack networks / IP's?

dulek commented 6 months ago

We're testing dual stack and IPv6 setups with OpenShift and upstream CPO is fine if you're not requesting dual stack LBs. If you need RequireDualStack, then CPO currently does not support that.

noelashford commented 6 months ago

If you need anyone to assist w testing, I have a working dual stack network. Also notably, I run private fc::/7 style network, global IPv6 & lan style 10.x ipv4. I would suggest that it have the ability for multiple IP's of either kind. Do you have the source code for these changes and does it need any changes to the amphora image or just Octavia ?

On Wed, Feb 28, 2024, 12:28 PM Michal Dulko @.***> wrote:

We're testing dual stack and IPv6 setups with OpenShift and upstream CPO is fine if you're not requesting dual stack LBs. If you need RequireDualStack, then CPO currently does not support that.

— Reply to this email directly, view it on GitHub https://github.com/kubernetes/cloud-provider-openstack/issues/2333#issuecomment-1969490015, or unsubscribe https://github.com/notifications/unsubscribe-auth/A2QZ4JZNK3AS7EBRDZU3UNLYV5SMHAVCNFSM6AAAAAA324RKXKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNRZGQ4TAMBRGU . You are receiving this because you were mentioned.Message ID: @.***>

dulek commented 6 months ago

I don't understand the source code request. We use upstream. We do not support dual stack services at the moment, only either IPv4 or IPv6.

noelashford commented 6 months ago

Ah sorry I misread the open shift comment to mean it's being tested there - was going to see if I could port it. No worries, as a question though as I may be able to fix it myself, where is the limitation generally speaking ? Amphora or Octavia code ? Ie does the image support it and it's just Octavia not supporting dual stack ?

On Wed, Feb 28, 2024, 12:34 PM Michal Dulko @.***> wrote:

I don't understand the source code request. We use upstream. We do not support dual stack services at the moment, only either IPv4 or IPv6.

— Reply to this email directly, view it on GitHub https://github.com/kubernetes/cloud-provider-openstack/issues/2333#issuecomment-1969499555, or unsubscribe https://github.com/notifications/unsubscribe-auth/A2QZ4J4D6Q6JDUEZFXOR45LYV5TBNAVCNFSM6AAAAAA324RKXKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNRZGQ4TSNJVGU . You are receiving this because you were mentioned.Message ID: @.***>

dulek commented 6 months ago

Octavia supports dual stack with Amphora for a bunch of OpenStack releases now. It's just not coded in CPO. See #1937 for a discussion on implementation.

dulek commented 6 months ago

Can this be closed now then?

k8s-triage-robot commented 5 months ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten