search

apache / cloudstack

Apache CloudStack is an opensource Infrastructure as a Service (IaaS) cloud computing platform
https://cloudstack.apache.org/
Apache License 2.0
2.06k stars 1.1k forks source link

Static IP address to Kuberenets Loadbalancer #7762

Closed Noelantogerorge closed 1 year ago

Noelantogerorge commented 1 year ago
ISSUE TYPE
COMPONENT NAME

Kubernetes

CLOUDSTACK VERSION

4.18

CONFIGURATION

Advance Networking

OS / ENVIRONMENT

All on 22.04 Ubuntu

SUMMARY

Do cloudstack supports static ip address to kubernetes service as loadbalancer.

STEPS TO REPRODUCE

Create a kubernetes service yml as type loadbalancer and provide loadbalancerIP:

EXPECTED RESULTS

Expected to get loadbalancer to run on that static IP address

ACTUAL RESULTS

It is showing external IP as pending and below error is shown

Error syncing load balancer: failed to ensure load balancer: error creating load balancer rule aa37ce2d20b854d9d9714da03c0e5951-tcp-8080: CloudStack API error 537 (CSExceptionErrorCode: 9999): The range specified, 8080-8080, conflicts with rule 658 which has 8080-8080

kiranchavala commented 1 year ago

@Noelantogerorge could you please provide the yaml file you have created

also the output of the command

kubectl get pods -A

for example if you expose a deployment, cloudstack (CKS) should automatically provison a loadbalaner Ip for

kubectl expose deploy/nginx-deployment --port=80 --type=LoadBalancer

Noelantogerorge commented 1 year ago

@kiranchavala Please find the service yml attached <svc.txt>

Kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE jenkins jenkins-0 1/1 Running 0 3d1h kube-system cloud-controller-manager-54d96fb86c-cx7h5 1/1 Running 1 (2d4h ago) 3d3h kube-system coredns-787d4945fb-54zmc 1/1 Running 0 3d3h kube-system coredns-787d4945fb-n7z28 1/1 Running 0 3d3h kube-system etcd-operations-control-189780d0e0d 1/1 Running 0 3d3h kube-system kube-apiserver-operations-control-189780d0e0d 1/1 Running 0 3d3h kube-system kube-controller-manager-operations-control-189780d0e0d 1/1 Running 1 (2d4h ago) 3d3h kube-system kube-proxy-6plxb 1/1 Running 0 3d3h kube-system kube-proxy-6qjcv 1/1 Running 0 3d3h kube-system kube-scheduler-operations-control-189780d0e0d 1/1 Running 1 (2d4h ago) 3d3h kube-system weave-net-6npz2 2/2 Running 1 (3d3h ago) 3d3h kube-system weave-net-7766c 2/2 Running 0 3d3h kubernetes-dashboard dashboard-metrics-scraper-7bc864c59-9zvl4 1/1 Running 0 3d3h kubernetes-dashboard kubernetes-dashboard-6c7ccbcf87-sdcmm 1/1 Running 0 3d3h mtest kanimango-mail-79dcc9c685-wqf4w 1/1 Running 0 3d2h

All pods are in running state and I can deploye a service as loadbalancer. Cloudstack is automatically assigning the public IP. But when I try to deploy service as loadbalancer and a specific IP I am getting below error

kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE jenkins LoadBalancer 10.97.136.234 8080:30406/TCP,50000:30718/TCP 5s

Its logs is as follow Events: Type Reason Age From Message

Normal EnsuringLoadBalancer 71s (x6 over 4m11s) service-controller Ensuring load balancer Warning SyncLoadBalancerFailed 66s (x6 over 4m6s) service-controller Error syncing load balancer: failed to ensure load balancer: error creating load balancer rule a4109576280094e5f851cc5b51e21f6c-tcp-8080: CloudStack API error 537 (CSExceptionErrorCode: 9999): The range specified, 8080-8080, conflicts with rule 658 which has 8080-8080

weizhouapache commented 1 year ago

@Noelantogerorge the warning message says "The range specified, 8080-8080, conflicts with rule 658 which has 8080-8080" can you try with another public IP or use different port ?

Noelantogerorge commented 1 year ago

@weizhouapache Ya I tried that. Changing the IP or PORT I am getting same error...

kiranchavala commented 1 year ago

@Noelantogerorge

Can you try writing the service as the following yaml file 

❯ kubectl get svc nginx-deployment
NAME               TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
nginx-deployment   LoadBalancer   10.109.88.98   10.0.54.83    80:32739/TCP   4m24s

❯ kubectl get svc nginx-deployment -o yaml
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2023-07-25T04:31:03Z"
  finalizers:
  - service.kubernetes.io/load-balancer-cleanup
  name: nginx-deployment
  namespace: default
  resourceVersion: "995303"
  uid: 224691be-fa86-4b07-938c-c86d948b99fb
spec:
  allocateLoadBalancerNodePorts: true
  clusterIP: 10.109.88.98
  clusterIPs:
  - 10.109.88.98
  externalTrafficPolicy: Cluster
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - nodePort: 32739
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  sessionAffinity: None
  type: LoadBalancer
status:
apiVersion: v1
  loadBalancer:
apiVersion: v1
    ingress:
    - ip: 10.0.54.83

@weizhouapache

I think this can be an enhancement request

https://github.com/apache/cloudstack-kubernetes-provider/blob/main/cloudstack_loadbalancer.go

Cloudstack randomly pick a public ip address , providing a specific ip is not working

Noelantogerorge commented 1 year ago

@kiranchavala I tried as per the above mentioned yml file. After applying the service I could get an external IP address for my service but not the IP that I mentioned. loadBalancer: apiVersion: v1 ingress:

weizhouapache commented 1 year ago 


@weizhouapache

I think this can be an enhancement request

https://github.com/apache/cloudstack-kubernetes-provider/blob/main/cloudstack_loadbalancer.go

Cloudstack randomly pick a public ip address , providing a specific ip is not working

@kiranchavala agree, this seems to be a new feature of https://github.com/apache/cloudstack-kubernetes-provider

kiranchavala commented 1 year ago

thanks @weizhouapache

@Noelantogerorge i have a logged a enhancement request here

https://github.com/apache/cloudstack-kubernetes-provider/issues/51

weizhouapache commented 1 year ago

thanks @kiranchavala

can we close this issue ? @Noelantogerorge @kiranchavala