search

oracle / oci-cloud-controller-manager

Kubernetes Cloud Controller Manager implementation for Oracle Cloud Infrastructure
Apache License 2.0
135 stars 86 forks source link

Can not add NLB backend automatically after create load balancer service. #404

Open alex-wong123 opened 1 year ago

alex-wong123 commented 1 year ago

Is this a BUG REPORT or FEATURE REQUEST?

Choose one: BUG REPORT or FEATURE REQUEST

Versions

CCM Version: V1.23.0 Environment:

What happened?

Created a load balancer service successfully, and then a NLB was created automatically. But access the service failed, then check the NLB, there was no member in backend.

What you expected to happen?

The load balancer service can be accessed successfully.

How to reproduce it (as minimally and precisely as possible)?

CCM was installed as doc

Anything else we need to know?

The logs of oci-cloud-controller-manager pod using 'kubectl logs oci-cloud-controller-manager-7lblr' as bellow 2022-11-16T14:27:10.622Z INFO oci/load_balancer.go:849 Updating load balancer {"component": "cloud-controller-manager", "loadBalancerName": "default/nginx-nlb-svc/5f63bfab-39af-477d-b6e1-9d4c4b39dc44", "loadBalancerType": "nlb"} 2022-11-16T14:27:10.622Z INFO oci/load_balancer.go:409 Ensuring load balancer {"component": "cloud-controller-manager", "loadBalancerName": "default/nginx-nlb-svc/5f63bfab-39af-477d-b6e1-9d4c4b39dc44", "serviceName": "nginx-nlb-svc", "loadBalancerType": "nlb", "nodes": 3} 2022-11-16T14:27:10.664Z WARN oci/load_balancer_spec.go:515 Node "k8s-c2node3" has an empty ProviderID. {"component": "cloud-controller-manager", "loadBalancerName": "default/nginx-nlb-svc/5f63bfab-39af-477d-b6e1-9d4c4b39dc44", "serviceName": "nginx-nlb-svc", "loadBalancerType": "nlb", "loadBalancerID": "ocid1.networkloadbalancer.oc1.us-sanjose-1.amaaaaaay5l3z3ya77fawpnnntrrdghzgq6e5c4rb7aldpvjovufayp2tapa", "loadBalancerType": "nlb"} 2022-11-16T14:27:10.664Z WARN oci/load_balancer_spec.go:515 Node "k8s-c2node1" has an empty ProviderID. {"component": "cloud-controller-manager", "loadBalancerName": "default/nginx-nlb-svc/5f63bfab-39af-477d-b6e1-9d4c4b39dc44", "serviceName": "nginx-nlb-svc", "loadBalancerType": "nlb", "loadBalancerID": "ocid1.networkloadbalancer.oc1.us-sanjose-1.amaaaaaay5l3z3ya77fawpnnntrrdghzgq6e5c4rb7aldpvjovufayp2tapa", "loadBalancerType": "nlb"} 2022-11-16T14:27:10.664Z WARN oci/load_balancer_spec.go:515 Node "k8s-c2node2" has an empty ProviderID. {"component": "cloud-controller-manager", "loadBalancerName": "default/nginx-nlb-svc/5f63bfab-39af-477d-b6e1-9d4c4b39dc44", "serviceName": "nginx-nlb-svc", "loadBalancerType": "nlb", "loadBalancerID": "ocid1.networkloadbalancer.oc1.us-sanjose-1.amaaaaaay5l3z3ya77fawpnnntrrdghzgq6e5c4rb7aldpvjovufayp2tapa", "loadBalancerType": "nlb"} 2022-11-16T14:27:10.664Z INFO oci/load_balancer_security_lists.go:96 Security list management mode: "None". Not managing security lists. {"component": "cloud-controller-manager"} 2022-11-16T14:27:10.664Z INFO oci/load_balancer.go:543 Successfully updated loadbalancer {"component": "cloud-controller-manager", "loadBalancerName": "default/nginx-nlb-svc/5f63bfab-39af-477d-b6e1-9d4c4b39dc44", "serviceName": "nginx-nlb-svc", "loadBalancerType": "nlb", "loadBalancerID": "ocid1.networkloadbalancer.oc1.us-sanjose-1.amaaaaaay5l3z3ya77fawpnnntrrdghzgq6e5c4rb7aldpvjovufayp2tapa", "loadBalancerType": "nlb"} I1116 14:27:10.664509 1 controller.go:748] Successfully updated 1 out of 1 load balancers to direct traffic to the updated set of nodes I1116 14:27:10.664593 1 event.go:294] "Event occurred" object="default/nginx-nlb-svc" kind="Service" apiVersion="v1" type="Normal" reason="UpdatedLoadBalancer" message="Updated load balancer with new hosts" E1116 14:29:34.871159 1 node_controller.go:242] Error getting instance metadata for node addresses: error fetching node by provider ID: MapProviderIDToInstanceID: provider ID is empty, and error by node name: Failed to map providerID to instanceID E1116 14:29:35.056435 1 node_controller.go:242] Error getting instance metadata for node addresses: error fetching node by provider ID: MapProviderIDToInstanceID: provider ID is empty, and error by node name: Failed to map providerID to instanceID E1116 14:29:35.256508 1 node_controller.go:242] Error getting instance metadata for node addresses: error fetching node by provider ID: MapProviderIDToInstanceID: provider ID is empty, and error by node name: Failed to map providerID to instanceID E1116 14:34:36.270177 1 node_controller.go:242] Error getting instance metadata for node addresses: error fetching node by provider ID: MapProviderIDToInstanceID: provider ID is empty, and error by node name: Failed to map providerID to instanceID E1116 14:34:36.441161 1 node_controller.go:242] Error getting instance metadata for node addresses: error fetching node by provider ID: MapProviderIDToInstanceID: provider ID is empty, and error by node name: Failed to map providerID to instanceID E1116 14:34:36.705268 1 node_controller.go:242] Error getting instance metadata for node addresses: error fetching node by provider ID: MapProviderIDToInstanceID: provider ID is empty, and error by node name: Failed to map providerID to instanceID E1116 14:39:37.425184 1 node_controller.go:242] Error getting instance metadata for node addresses: error fetching node by provider ID: MapProviderIDToInstanceID: provider ID is empty, and error by node name: Failed to map providerID to instanceID E1116 14:39:37.551867 1 node_controller.go:242] Error getting instance metadata for node addresses: error fetching node by provider ID: MapProviderIDToInstanceID: provider ID is empty, and error by node name: Failed to map providerID to instanceID E1116 14:39:37.642478 1 node_controller.go:242] Error getting instance metadata for node addresses: error fetching node by provider ID: MapProviderIDToInstanceID: provider ID is empty, and error by node name: Failed to map providerID to instanceID E1116 14:44:38.310129 1 node_controller.go:242] Error getting instance metadata for node addresses: error fetching node by provider ID: MapProviderIDToInstanceID: provider ID is empty, and error by node name: Failed to map providerID to instanceID E1116 14:44:38.552157 1 node_controller.go:242] Error getting instance metadata for node addresses: error fetching node by provider ID: MapProviderIDToInstanceID: provider ID is empty, and error by node name: Failed to map providerID to instanceID E1116 14:44:38.776831 1 node_controller.go:242] Error getting instance metadata for node addresses: error fetching node by provider ID: MapProviderIDToInstanceID: provider ID is empty, and error by node name: Failed to map providerID to instanceID

AkarshES commented 1 year ago

It looks like you are not setting the provider ID during Kubelet initialization. Please see the kubelet docs for adding provider ID via command line arg or kubelet config file https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/ image

alex-wong123 commented 1 year ago

The flags are set for kube-controller-manager and kubelet ordering the docs. root 4586 0.8 0.7 826256 118492 ? Ssl 14:30 4:16 kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf --bind-address=127.0.0.1 --client-ca-file=/etc/kubernetes/pki/ca.crt --cloud-provider=external --cluster-cidr=10.246.0.0/16 --cluster-name=kubernetes --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt --cluster-signing-key-file=/etc/kubernetes/pki/ca.key --controllers=*,bootstrapsigner,tokencleaner --kubeconfig=/etc/kubernetes/controller-manager.conf --leader-elect=true --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --root-ca-file=/etc/kubernetes/pki/ca.crt --service-account-private-key-file=/etc/kubernetes/pki/sa.key --service-cluster-ip-range=10.98.0.0/16 --use-service-account-credentials=true root 20483 2.2 0.7 1783240 121780 ? Ssl 14:27 12:10 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cloud-provider=external --provider-id=ocid1.instance.oc1.us-sanjose-1.anzwuljry5l3z3ycjdgcnhctmu3aljxjhki2oglgjmqd75ymzpzxmnudeg3a --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.4.1

keinou commented 1 year ago

+1

4kord commented 10 months ago

node_controller.go takes ProviderID from a node spec (https://github.com/kubernetes/cloud-provider/blob/master/controllers/node/node_controller.go#L663C30-L663C30). After adding ProviderID to spec for each node the "Error getting instance metadata for node addresses" errors were gone and the backends were added