search

submariner-io / submariner

Networking component for interconnecting Pods and Services across Kubernetes clusters.
https://submariner.io
Apache License 2.0
2.43k stars 193 forks source link

Remove globalIP annotation from node as part of uninstall operation #1780

Closed sridhargaddam closed 2 years ago

sridhargaddam commented 2 years ago

What happened: Globalnet adds a globalIP annotation to the Gateway node which is used as healthcheck-ip. Ideally this annotation should be removed as part of subctl uninstall .... operation, otherwise it could create issues during re-deployment.

How to reproduce it (as minimally and precisely as possible): Check nodes annotations on cluster2 Run Submariner deployment

bin/subctl deploy-broker --kubeconfig output/kubeconfigs/kind-config-cluster1 --globalnet bin/subctl join --kubeconfig output/kubeconfigs/kind-config-cluster1 broker-info.subm --clusterid cluster1 --natt=false bin/subctl join --kubeconfig output/kubeconfigs/kind-config-cluster2 broker-info.subm --clusterid cluster2 --natt=false Uninstall Submariner from cluster2

bin/subctl uninstall --kubeconfig output/kubeconfigs/kind-config-cluster2 Check node annotations on cluster2 gateway node

Environment: Kind

sridhargaddam commented 2 years ago

Leaving the annotation on the node could create issues when submariner is re-installed on the cluster.

sridhargaddam commented 2 years ago

Logs from one of the clusters which initially had a globalCIDR of 242.2.0.0/16 and after re-deployment of Submariner a different globalCIDR of 242.1.0.0/16 was allocated.

I0408 06:15:14.464203       1 cableengine.go:102] CableEngine controller started, driver: "libreswan"
I0408 06:15:14.464368       1 datastoresyncer.go:68] Starting the datastore syncer
I0408 06:15:14.464614       1 tunnel.go:38] Starting the tunnel controller
I0408 06:15:14.565274       1 healthchecker.go:108] CableEngine HealthChecker started with PingInterval: 1, MaxPacketLossCount: 5
I0408 06:15:14.732686       1 datastoresyncer.go:214] Ensuring we are the only endpoint active for this cluster
I0408 06:15:14.733322       1 datastoresyncer.go:293] Creating local submariner Cluster: types.SubmarinerCluster{ID:"cluster3", Spec:v1.ClusterSpec{ClusterID:"cluster3", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"10.43.0.0/16"}, ClusterCIDR:[]string{"10.42.0.0/24"}, GlobalCIDR:[]string{"242.1.0.0/16"}}} 
I0408 06:15:14.739883       1 datastoresyncer.go:306] Creating local submariner Endpoint: types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster3", CableName:"submariner-cable-cluster3-10-66-208-164", HealthCheckIP:"", Hostname:"edge-3.example.com", Subnets:[]string{"242.1.0.0/16"}, PrivateIP:"10.66.208.164", PublicIP:"x.x.x.x", NATEnabled:true, Backend:"libreswan", BackendConfig:map[string]string{"natt-discovery-port":"4490", "preferred-server":"false", "udp-port":"4500"}}} 
I0408 06:15:14.821108       1 natdiscovery.go:160] Starting NAT discovery for endpoint "submariner-cable-cluster1-10-66-208-162"
I0408 06:15:14.821652       1 pinger.go:104] Starting pinger for IP "242.0.255.254"
I0408 06:15:14.821668       1 healthchecker.go:157] CableEngine HealthChecker started pinger for CableName: "submariner-cable-cluster1-10-66-208-162" with HealthCheckIP "242.0.255.254"
I0408 06:15:14.884195       1 datastoresyncer.go:100] Datastore syncer started
I0408 06:15:14.885081       1 node_handler.go:58] Updating the endpoint HealthCheckIP to globalIP "242.2.255.254"
I0408 06:15:14.885102       1 datastoresyncer.go:306] Creating local submariner Endpoint: types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster3", CableName:"submariner-cable-cluster3-10-66-208-164", HealthCheckIP:"242.2.255.254", Hostname:"edge-3.example.com", Subnets:[]string{"242.1.0.0/16"}, PrivateIP:"10.66.208.164", PublicIP:"x.x.x.x", NATEnabled:true, Backend:"libreswan", BackendConfig:map[string]string{"natt-discovery-port":"4490", "preferred-server":"false", "udp-port":"4500"}}} 
I0408 06:15:15.041846       1 request_handle.go:53] Received request from 10.66.208.162:4490 - REQUEST_NUMBER: 0x60496bb0d200f1b, SENDER: "submariner-cable-cluster1-10-66-208-162", RECEIVER: "submariner-cable-cluster3-10-66-208-164"
I0408 06:15:15.041920       1 request_handle.go:118] Sending response to 10.66.208.162:4490 - REQUEST_NUMBER: 0x60496bb0d200f1b, RESPONSE: OK, SENDER: "submariner-cable-cluster3-10-66-208-164", RECEIVER: "submariner-cable-cluster1-10-66-208-162"
I0408 06:15:15.403596       1 request_send.go:116] Sending request - REQUEST_NUMBER: 0xbcaaa3457dd947bf, SENDER: "submariner-cable-cluster3-10-66-208-164", RECEIVER: "submariner-cable-cluster1-10-66-208-162", USING_SRC: 10.66.208.164:4490, USING_DST: 10.66.208.162:4490
I0408 06:15:15.403713       1 request_send.go:116] Sending request - REQUEST_NUMBER: 0xbcaaa3457dd947c0, SENDER: "submariner-cable-cluster3-10-66-208-164", RECEIVER: "submariner-cable-cluster1-10-66-208-162", USING_SRC: 10.66.208.164:4490, USING_DST: 119.254.120.68:4490
I0408 06:15:15.404128       1 response_handle.go:32] Received response from 10.66.208.162:4490 - REQUEST_NUMBER: 0xbcaaa3457dd947bf, RESPONSE: OK, SENDER: "submariner-cable-cluster1-10-66-208-162", RECEIVER: "submariner-cable-cluster3-10-66-208-164"
I0408 06:15:15.404149       1 remote_endpoint.go:185] selected private IP "10.66.208.162" for endpoint "submariner-cable-cluster1-10-66-208-162"
I0408 06:15:15.404181       1 cableengine.go:193] Installing Endpoint cable "submariner-cable-cluster1-10-66-208-162"
002 listening for IKE messages
002 adding UDP interface vx-submariner 240.66.208.164:500
002 adding UDP interface vx-submariner 240.66.208.164:4500
002 forgetting secrets
002 loading secrets from "/etc/ipsec.secrets"
002 loading secrets from "/etc/ipsec.d/submariner.secrets"
I0408 06:15:15.406679       1 libreswan.go:342] Creating connection(s) for {"metadata":{"name":"cluster1-submariner-cable-cluster1-10-66-208-162","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-10-66-208-162","uid":"275574e8-0bbc-4fe1-bfab-4b58f8e7c5b5","resourceVersion":"751242","generation":1,"creationTimestamp":"2022-04-08T06:15:14Z","labels":{"submariner-io/clusterID":"cluster1"},"managedFields":[{"manager":"submariner-gateway","operation":"Update","apiVersion":"submariner.io/v1","time":"2022-04-08T06:15:14Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:submariner-io/clusterID":{}}},"f:spec":{".":{},"f:backend":{},"f:backend_config":{".":{},"f:natt-discovery-port":{},"f:preferred-server":{},"f:udp-port":{}},"f:cable_name":{},"f:cluster_id":{},"f:healthCheckIP":{},"f:hostname":{},"f:nat_enabled":{},"f:private_ip":{},"f:public_ip":{},"f:subnets":{}}}}]},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-10-66-208-162","healthCheckIP":"242.0.255.254","hostname":"edge-1.example.com","subnets":["242.0.0.0/16"],"private_ip":"10.66.208.162","public_ip":"119.254.120.68","nat_enabled":true,"backend":"libreswan","backend_config":{"natt-discovery-port":"4490","preferred-server":"false","udp-port":"4500"}}} in bi-directional mode
I0408 06:15:15.406805       1 libreswan.go:403] Executing whack with args: [--psk --encrypt --name submariner-cable-cluster1-10-66-208-162-0-0 --id 10.66.208.164 --host 10.66.208.164 --client 242.1.0.0/16 --ikeport 4500 --to --id 10.66.208.162 --host 10.66.208.162 --client 242.0.0.0/16 --ikeport 4500]
002 "submariner-cable-cluster1-10-66-208-162-0-0": added IKEv2 connection
181 "submariner-cable-cluster1-10-66-208-162-0-0" #1: initiating IKEv2 connection
I0408 06:15:15.440649       1 cableengine.go:200] Successfully installed Endpoint cable "submariner-cable-cluster1-10-66-208-162" with remote IP 10.66.208.162
I0408 06:15:17.265987       1 node_handler.go:50] areNodesEquivalent called for "edge-3.example.com", existingGlobalIP "242.2.255.254", newGlobalIP ""
I0408 06:15:17.663197       1 node_handler.go:50] areNodesEquivalent called for "edge-3.example.com", existingGlobalIP "", newGlobalIP "242.1.255.254"
I0408 06:15:17.668841       1 node_handler.go:58] Updating the endpoint HealthCheckIP to globalIP "242.1.255.254"
I0408 06:15:17.668865       1 datastoresyncer.go:306] Creating local submariner Endpoint: types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster3", CableName:"submariner-cable-cluster3-10-66-208-164", HealthCheckIP:"242.1.255.254", Hostname:"edge-3.example.com", Subnets:[]string{"242.1.0.0/16"}, PrivateIP:"10.66.208.164", PublicIP:"x.x.x.x", NATEnabled:true, Backend:"libreswan", BackendConfig:map[string]string{"natt-discovery-port":"4490", "preferred-server":"false", "udp-port":"4500"}}} 
I0408 06:15:18.043399       1 request_handle.go:53] Received request from 10.66.208.162:4490 - REQUEST_NUMBER: 0x60496bb0d200f1d, SENDER: "submariner-cable-cluster1-10-66-208-162", RECEIVER: "submariner-cable-cluster3-10-66-208-164"
I0408 06:15:18.043442       1 request_handle.go:118] Sending response to 10.66.208.162:4490 - REQUEST_NUMBER: 0x60496bb0d200f1d, RESPONSE: OK, SENDER: "submariner-cable-cluster3-10-66-208-164", RECEIVER: "submariner-cable-cluster1-10-66-208-162"
I0408 06:15:38.860116       1 node_handler.go:50] areNodesEquivalent called for "edge-3.example.com", existingGlobalIP "242.1.255.254", newGlobalIP "242.1.255.254"
I0408 06:20:41.458010       1 node_handler.go:50] areNodesEquivalent called for "edge-3.example.com", existingGlobalIP "242.1.255.254", newGlobalIP "242.1.255.254"