Loadbalancer example stops working when the deployment is scaled up.

Related to vl3-lb feature: https://github.com/networkservicemesh/deployments-k8s/tree/main/examples/features/vl3-lb

Expected Behavior

The traffic between the server and the client should not be affected by the addition of the new server (e.g., scaling up of the finance deployment). I would also expect that switching to iperf traffic (as in changing the image of the server from http-echo to net-tools) will have no effect as longs as the ports are the same and match the network service description.

Current Behavior

TCP traffic between the server and client stops when the deployment is scaled up and a new server is added.

Failure Information (for bugs)

TCP traffic between the server and client stops when the deployment is scaled up and a new server is added.

Steps to Reproduce

install spire via:

kubectl apply -k https://github.com/networkservicemesh/deployments-k8s/examples/spire/single_cluster?ref=a070c14f871cb904f375727aa61f4cd031abc8a7
kubectl apply -f https://raw.githubusercontent.com/networkservicemesh/deployments-k8s/a070c14f871cb904f375727aa61f4cd031abc8a7/examples/spire/single_cluster/clusterspiffeid-template.yaml
kubectl apply -f https://raw.githubusercontent.com/networkservicemesh/deployments-k8s/a070c14f871cb904f375727aa61f4cd031abc8a7/examples/spire/base/clusterspiffeid-webhook-template.yaml

link: https://github.com/networkservicemesh/deployments-k8s/tree/main/examples/spire/single_cluster

install NSM:

kubectl apply -k https://github.com/networkservicemesh/deployments-k8s/examples/basic?ref=a070c14f871cb904f375727aa61f4cd031abc8a7

link: https://github.com/networkservicemesh/deployments-k8s/tree/main/examples/basic

[working example] install Loadbalancer feature example:

kubectl apply -k https://github.com/networkservicemesh/deployments-k8s/examples/features/vl3-lb?ref=a070c14f871cb904f375727aa61f4cd031abc8a7

link: https://github.com/networkservicemesh/deployments-k8s/tree/main/examples/features/vl3-lb

Output as expected:

$ kubectl exec deployments/finance-client -n ns-vl3-lb -- curl -s finance.vl3-lb:8080 | grep "Hello! I'm finance-server"
Defaulted container "nettools" out of: nettools, cmd-nsc, cmd-nsc-init (init)
Hello! I'm finance-server-6b9d477dbc-rmsmd

$ kubectl exec deployments/finance-client -n ns-vl3-lb -- curl -s finance.vl3-lb:8080 | grep "Hello! I'm finance-server"
Defaulted container "nettools" out of: nettools, cmd-nsc, cmd-nsc-init (init)
Hello! I'm finance-server-6b9d477dbc-xg75j

$ kubectl exec deployments/finance-client -n ns-vl3-lb -- curl -s finance.vl3-lb:8080 | grep "Hello! I'm finance-server"
Defaulted container "nettools" out of: nettools, cmd-nsc, cmd-nsc-init (init)
Hello! I'm finance-server-6b9d477dbc-rmsmd

Scaling up and down the server deployment was successful.

[not working example] install loadbalancer feature example with iperf instead of http-echo. The used ports are the same, as well as the traffic (TCP). The goal was to have a longer connection.

$ git diff                           
diff --git a/examples/features/vl3-lb/finance-server.yaml b/examples/features/vl3-lb/finance-server.yaml       
index b7a38386186..31e8aef8534 100644                                                                          
--- a/examples/features/vl3-lb/finance-server.yaml                                                             
+++ b/examples/features/vl3-lb/finance-server.yaml                                                             
@@ -18,14 +18,16 @@ spec:                                                                                      
     spec:                                                                                                     
       containers:                                                                                             
         - name: nginx                                                                                         
-          image: hashicorp/http-echo:1.0                                                                      
+          #image: hashicorp/http-echo:1.0                                                                     
+          image: aeciopires/nettools:1.0.0                                                                    
+          command: ["/bin/bash", "-c", "iperf3 -s -p 8081"]                                                   
           env:                                                                                                
           - name: POD_NAME                                                                                    
             valueFrom:                                                                                        
               fieldRef:                                                                                       
                 fieldPath: metadata.name                                                                      
-          args:                                                                                               
-            - "-text=Hello! I'm $(POD_NAME)"                                                                  
-            - -listen=:8081                                                                                   
+          #args:                                                                                              
+          #  - "-text=Hello! I'm $(POD_NAME)"                                                                 
+          #  - -listen=:8081                                                                                  
           ports:                                                                                              
             - containerPort: 8081

When I try running iperf3 client to connect to one of the servers, I get mixed results. 9/10 times, iperf established a connection but doesnt actually send any data and just "hangs" (see bellow - not working output). Sometimes, it does connect, and I get the expected output 1Mbits/sec (see bellow - working output)

[not working output]

finance-client-765fdddd77-jx6hl:/# iperf3 -c finance.vl3-lb -p 8080 -b 1M -t 1000
Connecting to host finance.vl3-lb, port 8080
[  5] local 172.16.0.6 port 48586 connected to 172.16.0.2 port 8080

^C[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-13891.32 sec  0.00 Bytes  0.00 bits/sec    0   87.4 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-13891.32 sec  0.00 Bytes  0.00 bits/sec    0             sender
[  5]   0.00-13891.32 sec  0.00 Bytes  0.00 bits/sec                  receiver
iperf3: interrupt - the client has terminated

[working output]

finance-client-765fdddd77-jx6hl:/# iperf3 -c finance.vl3-lb -p 8080 -b 1M -t 1000
Connecting to host finance.vl3-lb, port 8080
[  5] local 172.16.0.6 port 49280 connected to 172.16.0.2 port 8080
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   128 KBytes  1.05 Mbits/sec    0    140 KBytes
[  5]   1.00-2.00   sec   128 KBytes  1.05 Mbits/sec    0    140 KBytes
[  5]   2.00-3.00   sec   128 KBytes  1.05 Mbits/sec    0    140 KBytes
[  5]   3.00-4.00   sec   128 KBytes  1.05 Mbits/sec    0    140 KBytes
[  5]   4.00-5.00   sec   128 KBytes  1.05 Mbits/sec    0    149 KBytes
^C[  5]   5.00-5.60   sec   128 KBytes  1.76 Mbits/sec    0    140 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-5.60   sec   768 KBytes  1.12 Mbits/sec    0             sender
[  5]   0.00-5.60   sec  0.00 Bytes  0.00 bits/sec                  receiver

4.1 Scaling while the traffic is running. I tried to scale up the deployment, while the traffic was running correctly between one of the finance-servers and the finance-client, using the command bellow (to go from 3 to 4 servers).

kubectl scale deployment finance-server -n ns-vl3-lb --replicas=4

The moment the 4th finance server started running, the iperf reported 0bits/sec (see below).

finance-client-765fdddd77-jx6hl:/# iperf3 -c finance.vl3-lb -p 8080 -b 1M -t 1000      
Connecting to host finance.vl3-lb, port 8080                                           
[  5] local 172.16.0.6 port 58198 connected to 172.16.0.2 port 8080                    
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd                       
[  5]   0.00-1.00   sec   128 KBytes  1.05 Mbits/sec    0    140 KBytes                
[  5]   1.00-2.00   sec   128 KBytes  1.05 Mbits/sec    0    140 KBytes                
[  5]   2.00-3.00   sec   128 KBytes  1.05 Mbits/sec    0    140 KBytes                
[  5]   3.00-4.00   sec   128 KBytes  1.05 Mbits/sec    0    149 KBytes                
[  5]   4.00-5.00   sec   128 KBytes  1.05 Mbits/sec    0    140 KBytes                
[  5]   5.00-6.00   sec   128 KBytes  1.04 Mbits/sec    0    140 KBytes                
[  5]   6.00-7.00   sec   128 KBytes  1.05 Mbits/sec    0    149 KBytes                
[  5]   7.00-8.00   sec   128 KBytes  1.05 Mbits/sec    0    149 KBytes                
[  5]   8.00-9.00   sec   128 KBytes  1.05 Mbits/sec    0    140 KBytes                
[  5]   9.00-10.00  sec   128 KBytes  1.05 Mbits/sec    0    140 KBytes                
[  5]  10.00-11.00  sec   128 KBytes  1.05 Mbits/sec    1   8.74 KBytes                
[  5]  11.00-12.00  sec   128 KBytes  1.05 Mbits/sec    2   8.74 KBytes                
[  5]  12.00-13.00  sec  0.00 Bytes  0.00 bits/sec    0   8.74 KBytes                  
[  5]  13.00-14.00  sec  0.00 Bytes  0.00 bits/sec    1   8.74 KBytes                  
[  5]  14.00-15.00  sec  0.00 Bytes  0.00 bits/sec    0   8.74 KBytes                  
[  5]  15.00-16.00  sec  0.00 Bytes  0.00 bits/sec    0   8.74 KBytes                  
[  5]  16.00-17.00  sec  0.00 Bytes  0.00 bits/sec    1   8.74 KBytes

Context

The setup is running a singular k8s cluster with only one node. These are the only deployments running in the cluster.

Kubernetes Version: 1.28

networkservicemesh / deployments-k8s