Service NodePort works Intermittently

[rowhit]

I have the following setup running Ubuntu 20.04 with 2 cores and 2GB each:

v110: k0s-controller v111: k0sworker-1 v112: k0sworker-2

I am configuring the above using the following configuration file:

apiVersion: k0sctl.k0sproject.io/v1beta1                                                                                                                                                                             
kind: Cluster                                                                                                                                                                                                        
metadata:                                                                                                                                                                                                            
  name: k0s-cluster                                                                                                                                                                                                  
spec:                                                                                                                                                                                                                
  hosts:                                                                                                                                                                                                             
  - ssh:                                                                                                                                                                                                             
  | ¦ address: 10.118.21.1                                                                                                                                                                                           
  | ¦ user: root                                                                                                                                                                                                     
  | ¦ port: 22                                                                                                                                                                                                       
  | ¦ keyPath: /root/.ssh/id_rsa                                                                                                                                                                                     
  | role: controller                                                                                                                                                                                                 
  | privateInterface: nm-vx-ol                                                                                                                                                                                       
  | privateAddress: 10.118.21.1                                                                                                                                                                                      
  - ssh:                                                                                                                                                                                                             
  | ¦ address: 10.118.21.2                                                                                                                                                                                           
  | ¦ user: root                                                                                                                                                                                                     
  | ¦ port: 22                                                                                                                                                                                                       
  | ¦ keyPath: /root/.ssh/id_rsa                                                                                                                                                                                     
  | role: worker                                                                                                                                                                                                     
  | privateInterface: nm-vx-ol                                                                                                                                                                                       
  | privateAddress: 10.118.21.2                                                                                                                                                                                      
  - ssh:                                                                                                                                                                                                             
  | ¦ address: 10.118.21.3                                                                                                                                                                                           
  | ¦ user: root                                                                                                                                                                                                     
  | ¦ port: 22                                                                                                                                                                                                       
  | ¦ keyPath: /root/.ssh/id_rsa                                                                                                                                                                                     
  | role: worker                                                                                                                                                                                                     
  | privateInterface: nm-vx-ol                                                                                                                                                                                       
  | privateAddress: 10.118.21.3                                                                                                                                                                                      
  k0s:                                                                                                                                                                                                               
  | version: 1.23.8+k0s.0                                                                                                                                                                                            
  | dynamicConfig: false   

The nm-vx-ol is a wireguard interface managed by netmaker service. My ufw rules are as follows on all the worker nodes to ensure protected network with no data accepted from anywhere except the wireguard interface.

$ ufw status
Status: active

To                         Action      From
--                         ------      ----                 
Anywhere                   ALLOW       10.118.21.0/24

I am running a simple nodeport service based webserver for testing:

# file: web-server.yaml                                                                                                                                                                                                                                                                                                                                                                        
################################################################################                                                                                                                                                                                                                                                                                                               

#*******************************************************************************                                                                                                                                                                                                                                                                                                               
+--  7 lines: create namespace-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
#*******************************************************************************                                                                                                                                                                                                                                                                                                               

#*******************************************************************************                                                                                                                                                                                                                                                                                                               
# create deployment                                                                                                                                                                                                                                                                                                                                                                            
#*******************************************************************************                                                                                                                                                                                                                                                                                                               
---                                                                                                                                                                                                                                                                                                                                                                                            
apiVersion: apps/v1                                                                                                                                                                                                                                                                                                                                                                            
kind: Deployment                                                                                                                                                                                                                                                                                                                                                                               
metadata:                                                                                                                                                                                                                                                                                                                                                                                      
  name: web-server-deployment                                                                                                                                                                                                                                                                                                                                                                  
  namespace: web                                                                                                                                                                                                                                                                                                                                                                               
spec:                                                                                                                                                                                                                                                                                                                                                                                          
  replicas: 3                                                                                                                                                                                                                                                                                                                                                                                  
  selector:                                                                                                                                                                                                                                                                                                                                                                                    
  | matchLabels:                                                                                                                                                                                                                                                                                                                                                                               
  | ¦ app: web                                                                                                                                                                                                                                                                                                                                                                                 
  template:                                                                                                                                                                                                                                                                                                                                                                                    
  | metadata:                                                                                                                                                                                                                                                                                                                                                                                  
  | ¦ labels:                                                                                                                                                                                                                                                                                                                                                                                  
  | ¦ ┆ app: web                                                                                                                                                                                                                                                                                                                                                                               
  | spec:                                                                                                                                                                                                                                                                                                                                                                                      
  | ¦ containers:                                                                                                                                                                                                                                                                                                                                                                              
  | ¦ - name: httpd                                                                                                                                                                                                                                                                                                                                                                            
  | ¦ ┆ image: httpd:2.4.48-alpine3.14                                                                                                                                                                                                                                                                                                                                                         
  | ¦ ┆ ports:                                                                                                                                                                                                                                                                                                                                                                                 
  | ¦ ┆ - containerPort: 80                                                                                                                                                                                                                                                                                                                                                                    
#*******************************************************************************                                                                                                                                                                                                                                                                                                               

#*******************************************************************************                                                                                                                                                                                                                                                                                                               
# create service                                                                                                                                                                                                                                                                                                                                                                               
#*******************************************************************************                                                                                                                                                                                                                                                                                                               
---                                                                                                                                                                                                                                                                                                                                                                                            
apiVersion: v1                                                                                                                                                                                                                                                                                                                                                                                 
kind: Service                                                                                                                                                                                                                                                                                                                                                                                  
metadata:                                                                                                                                                                                                                                                                                                                                                                                      
  name: web-server-service                                                                                                                                                                                                                                                                                                                                                                     
  namespace: web                                                                                                                                                                                                                                                                                                                                                                               
spec:                                                                                                                                                                                                                                                                                                                                                                                          
  selector:                                                                                                                                                                                                                                                                                                                                                                                    
  | app: web                                                                                                                                                                                                                                                                                                                                                                                   
  ports:                                                                                                                                                                                                                                                                                                                                                                                       
  | - protocol: TCP                                                                                                                                                                                                                                                                                                                                                                            
  | ¦ port: 8080                                                                                                                                                                                                                                                                                                                                                                               
  | ¦ targetPort: 80                                                                                                                                                                                                                                                                                                                                                                           
  | ¦ nodePort: 30000                                                                                                                                                                                                                                                                                                                                                                          
  type: NodePort                                                                                                                                                                                                                                                                                                                                                                               
#*******************************************************************************                                                                                                                                                                                                                                                                                                               

################################################################################        

I am able to curl from on the nodeport from both worker nodes but only for first few tries (2-3 times) before it stops.

curl v111.nm-vx-ol:30000
<html><body><h1>It works!</h1></body></html>

I would eventually want to load balance these nodePorts only accessible via wireguard with ingress to Traefik. However, I am unable to get past the first stage. I would be grateful if you can provide me some insight. I am new to kubernetes (thanks to k0s that I have been able to spin a cluster quickly unlike any other alternative I have tried) and have limited knowledge of iptables thus using UFW. I looked at a few suggestions online where it was suggested enable port fowarding for UFW which I have already done.

ufw default allow FORWARD && ufw reload
# above setups /etc/default/ufw DEFAULT_FORWARD_POLICY correctly

I looked at the calico settings for using the wireguard in the documentation(https://docs.k0sproject.io/v1.23.3+k0s.1/configuration/#specnetworkcalico) but was not able to fully understand the consequences. I would be grateful if some one could me an alternative config file that may work with existing wireguard interface.

[rowhit]

The issue lies in the firewall configuration. After have dropped the restrictive firewall configuration, the system works fine. I apologize for putting out a trivial issue. I need to dig a little deeper into the iptables to create right firewall that would allow k0s to only use the wireguard interface for encrypted communication with servers using public IPs.