search

kubeovn / kube-ovn

A Bridge between SDN and Cloud Native (Project under CNCF)
https://kubeovn.github.io/docs/stable/en/
Apache License 2.0
2k stars 451 forks source link

Issue with SwitchLBRule and DNAT for connectivity from external to service within VPC #4279

Closed tgdfool2 closed 2 days ago

tgdfool2 commented 4 months ago

Kube-OVN Version

v1.12.18

Kubernetes Version

Client Version: v1.28.10 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.28.10

Operation-system/Kernel Version

Ubuntu 22.04.4 LTS 6.5.0-1023-azure

kubespray cluster deployed to an Azure VM.

Description

Dear Community,

I am trying to create a DNAT rule to allow external connections to reach Pods created by a deployment within a VPC. According to the docs (and other older issues, e.g. https://github.com/kubeovn/kube-ovn/issues/3130 or https://github.com/kubeovn/kube-ovn/issues/2636), I created a Vip, a SwitchLBRule, an EIP as well as a DnatRule, but still did not manage to connect to the service from outside.

(My base setup is the one described under https://github.com/kubeovn/kube-ovn/issues/4259#issue-2392733349)

Some questions to help me troubleshooting this issue:

  1. Is the Vip resource necessary or not? There is no mention of it under https://kubeovn.github.io/docs/stable/en/advance/vpc-internal-lb/, but only under https://kubeovn.github.io/docs/stable/en/advance/vip/
  2. Am I supposed to use an IP from the service-cluster-ip-range as Vip, or can it also be an IP from the internal VPC range? I initially tried with the internal VPC range, but I was even not able to access the VIP from within the VPC, so I switched to an IP from the service-cluster-ip-range
  3. Based off https://github.com/kubeovn/kube-ovn/issues/2636, I understand that I may be missing some routes, but I don't really know which ones
  4. I am currently using Iptables-*-based resources, but I have also seen documentation about Ovn-*-based resources (https://kubeovn.github.io/docs/stable/en/advance/ovn-eip-fip-snat/#4-ovn-dnat), so not sure which ones to use

I would be happy to provide further details if needed.

Thanks in advance for your help!

Steps To Reproduce

Please note:

Steps:

  1. Create the following resources:

    ---
apiVersion: kubeovn.io/v1
kind: Vip
metadata:
name: test-ovn-arp-proxy-vip-lb-http
spec:
namespace: test-ovn
subnet: test-ovn-internal-subnet
v4ip: 10.233.0.237
type: switch_lb_vip
---
apiVersion: kubeovn.io/v1
kind: SwitchLBRule
metadata:
name: test-ovn-incoming-lb-http
spec:
vip: 10.233.0.237
sessionAffinity: ClientIP
namespace: test-ovn
selector:
- app:nginx
ports:
- name: http
port: 80
targetPort: 80
protocol: TCP
---
kind: IptablesEIP
apiVersion: kubeovn.io/v1
metadata:
name: test-ovn-incoming-ip-http
spec:
natGwDp: test-ovn-gw
v4ip: 10.1.0.200
externalSubnet: ovn-vpc-external-network
---
kind: IptablesDnatRule
apiVersion: kubeovn.io/v1
metadata:
name: test-ovn-incoming-dnat-http
spec:
eip: test-ovn-incoming-ip-http
externalPort: '80'
internalIp: 10.233.0.237
internalPort: '80'
protocol: tcp

  2. Try to access the service from outside:

    $ curl -v http://10.1.0.200
*   Trying 10.1.0.200:80...
* connect to 10.1.0.200 port 80 failed: Connection timed out
* Failed to connect to 10.1.0.200 port 80 after 134683 ms: Connection timed out
* Closing connection 0
curl: (28) Failed to connect to 10.1.0.200 port 80 after 134683 ms: Connection timed out

  3. Access from a Pod within the VPC is working, through Service IP:

    $ curl http://10.233.0.237
Welcome to nginx!

Current Behavior

Access to VIP within a VPC through VPC Gateway / DNAT not working

Expected Behavior

Access working :-)

bobz965 commented 4 months ago

about this case: how about using loadbalancer instead of SwitchLBRule and DNAT? @hongzhen-ma @tgdfool2

tgdfool2 commented 4 months ago

about this case: how about using loadbalancer instead of SwitchLBRule and DNAT? @hongzhen-ma @tgdfool2

Any example of such a config?

tgdfool2 commented 4 months ago

@bobz965: you mean this I assume? https://kube-ovn.readthedocs.io/zh-cn/latest/en/guide/loadbalancer-service/

The issue is that it only supports services in the default VPC. In our case, we would have several VPCs on our cluster, and all of them would have services that need to reachable.

So maybe you have an idea on how to make DNAT rules to work? You mentioned some missing routes, but how can I find which ones?

bobz965 commented 4 months ago

@bobz965: you mean this I assume? https://kube-ovn.readthedocs.io/zh-cn/latest/en/guide/loadbalancer-service/

yes.

@hongzhen-ma can this lb support custom vpc?

tgdfool2 commented 4 months ago

In the end, this would be the setup that we would like to achieve:

image

Would be great if you could give us some hints on how to achieve this :-)

Thanks in advance for your support!

tgdfool2 commented 4 months ago

Other question: is this expected that the NAT Gateway is not able to ping the Pods in the same VPC?

In my picture above, from the NAT Gateway:

Just wondering if this is by design or if something is wrong...

tgdfool2 commented 4 months ago

Other question: is this expected that the NAT Gateway is not able to ping the Pods in the same VPC?

In my picture above, from the NAT Gateway:

  • Ping works to 100.105.1.100 (its own IP)
  • Ping doesn't work to 100.105.1.4 and 100.105.1.5 (IPs of other Pods in the VPC/Namespace)

Just wondering if this is by design or if something is wrong...

This is working in my kind setup, so I would need to investigate why this is failing in the kubespray one.

tgdfool2 commented 4 months ago

Regarding the missing routes with the SwitchLBRule setup, I did some further investigation, but sill with no luck. Here is my current setup:

image

With this setup:

  1. Access from all Pods, including VPC Gateway to Nginx Pod (100.105.0.2:80) is working
  2. Access from Test Pod Alpine (100.105.0.3) to SwitchLBRule VIP (10.96.0.237) is working
  3. Access from VPC Gateway to SwitchLBRule VIP (10.96.0.237) is NOT working

Remarks:

My assumption is, prior to configure the DNAT rule to allow external connections to the SwitchLBRule VIP, we need to ensure that the VPC Gateway can access the VIP. Is this correct? If so, then what I am missing to get this connection to work?

Thanks in advance for your support!

bobz965 commented 4 months ago

you can try use lb svc， if have any bug. please post .

I think it could be used in custom vpc.

bobz965 commented 4 months ago

Using switch lb rule with nat gw is much more complex than LB service.

tgdfool2 commented 4 months ago

Thanks. Will give a try and let you know the outcome.

tgdfool2 commented 4 months ago

@bobz965

The following Service has been deployed:

---
apiVersion: v1
kind: Service
metadata:
  annotations:
    lb-svc-attachment.kube-system.kubernetes.io/logical_switch: ovn-vpc-external-network
    ovn.kubernetes.io/attachmentprovider: ovn-vpc-external-network.kube-system
  name: test-clearml-ttr-svc
  namespace: test-clearml
spec:
  loadBalancerIP: 10.1.0.200
  ports:
  - name: http
    protocol: TCP
    port: 80
    targetPort: 80
  selector:
    app: nginx
  sessionAffinity: None
  type: LoadBalancer

Which resulted in the following error in the controller Pod:

W0712 12:56:28.091566       7 service.go:495] pod for lb svc test-ovn/test-ovn-ttr-svc is not running: pod of deployment test-ovn/lb-svc-test-ovn-ttr-svc not found
I0712 12:56:29.190111       7 service_lb.go:311] add eip rules for lb svc pod, [10.1.0.200/24,10.1.0.1]
E0712 12:56:32.295601       7 service_lb.go:271] failed to ExecuteCommandInContainer, errOutput: failed to exec command "arping -c 3 -s 10.1.0.200 10.1.0.1"
E0712 12:56:32.295834       7 service_lb.go:276] command terminated with exit code 1
E0712 12:56:32.295854       7 service_lb.go:313] failed to add eip for pod, err: command terminated with exit code 1
E0712 12:56:32.295868       7 service.go:529] failed to update pod attachment network for service test-ovn/test-ovn-ttr-svc: command terminated with exit code 1
E0712 12:56:32.295942       7 service.go:166] error syncing 'test-ovn/test-ovn-ttr-svc': command terminated with exit code 1, requeuing

Could this be related to the fact that this infrastructure is running in Azure, with ipvlan and not macvlan? Do you see a way to overcome this?

bobz965 commented 4 months ago

run arping -c 3 -s 10.1.0.200 10.1.0.1 manually in vpc nat gw pod, and see the err details.

tgdfool2 commented 4 months ago

Output:

# arping -c 3 -s 10.1.0.200 10.1.0.1
Interface "net1" is not ARPable
bobz965 commented 4 months ago

do you have iptables-eip?

bobz965 commented 4 months ago

do you have iptables-eip?

if you do not have eip, the net1 arp is off.

tgdfool2 commented 4 months ago

We do have an EIP, but we had to disable ARP on net1 due to the following limitation: https://github.com/kubeovn/kube-ovn/issues/4259#issuecomment-2211161528

tgdfool2 commented 4 months ago

And after re-enabling ARP on net1, we get the following:

# arping -c 3 -s 10.1.0.200 10.1.0.1
ARPING 10.1.0.1 from 10.1.0.200 net1
Sent 3 probes (3 broadcast(s))
Received 0 response(s)

(10.1.0.1 is the default gw of our Azure subnet)

bobz965 commented 4 months ago

if you has a iptables-eip, and net1 0.0.0.0 default routes is already exist, you should turn arp on

tgdfool2 commented 4 months ago

Yes, that's what we tried to do but unfortunately, as soon as ARP gets turned on, the NAT GW loses connectivity with the other Azure VMs in the same subnet. Probably something to do with Azure networking.

bobz965 commented 4 months ago

sorry, I'm not familiar with Azure.

github-actions[bot] commented 2 months ago

Issues go stale after 60d of inactivity. Please comment or re-open the issue if you are still interested in getting this issue fixed.

jcshare commented 2 months ago

In the end, this would be the setup that we would like to achieve: image

Would be great if you could give us some hints on how to achieve this :-)

Thanks in advance for your support!

Hi kube-ovn experts, it looks like the lb svc mechanism only support external user to access those SVC in the default VPC(see below link:).

https://kube-ovn.readthedocs.io/zh-cn/latest/en/guide/loadbalancer-service/
...
" It supports creating LoadBalancer type Services in the default VPC, allowing access to Services in the default VPC through LoadBalancerIP."
...

how external user to access the SVC which created in their private VPC?

can we support it?

many thanks

bobz965 commented 2 months ago

the lb svc is switch lb rule, and the switch is isolated by vpc.

so the lb svc is isolated by vpc.

jcshare commented 2 months ago

the lb svc is switch lb rule, and the switch is isolated by vpc.

so the lb svc is isolated by vpc.

yes, can we have any mechanism to expose these types of SVC (defined in user private VPC) to public network? we do have such a requirement.

I found a workaround as below and it works, but it may produce other problem. can you experts have take a look and see if we can support it?

# --------------  Workaround for exposing LB VIP to outside  -----------------------------
# step-1> Create SNAT rule for the subnet in VPC
#
cat <<EOF > snat.yaml
kind: IptablesSnatRule
apiVersion: kubeovn.io/v1
metadata:
  name: tanant1-pro1-vpc1-net1-snat
spec:
  eip: tanant1-pro1-vpc1-net1-eip    # the eip's name which get created in step-6
  internalCIDR: 172.22.1.0/24        # the CIDR of the subnet
EOF
kubectl apply -f snat.yaml

# step-2> Create SLR  in VPC
#
cat <<EOF > loadbalancer.yaml
apiVersion: kubeovn.io/v1
kind: SwitchLBRule
metadata:
  name:  webapp-lb-vpc1
spec:
  vip: 169.192.1.1            # VIP assigned to the LB
  sessionAffinity: ClientIP
  namespace: ns1              # namespace name
  selector:
    - app.kubernetes.io/name:webtest   # keep consistent with the label on POD
  ports:
  - name: svc
    port: 8080                # LB service port that will be exposed to outside
    targetPort: 80            # internal service port
    protocol: TCP             # internal service protocol
EOF
kubectl apply -f loadbalancer.yaml

# step-3> Create Dnat rule for the SLR
#
# kubectl get slr
# NAME             VIP           PORT(S)    SERVICE                  AGE
# webapp-lb-vpc1   169.192.1.1   8080/TCP   ns1/slr-webapp-lb-vpc1   40h
#
cat <<EOF > dnat-lb.yaml
kind: IptablesDnatRule
apiVersion: kubeovn.io/v1
metadata:
  name: dnat-for-lb
spec:
  eip: eip-vpc-1            # 
  externalPort: '9999'      # the port to be exposed
  internalIp: 169.192.1.1   # the SLR's IP address as above
  internalPort: '8080'      # the SLR's port as above
  protocol: tcp             # the SLR's protocoal as above
EOF
kubectl apply -f dnat-lb.yaml

#step-4> Login into the vpc nat gateway pod and update route rules as below:
#4.1> configure the corresponding Logical route ports IP as the default gateway 
    ip r r default via <LRP_IP> dev eth0 onlink

# the LRP's ip can be obtained by below CMD:
#
# kubectl ko nbctl show vpc2
# router 5d3d3181-26c5-4100-bdb1-4e6c54d659d3 (vpc2)
#     port vpc2-net1-vpc2
#         mac: "00:00:00:EE:62:24"
#         networks: ["172.22.1.1/24"] <<<<<<<< this one, LRP_IP

#4.2> Add route rule for external network
   ip r r <external_net_cidr> via <external_net_gw_ip> dev net1
bobz965 commented 2 months ago

there is a loadbalancer svc mechanism in kube-ovn, you can try.

jcshare commented 2 months ago

there is a loadbalancer svc mechanism in kube-ovn, you can try.

really appreciate your quick reply. actually, I have tried it and it works. but it need manually updated the VPC GW and it not convenient. what I mean is that can we add the support on kube-ovn side?

bobz965 commented 2 months ago

do you mean in ovn way instead of in the way of vpc nat gw?

jcshare commented 2 months ago

do you mean in ovn way instead of in the way of vpc nat gw?

each way should be OK to support exposing inside SVC to public network in my opinion, :)

bobz965 commented 2 months ago

ovn dnat is based on ovn router lb, you can try to make a lb feature like ovn dnat based on ovn router lb.

jcshare commented 2 months ago

ovn dnat is based on ovn router lb, you can try to make a lb feature like ovn dnat based on ovn router lb.

thanks, as kube-ovn's code is very complicate, could you kubeovn experts do us a favor?

bobz965 commented 2 months ago

ovn dnat is based on ovn router lb, you can try to make a lb feature like ovn dnat based on ovn router lb.

thanks, as kube-ovn's code is very complicate, could you kubeovn experts do us a favor?

next month, maybe I will have time to establish a ovn lb svc

github-actions[bot] commented 1 week ago

Issues go stale after 60d of inactivity. Please comment or re-open the issue if you are still interested in getting this issue fixed.