search

kubeovn / kube-ovn

A Bridge between SDN and Cloud Native (Project under CNCF)
https://kubeovn.github.io/docs/stable/en/
Apache License 2.0
1.87k stars 433 forks source link

[BUG][KIND] Pod inside Custom VPC+NatGateway cannot access Internet using SNAT #4241

Closed reski-rukmantiyo closed 5 days ago

reski-rukmantiyo commented 5 days ago

Kube-OVN Version

v1.12.12

Kubernetes Version

Client Version: v1.30.2 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.30.0

Operation-system/Kernel Version

"Ubuntu 22.04.4 LTS" 5.15.0-113-generic

Description

Link from #4117

But this is using KIND with 2 nodes

ubuntu@devspace-vm:~/operator/$ kind get nodes
kind-control-plane
kind-worker

Pod can reach gateway

ubuntu@devspace-vm:~/operator$ k exec -it pod/vpc1-pod -n ns1 -- /bin/sh
/ # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
47: eth0@if48: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1400 qdisc noqueue state UP 
    link/ether e6:8f:14:94:cc:b1 brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.2/24 brd 10.0.1.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::e48f:14ff:fe94:ccb1/64 scope link 
       valid_lft forever preferred_lft forever
/ # ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
^C
--- 8.8.8.8 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
/ # ping 10.0.1.1
PING 10.0.1.1 (10.0.1.1): 56 data bytes
64 bytes from 10.0.1.1: seq=0 ttl=254 time=1.258 ms
64 bytes from 10.0.1.1: seq=1 ttl=254 time=0.821 ms
64 bytes from 10.0.1.1: seq=2 ttl=254 time=0.710 ms
64 bytes from 10.0.1.1: seq=3 ttl=254 time=0.675 ms
^C
--- 10.0.1.1 ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 0.675/0.866/1.258 ms

Steps To Reproduce

  1. kind get nodes

    ubuntu@devspace-vm:~/operator/$ kind get nodes
kind-control-plane
kind-worker

  2. docker network inspect bridge

    ubuntu@devspace-vm:~/operator/$ docker network inspect bridge
[
{
    "Name": "bridge",
    "Id": "ce6aac7125e0a06f4a133101d13832223ee5427bcf4cee8b0f096baef7b63fbb",
    "Created": "2024-06-28T03:45:20.665647314Z",
    "Scope": "local",
    "Driver": "bridge",
    "EnableIPv6": false,
    "IPAM": {
        "Driver": "default",
        "Options": null,
        "Config": [
            {
                "Subnet": "172.17.0.0/16",
                "Gateway": "172.17.0.1"
            }
        ]
    },
    "Internal": false,
    "Attachable": false,
    "Ingress": false,
    "ConfigFrom": {
        "Network": ""
    },
    "ConfigOnly": false,
    "Containers": {
        "7b526e28ed713a39b8f4c8175f99d66fc719ef8df5364ffba372c9ebc8645293": {
            "Name": "kind-control-plane",
            "EndpointID": "4740b93f0f6e9862a735b533801c121ec6bbd5b13f4b057959657e7adf9a083b",
            "MacAddress": "02:42:ac:11:00:02",
            "IPv4Address": "172.17.0.2/16",
            "IPv6Address": ""
        },
        "d8d6b095e62abf5df37b630a3032d4de5c664928280125613b1ed8cc487d3038": {
            "Name": "kind-worker",
            "EndpointID": "2fb285b0e4177ddeade7cda7d85daf9e6a58edf171d564c5e42af1fd0aa5ec5e",
            "MacAddress": "02:42:ac:11:00:03",
            "IPv4Address": "172.17.0.3/16",
            "IPv6Address": ""
        }
    },
    "Options": {
        "com.docker.network.bridge.default_bridge": "true",
        "com.docker.network.bridge.enable_icc": "true",
        "com.docker.network.bridge.enable_ip_masquerade": "true",
        "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
        "com.docker.network.bridge.name": "docker0",
        "com.docker.network.driver.mtu": "1500"
    },
    "Labels": {}
}
]

    I've seen subnet 172.17.0.0/16 with gateway 172.17.0.1

  3. Apply into subnet

apiVersion: kubeovn.io/v1
kind: Subnet
metadata:
  name: ovn-vpc-external-network
spec:
  protocol: IPv4
  provider: ovn-vpc-external-network.kube-system
  cidrBlock: 172.17.0.0/16
  gateway: 172.17.0.1  # IP address of the physical gateway
  excludeIps:
  - 172.17.0.1..172.17.0.50
  1. Try again
ubuntu@devspace-vm:~/operator$ k exec -it pod/vpc1-pod -n ns1 -- ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
^C
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss
command terminated with exit code 1

Additional configuration

kind: Vpc
apiVersion: kubeovn.io/v1
metadata:
  name: test-vpc-1
spec:
  namespaces:
  - ns1
---
kind: Subnet
apiVersion: kubeovn.io/v1
metadata:
  name: net1
spec:
  vpc: test-vpc-1
  cidrBlock: 10.0.1.0/24
  protocol: IPv4
  namespaces:
    - ns1
  excludeIps:
  - 10.0.1.1
  gateway: 10.0.1.1
  gatewayType: distributed
  natOutgoing: false
---
apiVersion: v1
kind: Pod
metadata:
  namespace: ns1
  name: vpc1-pod
  labels:
    app: vpc1-pod
spec:
  containers:
    - name: vpc1-pod
      image: docker.io/library/nginx:alpine
---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: ovn-vpc-external-network
  namespace: kube-system
spec:
  config: '{
      "cniVersion": "0.3.0",
      "type": "macvlan",
      "master": "eth0",
      "mode": "bridge",
      "ipam": {
        "type": "kube-ovn",
        "server_socket": "/run/openvswitch/kube-ovn-daemon.sock",
        "provider": "ovn-vpc-external-network.kube-system"
      }
    }'
---
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: ovn-vpc-nat-config
  namespace: kube-system
data:
  image: docker.io/kubeovn/vpc-nat-gateway:v1.12.18
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: ovn-vpc-nat-gw-config
  namespace: kube-system
data:
  enable-vpc-nat-gw: 'true'
---
kind: VpcNatGateway
apiVersion: kubeovn.io/v1
metadata:
  name: gw1
spec:
  vpc: test-vpc-1
  subnet: net1
  lanIp: 10.0.1.254
  selector:
    - "kubernetes.io/hostname: kind-worker"
    - "kubernetes.io/os: linux"
  externalSubnets:
    - ovn-vpc-external-network
  ---
kind: IptablesEIP
apiVersion: kubeovn.io/v1
metadata:
  name: eip-random
spec:
  natGwDp: gw1
  externalSubnet: ovn-vpc-external-network
---
kind: IptablesSnatRule
apiVersion: kubeovn.io/v1
metadata:
  name: snat01
spec:
  eip: eips01
  internalCIDR: 10.0.1.0/24

Current Behavior

Dont have internet connectivity inside the pod

ubuntu@devspace-vm:~/operator$ k exec -it pod/vpc1-pod -n ns1 -- ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
^C
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss
command terminated with exit code 1

Expected Behavior

Internet can be used inside Pod with VPC + NatGateway

zhangzujian commented 5 days ago 
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: ovn-vpc-external-network
  namespace: kube-system
spec:
  config: '{
      "cniVersion": "0.3.0",
      "type": "macvlan",
      "master": "eth0",
      "mode": "bridge",
      "ipam": {
        "type": "kube-ovn",
        "server_socket": "/run/openvswitch/kube-ovn-daemon.sock",
        "provider": "ovn-vpc-external-network.kube-system"
      }
    }'

The macvlan master should be eth1. If you are using eth0, the subnet CIDR should be subnet of the kind network.

reski-rukmantiyo commented 5 days ago

After I apply again still dont work

ubuntu@devspace-vm:~/operator$ k exec -it pod/vpc1-pod -n ns1 -- ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
^C
--- 8.8.8.8 ping statistics ---
52 packets transmitted, 0 packets received, 100% packet loss
command terminated with exit code 1

Configuration

Docker inspect bridge

ubuntu@devspace-vm:~/operator/$ docker network inspect bridge
[
    {
        "Name": "bridge",
        "Id": "ce6aac7125e0a06f4a133101d13832223ee5427bcf4cee8b0f096baef7b63fbb",
        "Created": "2024-06-28T03:45:20.665647314Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.17.0.0/16",
                    "Gateway": "172.17.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "7b526e28ed713a39b8f4c8175f99d66fc719ef8df5364ffba372c9ebc8645293": {
                "Name": "kind-control-plane",
                "EndpointID": "4740b93f0f6e9862a735b533801c121ec6bbd5b13f4b057959657e7adf9a083b",
                "MacAddress": "02:42:ac:11:00:02",
                "IPv4Address": "172.17.0.2/16",
                "IPv6Address": ""
            },
            "d8d6b095e62abf5df37b630a3032d4de5c664928280125613b1ed8cc487d3038": {
                "Name": "kind-worker",
                "EndpointID": "2fb285b0e4177ddeade7cda7d85daf9e6a58edf171d564c5e42af1fd0aa5ec5e",
                "MacAddress": "02:42:ac:11:00:03",
                "IPv4Address": "172.17.0.3/16",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
        "Labels": {}
    }
]

The rest of config

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: ovn-vpc-external-network
  namespace: kube-system
spec:
  config: '{
      "cniVersion": "0.3.0",
      "type": "macvlan",
      "master": "eth1",
      "mode": "bridge",
      "ipam": {
        "type": "kube-ovn",
        "server_socket": "/run/openvswitch/kube-ovn-daemon.sock",
        "provider": "ovn-vpc-external-network.kube-system"
      }
    }'
---
apiVersion: kubeovn.io/v1
kind: Subnet
metadata:
  name: ovn-vpc-external-network
spec:
  protocol: IPv4
  provider: ovn-vpc-external-network.kube-system
  cidrBlock: 172.17.0.0/16
  gateway: 172.17.0.1  # IP address of the physical gateway
  excludeIps:
  - 172.17.0.1..172.17.0.50

Ovn

ubuntu@devspace-vm:~/operator/$ kubectl ko nbctl show
switch 4044ac38-16c4-4b17-9571-2bdbafa1e136 (ovn-default)
    port tcpdump-7764d7cbcb-4t4fw.default
        addresses: ["e2:ef:c1:c2:bd:ad 10.244.0.22"]
    port virt-api-75859b7b7-lh2ds.kubevirt
        addresses: ["ba:83:72:7c:db:98 10.244.0.18"]
    port virt-api-75859b7b7-gxvgt.kubevirt
        addresses: ["02:60:f8:e4:a7:51 10.244.0.17"]
    port virt-controller-6855b4df79-q8v6g.kubevirt
        addresses: ["ea:82:88:a7:3b:b4 10.244.0.20"]
    port cdi-deployment-fb59bcc87-gnzqj.cdi
        addresses: ["9a:fe:3f:e0:a3:96 10.244.0.14"]
    port local-path-provisioner-988d74bc-nzmq9.local-path-storage
        addresses: ["ce:d8:e1:f6:c9:0d 10.244.0.8"]
    port virt-controller-6855b4df79-6fcmj.kubevirt
        addresses: ["96:9f:62:3f:51:52 10.244.0.19"]
    port ovn-default-ovn-cluster
        type: router
        router-port: ovn-cluster-ovn-default
    port cdi-operator-595bfb44cd-gprzb.cdi
        addresses: ["82:58:02:a2:05:97 10.244.0.12"]
    port virt-operator-56d79bb8bd-cphv8.kubevirt
        addresses: ["62:e5:0e:54:49:a8 10.244.0.10"]
    port cdi-uploadproxy-7657d8d89d-j692z.cdi
        addresses: ["86:8c:78:df:9e:48 10.244.0.15"]
    port virt-operator-56d79bb8bd-dmzk8.kubevirt
        addresses: ["f2:42:45:8b:bd:7b 10.244.0.11"]
    port coredns-7db6d8ff4d-z4x82.kube-system
        addresses: ["a6:b6:5f:e5:87:b6 10.244.0.7"]
    port coredns-7db6d8ff4d-xxw96.kube-system
        addresses: ["de:61:3b:99:c2:d3 10.244.0.6"]
    port kube-ovn-pinger-dsftk.kube-system
        addresses: ["16:c8:2d:44:72:b8 10.244.0.9"]
    port virt-handler-gw99z.kubevirt
        addresses: ["ca:b2:93:b4:bb:f5 10.244.0.21"]
    port cdi-apiserver-5d565ddb6-2rs4n.cdi
        addresses: ["d6:92:21:fc:b6:ee 10.244.0.13"]
    port busybox.default
        addresses: ["2a:4e:bd:f4:56:2a 10.244.0.23"]
switch 6d14d936-f2c8-465d-a4f2-dec8dff13d98 (join)
    port node-kind-worker
        addresses: ["fe:d3:a3:76:ff:7c 100.64.0.2"]
    port node-kind-control-plane
        addresses: ["de:7d:78:3d:b8:67 100.64.0.3"]
    port join-ovn-cluster
        type: router
        router-port: ovn-cluster-join
switch a97298c2-b3bf-432f-8934-08658395f3b8 (net1)
    port vpc-nat-gw-gw1-0.kube-system
        addresses: ["c6:80:36:8f:b6:69 10.0.1.254"]
    port net1-test-vpc-1
        type: router
        router-port: test-vpc-1-net1
    port vpc1-pod.ns1
        addresses: ["e6:8f:14:94:cc:b1 10.0.1.2"]
router bbb88850-345f-426c-863a-491a8d958b7b (test-vpc-1)
    port test-vpc-1-net1
        mac: "ae:6b:0f:64:ff:c9"
        networks: ["10.0.1.1/24"]
router 47effcd4-820b-4328-92d7-7c99cb64eb13 (ovn-cluster)
    port ovn-cluster-join
        mac: "e2:bc:e1:54:d6:86"
        networks: ["100.64.0.1/16"]
    port ovn-cluster-ovn-default
        mac: "ce:97:4c:ce:77:17"
        networks: ["10.244.0.1/16"]
---
ubuntu@devspace-vm:~/operator/$ kubectl ko nbctl lr-route-list test-vpc-1
ubuntu@devspace-vm:~/operator/$
zhangzujian commented 5 days ago

@reski-rukmantiyo The logical router static route is missing, you can add it in the vpc spec:

kind: Vpc
apiVersion: kubeovn.io/v1
metadata:
  name: vpc1
spec:
  staticRoutes:
    - cidr: 0.0.0.0/0
      nextHopIP: 10.0.1.254
      policy: policyDst

@bobz965 Is this a bug or by design?

reski-rukmantiyo commented 5 days ago

Thanks @zhangzujian .... I think this is the last pieces in terms of using Custom VPC + NatGateway in KIND Maybe you should emphasize in Docs

kind: Vpc
apiVersion: kubeovn.io/v1
metadata:
  name: test-vpc-1
spec:
  namespaces:
  - ns1
  staticRoutes:
  - cidr: 0.0.0.0/0
    nextHopIP: 10.0.1.254
    policy: policyDst

Next i'll try related with DNS. Thanks

bobz965 commented 5 days ago

@reski-rukmantiyo The logical router static route is missing, you can add it in the vpc spec:

kind: Vpc
apiVersion: kubeovn.io/v1
metadata:
  name: vpc1
spec:
  staticRoutes:
    - cidr: 0.0.0.0/0
      nextHopIP: 10.0.1.254
      policy: policyDst

@bobz965 Is this a bug or by design?

It is by design. the docs contain this, but not emphasized.

bobz965 commented 5 days ago

https://kubeovn.github.io/docs/v1.13.x/guide/vpc/#_3

reski-rukmantiyo commented 5 days ago

Hi @bobz965 , thanks for this. I've succesful installed and use this. Maybe it's better to put under creating custom vpc - custom vpc under 1 tree. While custom dns, since it's different topics, put in different tree. but it's all in 1 config vpc page...my 2 cents

reski-rukmantiyo commented 5 days ago

Let me close this. Thanks @zhangzujian @bobz965

bobz965 commented 4 days ago

Hi @bobz965 , thanks for this. I've succesful installed and use this. Maybe it's better to put under creating custom vpc - custom vpc under 1 tree. While custom dns, since it's different topics, put in different tree. but it's all in 1 config vpc page...my 2 cents

ok, i will try!