[Feature Request] Use pod as subnet gateway

[zsxsoft]

Description

Consider the following scenario:

The cluster can only access the public network. There is an OpenVPN server on the public network that allows connection to the enterprise's internal network. Currently, the common solution is to deploy a Sidecar Container for any Pod that needs to connect to the internal network. However, by default, an OpenVPN client can only have one connection. If multiple Pods need to access the server, sidecar container cannot solve this problem.

Therefore, one possible solution is to deploy an OpenVPN Pod and modify the VPC's policy routing to allow the specified subnet/pod/ip to access the internal network through this Pod.

The problem is, if the subnet that needs to access the internal network / the VPN pod frequently changes, maintaining the policy routing could become challenging(It's array not a map, cannot easily patch). In the ovn-cluster VPC, due to the presence of policy routes, static routing + routing table seems ineffective.

Hence, I would like to propose adding a new field, gatewayIp or gatewayPod, based on the Subnet's gatewayType: centralized. The current gatewayNode field and gatewayType: distributedare only suitable for scenarios where the gateway is outside the cluster, they suffer from performance issues when forwarding to the cluster's internal network.

I hope this can be taken into consideration.

Who will benefit from this feature?

No response

Anything else?

No response

[bobz965]

I made a openvpn solution based on kube-ovn，could you please try this project？ https://github.com/kubecombo/kube-combo

[bobz965]

In my opinion, I think fip:vip is a good choice. no need to handle the vpc routes details.

[zsxsoft]

@bobz965 I think FIP/VIP is better. kube-combo can't meet my needs, I don't have underlay/eip/fip environment.

[bobz965]

is your OpenVPN server inside the kube-ovn vpc ?

[zsxsoft]

No, server is in external

[bobz965]

you can try set up a openvpn client gw pod in the ovn-cluster vpc, and set ovn-cluster vpc spec policy based routes to dest cidr via the openvpn client gw pod.

[zsxsoft]

Yes I did that.

My point is, if the subnet that needs to access the internal network / the VPN pod frequently changes, maintaining the policy routing could become challenging(It's array not a map, cannot easily patch).

[bobz965]

the internal network / the VPN pod frequently changes, but you can keep the openvpn client gw pod use the static IP. policy routing controls the traffic via the gw pod.

pod --> gw-openvpn-client-pod -->OpenVPN server

policy routing only controls: pod --> gw-openvpn-client-pod

[zsxsoft]

For example

kind: Vpc
apiVersion: kubeovn.io/v1
metadata:
  name: test-vpc-1
spec:
  policyRoutes:
    - action: reroute
      match: ip4.src==10.0.1.0/24
      nextHopIP: 10.0.1.252
      priority: 10

Add a new subnet:

kind: Vpc
apiVersion: kubeovn.io/v1
metadata:
  name: test-vpc-1
spec:
  policyRoutes:
    - action: reroute
      match: ip4.src==10.0.1.0/24 || ip4.src == 10.0.2.0/24 || ip4.src == 10.0.3.0/24
      nextHopIP: 10.0.1.252
      priority: 10

It's hard to maintain.

And this

kind: Vpc
apiVersion: kubeovn.io/v1
metadata:
  name: test-vpc-1
spec:
  policyRoutes:
    - action: reroute
      match: ip4.src==10.0.1.0/24 
      nextHopIP: 10.0.1.252
      priority: 10
    - action: reroute
      match: ip4.src==10.0.2.0/24 
      nextHopIP: 10.0.1.252
      priority: 10
    - action: reroute
      match: ip4.src==10.0.2.0/24 
      nextHopIP: 10.0.1.252
      priority: 10

It's easy to add(PATCH {"path": "/spec/policyRoutes/-", op: "add"}) but hard to delete (need a mutex, then loop to get all policyRoutes to find the index, then delete)

[bobz965]

why not use a bigger cidr at first? I do not think user should add or delete a subnet frequently.

if a user create a subnet, and put its apps in it. it will run for a long time(at least half or one year).

If the user OpenVPN server is in cloud vpc, I suggest the user to choose a bigger cidr subnet.

even if user use 10.0.1.0/24, 10.0.2.0/24 , 10.0.x.0/24, you can also just add one 10.0.0.0/16 routes.

[zsxsoft]

My project is quite similar to AWS Lambda, and as an infrastructure provider, I don't have control workloads running on. Different workloads correspond to different egresses, and these egresses are also frequently changing. So I need a more universal solution rather than a workaround.

Currently, I'm doing this as my workaround:

return command("kubectl", "ko", "nbctl", "lr-policy-add", VpcCluster, EgressPriority, fmt.Sprintf("ip4.src == %s", net), "reroute", rerouteIp)

[bobz965]

About your background: There is an OpenVPN server on the public network that allows connection to the enterprise's internal network.

if your each kube-ovn vpc only has one OpenVPN client gw pod, and if the kube-ovn vpc only has one subnet(eg: 192.168.1.0/24), how about using "!192.168.1.0/24 via OpenVPN client gw pod IP" ?

your OpenVPN server side subnet may changed, but , is your client kube-ovn side vpc subnet not changed?

if your client kube-ovn side vpc subnet not change, you can use ! xxx policy routes.

[zsxsoft]

I have only 1 VPC, multiple and frequently changing gateway pods. My workloads can communicate with each other by whitelist, so I don’t need VPC to increase development difficulty.

[zsxsoft]

https://docs.cilium.io/en/latest/network/kubernetes/local-redirect-policy/ I didn't use Cilium, but it seems good

[zsxsoft]

https://github.com/kubeovn/kube-ovn/commit/17fe230273fdac9a979f2c3e523b5940f42e1286 This uses the Linux networking stack for policy routing, which does not meet my needs. Node and Pod are not reachable at Layer 2. I'm trying implement a OVN-based policy routing using the same field.