How to make the Node use EGRESS IP to access the internet

[yeshl]

i know pods can use egress ip，but How to make the nodes use egress ip to access the internet？

[rajnkamr]

Usually if egress ip is in node's subnet and node ip is accessible externally, internet access via egress ip would work seamlessly, tht's true for pod, For nodes currently it is not possible, Could you provide more details of the set up u r having and use case !

[tnqn]

If you mean making arbitrary Nodes access internet using specific Egress IP like Pods, currently this can't be achieved by configuring Egress API.

But after a quick thought, it should be possible to apply Egress to Node's traffic, though not sure if it can be implemented in near future. Could you share more information about how it helps your use case, which may help prioritize it in the community?

For now, maybe you could consider setting up a proxy in a Pod which has Egress applied, then configure Nodes to use that proxy?

[yeshl]

Accessing the external network usually requires NAT gateway or a router. I hope to use Kubernetes to deploy a highly available SNAT gateway (or router) as the default gateway for all nodes. all nodes can set the default gateway to this egress ip！This IP can be set up through the network policy to only allow outbound traffic and prohibit inbound traffic！

[rajnkamr]

currently supported egress feature like externalippool is specific to pod traffic, gateway ip can be provided for the egress nodeSelector: matchLabels: network-role: egress-gateway cc @tnqn can add more

[yeshl]

Could you give the example yaml ， thank u so much！

[rajnkamr]

Yes, It is possible with currently supported egress feature like externalippool , gateway ip can be provided for the egress nodeSelector: matchLabels: network-role: egress-gateway cc @tnqn can add more

This is applicable for pod level and not node level use case as mentioned by @tnqn

[yeshl]

setting up a proxy in a Pod which has Egress applied, then configure Nodes to use that proxy

it‘s a good idea，but i have no idea how to do, could you please tell me？@tnqn

[antoninbas]

It depends on what your needs are. If you just need an HTTP proxy, it's pretty straightforward.

I was able to get it working in a Kind cluster in a few minutes:

# Create Kind cluster. This Kind config file is available in the Antrea repo
$ kind create cluster --config ci/kind/config-3nodes.yml
# Install Antrea
$ helm install -n kube-system antrea antrea/antrea
# Run HTTP proxy as a Pod, there are many other options available out there. 
$ kubectl apply -f https://raw.githubusercontent.com/kalaksi/docker-tinyproxy/master/kubernetes/deployment.yml
# Create ExternalIPPool and Egress (see below)
$ kubectl apply -f egress.md
# Check Egress
$ kubectl get egress
NAME               EGRESSIP      AGE   NODE
egress-for-proxy   172.18.0.90   23m   kind-control-plane
# Exec into a Node to validate that everything works
$ docker exec -ti kind-worker bash

**********

# First HTTP:
root@kind-worker:/# http_proxy=10.10.2.2:8888 curl -v http://www.google.com > /dev/null
* Uses proxy env variable http_proxy == '10.10.2.2:8888'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 10.10.2.2:8888...
* Connected to 10.10.2.2 (10.10.2.2) port 8888 (#0)
> GET http://www.google.com/ HTTP/1.1
> Host: www.google.com
> User-Agent: curl/7.88.1
> Accept: */*
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 OK
# ...

**********

# Now HTTPs:
root@kind-worker:/# https_proxy=10.10.2.2:8888 curl -v https://www.google.com > /dev/null
* Uses proxy env variable https_proxy == '10.10.2.2:8888'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 10.10.2.2:8888...
* Connected to 10.10.2.2 (10.10.2.2) port 8888 (#0)
* allocate connect buffer
* Establish HTTP proxy tunnel to www.google.com:443
> CONNECT www.google.com:443 HTTP/1.1
> Host: www.google.com:443
> User-Agent: curl/7.88.1
> Proxy-Connection: Keep-Alive
>
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0< HTTP/1.1 200 Connection established
# ...

I validated that the Egress IP was used correctly when exiting the cluster, using traffic capture (tcpdump).

And here are my ExternalIPPool and Egress resources:

apiVersion: crd.antrea.io/v1beta1
kind: ExternalIPPool
metadata:
  name: external-ippool-for-proxy
spec:
  ipRanges:
  - start: 172.18.0.90
    end: 172.18.0.100
  nodeSelector: {}
---
apiVersion: crd.antrea.io/v1beta1
kind: Egress
metadata:
  name: egress-for-proxy
spec:
  appliedTo:
    namespaceSelector:
      matchLabels:
        kubernetes.io/metadata.name: default
    podSelector:
      matchLabels:
        io.kompose.service: tinyproxy
  externalIPPool: external-ippool-for-proxy

The IP range I use for the ExternalIPPool is not random of course. It matches the subnet allocated by Kind / docker for my Nodes, so that reply traffic can be routed correctly. That's not specific to your question though, that's how the Egress feature works.

If an HTTP / HTTPS proxy is not enough for you, you will need to look into a SOCKS proxy.

[reski-rukmantiyo]

Somehow cannot make this thing works....especially when working with KIND. any guidance for this?

[antoninbas]

@reski-rukmantiyo Please double check that you used a valid IP range for the ExternalIPPool. As I pointed out above, it has to match the subnet used by the K8s Nodes.

You can also check the Egress resource with kubectl get egress -o yaml to check if an EgressIP was allocated correctly, and check that the Egress was assigned to a specific Node.

especially when working with KIND

What do you mean by "especially"? If you need some specific assistance, please share more details and indicate what exactly is not working for you. Our Slack channel may be a more appropriate place for some interactive help.

[reski-rukmantiyo]

Noted @antoninbas

I just want to make egress works when running KIND. Because already implement egress, but still dont works

reski@WIN10-X300-RES:~/kerjaan/multus-cni$ k get egress
NAME             EGRESSIP   AGE   NODE
egress-default              23m

[antoninbas]

@reski-rukmantiyo This seems unrelated to this specific Github issue, which is about using the Egress IP for Node traffic. You seem to have a general query about Egress. A new issue would be more appropriate in your case, or you can talk to us on Slack. In any case, we will need a lot more details than that in order to assist you. For example, what's the definition of the resources (Egress, ExternalIPPool) that are you creating? The snippet you are sharing here shows that no Egress IP has been allocated and that no Egress Node has been elected, so you may just have invalid resource definitions (for example, and ExternalIPPool that does not select any Node). Please refer to the Egress documentation / Youtube video (https://www.youtube.com/watch?v=rAY76lG4VLA), and make sure you have a good understanding of the feature.

I also see a reference to Multus in your code snippet, is that relevant to your issue?

[reski-rukmantiyo]

Just realized already out of topics. let me continue on slack. thanks