Egress and ServiceExternalIP can not works together for the same External IP

ikandars commented 2 years ago

Describe the bug

If we create Egress object where we defined egressIP for example 103.15.226.42, we can't use that IP in Service LoadBalancer type in loadBalancerIP key. The status will be pending.

To Reproduce

create external IP Pool object:

apiVersion: crd.antrea.io/v1alpha2
kind: ExternalIPPool
metadata:
  name: prod-external-ip-pool
spec:
  ipRanges:
  - start: 103.15.226.41
    end: 103.15.226.42
  nodeSelector:
    matchLabels:
      kubernetes.io/hostname: awid5

Create Egress object:

kind: Egress
metadata:
  name: egress-prod-web
spec:
  appliedTo:
    namespaceSelector:
      matchLabels:
        kubernetes.io/metadata.name: default
    podSelector:
      matchLabels:
        app: network-tools
  egressIP: 103.15.226.42
  externalIPPool: prod-external-ip-pool

Create a pod:

apiVersion: v1
kind: Pod
metadata:
  name: network-tools
  labels:
    app: network-tools
spec:
  containers:
  - name: network-multitool
    image: praqma/network-multitool:d57af61
    resources:
      limits:
        memory: "200Mi"
        cpu: "0.3"
      requests:
        memory: "100Mi"
        cpu: "0.2"

Create service object:

kind: Service
metadata:
  name: network-tools
  annotations:
    service.antrea.io/external-ip-pool: prod-external-ip-pool
spec:
  ports:
  - port: 80
    targetPort: 80
  selector:
    app: network-tools
  type: LoadBalancer
  loadBalancerIP: 103.15.226.42

Check service:

kubectl get svc
NAME            TYPE           CLUSTER-IP    EXTERNAL-IP   PORT(S)        AGE
kubernetes      ClusterIP      10.1.0.1      <none>        443/TCP        5d15h
network-tools   LoadBalancer   10.1.108.67   <pending>     80:30207/TCP   2m23s

Expected

We can use Egress and ServiceExternalIP for the same Ip at the same time.

Actual behavior

Egress and ServiceExternalIP can not works to together for the same External IP

Versions:

$ antctl version
antctlVersion: v1.5.0
controllerVersion: v1.5.0

kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.3", GitCommit:"816c97ab8cff8a1c72eccca1026f7820e93e0d25", GitTreeState:"clean", BuildDate:"2022-01-25T21:24:08Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"linux/amd64"}

sudo containerd --version
containerd github.com/containerd/containerd v1.5.9 1407cab509ff0d96baa4f0eb6ff9980270e6e620

sudo uname -r
5.4.0-99-generic

antoninbas commented 2 years ago

We can use Egress and ServiceExternalIP for the same Ip at the same time.

Can you clarify why this is the expected behavior for you? What's the use case for having the same IP for Egress traffic and for a LoadBalancer Service?

IMO, what you are observing would be the correct behavior since you are trying to use the same IP from the same pool for 2 different things. But we should try to find a way to report an error to the user (if this is not already the case).

ikandars commented 2 years ago

So, there is a case where a workload use both public IP for in ingress and egress. In my case, I use KubeVirt to manage Virtual Machine. Another example is, deploying smtp server in a pod, where the public IP should consistent.

So, if this behavior expected, then I should change the label to feature request then.

jsalatiel commented 2 years ago

This would be really great for containerized postfix.