loadBalancerSourceRanges not supported in AntreaProxy

[jayunit100]

Describe the bug

As of now antrea doesnt support loadBalancerSourceRanges in the antreaProxy. Since this is also not supported in the kube-proxy, we have no way to provdide this to Windows users.

To Reproduce

Similar to https://github.com/kubernetes/kubernetes/issues/120033 ...

Expected

AntreaProxy would fully support the Kubernetes service spec...

Actual behavior

LoadbalancerSourceRanges that are outside of an packet's IP are allowed into antrea clusters where antreaproxy is used.

Note this isnt a HUGE bug b/c alas, even the windows service proxy doesnt yet implement this

QUESTION:

Could this be done by reusing HNS packet filtering from the HNS ACLs ? Or does it require OVS?

aclPolicy := hns.ACLPolicy{
    Type:            hns.ACL,
    Action:          hns.Block,
    Direction:       hns.In,
    RemoteAddresses: loadBalancerSourceRanges
}
```?

[tnqn]

@hongliangl @wenyingd could you please take a look from Linux and Windows perspectives?

[hongliangl]

After discussing @wenyingd, we have two designs.

Take an example of two LoadBalancer Services

• Service A, ingress Ips are 192.168.77.100 and 192.168.77.101, destination port 80, no source range
• Service B, ingress Ips are 192.168.77.200 and 192.168.77.201, destination port 80, source ranges are 192.168.1.0/24 and 10.10.1.0/24

For design 1:

• Add a new reg mark ToLoadBalancerRegMark = binding.NewOneBitRegMark(4, 28)
• Add a new reg mark NotAllowedSourcesRegMark = binding.NewOneBitRegMark(4, 29)
• We reuse NodePortMarkTable and rename it to ServiceMarkTable, add the following flows:
  • For Service A:

    table=ServiceMarkTable, priority=200, tcp, nw_dst=192.168.77.100, tp_dst=80 actions=load:0x1->NXM_NX_REG4[28]
    table=ServiceMarkTable, priority=200, tcp, nw_dst=192.168.77.101, tp_dst=80 actions=load:0x1->NXM_NX_REG4[28]

  • For Service B:

    table=ServiceMarkTable, priority=200, tcp, nw_src=192.168.1.0/24, nw_dst=192.168.77.200, tp_dst=80 actions=load:0x1->NXM_NX_REG4[27]
    table=ServiceMarkTable, priority=200, tcp, nw_src=192.168.1.0/24, nw_dst=192.168.77.201, tp_dst=80 actions=load:0x1->NXM_NX_REG4[27]
    table=ServiceMarkTable, priority=200, tcp, nw_src=10.10.1.0/24, nw_dst=192.168.77.200, tp_dst=80 actions=load:0x1->NXM_NX_REG4[27]
    table=ServiceMarkTable, priority=200, tcp, nw_src=10.10.1.0/24, nw_dst=192.168.77.201, tp_dst=80 actions=load:0x1->NXM_NX_REG4[27]
    table=ServiceMarkTable, priority=100, tcp, nw_dst=192.168.77.200, tp_dst=80 actions=load:0x1->NXM_NX_REG4[28]
    table=ServiceMarkTable, priority=100, tcp, nw_dst=192.168.77.201, tp_dst=80 actions=load:0x1->NXM_NX_REG4[28]

• In ServiceLBTable:
  • For Service A:

    table=ServiceLBTable, priority=200,tcp, reg4=0x10000/0x70000, NXM_NX_REG4[27]=0x1, nw_dst=192.168.77.100, tp_dst=80 actions=set_field:0x200/0x200->reg0,set_field:0x20000/0x70000->reg4,set_field:0x1->reg7,group:1
    table=ServiceLBTable, priority=200,tcp, reg4=0x10000/0x70000, NXM_NX_REG4[27]=0x1, nw_dst=192.168.77.101, tp_dst=80 actions=set_field:0x200/0x200->reg0,set_field:0x20000/0x70000->reg4,set_field:0x2->reg7,group:2

  • For Service B:

    table=ServiceLBTable, priority=200,tcp, reg4=0x10000/0x70000, NXM_NX_REG4[27]=0x1, nw_dst=192.168.77.200, tp_dst=80 actions=set_field:0x200/0x200->reg0,set_field:0x20000/0x70000->reg4,set_field:0x3->reg7,group:3
    table=ServiceLBTable, priority=200,tcp, reg4=0x10000/0x70000, NXM_NX_REG4[27]=0x1, nw_dst=192.168.77.201, tp_dst=80 actions=set_field:0x200/0x200->reg0,set_field:0x20000/0x70000->reg4,set_field:0x4->reg7,group:4

  • Reject flow (fixed flow, install it when initializing flows):

    table=ServiceLBTable, priority=200,tcp, reg4=0x10000/0x70000, NXM_NX_REG4[28]=0x1 actions=controller(reason=no_match,max_len=128,id=28467,userdata=04)

For design 2:

• Add a new table called LoadBalancerSourceFilterTable after ServiceLBTable.
• Add a new reg mark AllowedSourcesRegMark = binding.NewZeroBitRegMark(4, 27)
• Add a new reg mark NotAllowedSourcesRegMark = binding.NewOneBitRegMark(4, 27)
• In ServiceLBTable:
  • For Service A:

    table=ServiceLBTable, priority=200,tcp, reg4=0x10000/0x70000, nw_dst=192.168.77.100, tp_dst=80 actions=set_field:0x200/0x200->reg0,set_field:0x20000/0x70000->reg4,set_field:0x1->reg7, resubmit(,LoadBalancerSourceFilterTable),  group:1
    table=ServiceLBTable, priority=200,tcp, reg4=0x10000/0x70000, nw_dst=192.168.77.101, tp_dst=80 actions=set_field:0x200/0x200->reg0,set_field:0x20000/0x70000->reg4,set_field:0x2->reg7, resubmit(,LoadBalancerSourceFilterTable),  group:2

  • For Service B:

    table=ServiceLBTable, priority=200,tcp, reg4=0x10000/0x70000 , nw_dst=192.168.77.200, tp_dst=80 actions=set_field:0x200/0x200->reg0,set_field:0x20000/0x70000->reg4,set_field:0x3->reg7, resubmit(,LoadBalancerSource), group:3
    table=ServiceLBTable, priority=200,tcp, reg4=0x10000/0x70000, nw_dst=192.168.77.201, tp_dst=80 actions=set_field:0x200/0x200->reg0,set_field:0x20000/0x70000->reg4,set_field:0x4->reg7, resubmit(,LoadBalancerSource), group:4

• In LoadBalancerSourceFilterTable:
  • For Service A:

    table=LoadBalancerSourceFilterTable, priority=200, tcp, nw_dst=192.168.77.100, tp_dst=80 actions=load:0x0->NXM_NX_REG4[27]
    table=LoadBalancerSourceFilterTable, priority=200, tcp, nw_dst=192.168.77.101, tp_dst=80 actions=load:0x0->NXM_NX_REG4[27]

  • For Service B:

    table=LoadBalancerSourceFilterTable, priority=200, tcp, nw_src=192.168.1.0/24, nw_dst=192.168.77.200, tp_dst=80 actions=load:0x0->NXM_NX_REG4[27]
    table=LoadBalancerSourceFilterTable, priority=200, tcp, nw_src=192.168.1.0/24, nw_dst=192.168.77.201, tp_dst=80 actions=load:0x0->NXM_NX_REG4[27]
    table=LoadBalancerSourceFilterTable, priority=200, tcp, nw_src=10.10.1.0/24, nw_dst=192.168.77.200, tp_dst=80 actions=load:0x0->NXM_NX_REG4[27]
    table=LoadBalancerSourceFilterTable, priority=200, tcp, nw_src=10.10.1.0/24, nw_dst=192.168.77.201, tp_dst=80 actions=load:0x0->NXM_NX_REG4[27]

  • Default drop flow (fixed flow, install it when initializing flows):

    table=LoadBalancerSourceFilterTable, priority=100, actions=load:0x1->NXM_NX_REG4[27]

• In EndpointDNATTable, there is flow to match the packets which are not allowed and send them to controller to generate a reject packet

  table=EndpointDNATTable, priority=200,NXM_NX_REG4[28]=0x1 actions=controller(reason=no_match,max_len=128,id=28467,userdata=04)

Could you give some suggestions @tnqn? Thanks a lot.

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment, or this will be closed in 90 days

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment, or this will be closed in 90 days

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment, or this will be closed in 90 days

[hongliangl]

For the current design implemented in #6181, see the git commit message. In an offline discussion, @wenyingd proposed a new design for the implementation:

Here are the flows of the latest Antrea main branch code with proxyAll enabled. I believe that everyone is very familiar with them.

...
table=ConntrackState, priority=0 actions=goto_table:PreRoutingClassifier

table=PreRoutingClassifier, priority=200,ip actions=resubmit(,NodePortMark),resubmit(,SessionAffinity),resubmit(,ServiceLB)
table=PreRoutingClassifier, priority=200,ipv6 actions=resubmit(,NodePortMark),resubmit(,SessionAffinity),resubmit(,ServiceLB)
table=PreRoutingClassifier, priority=0 actions=goto_table:NodePortMark

table=NodePortMark, priority=200,ip,nw_dst=10.176.27.91 actions=set_field:0x80000/0x80000->reg4
table=NodePortMark, priority=200,ip,nw_dst=192.168.77.100 actions=set_field:0x80000/0x80000->reg4
table=NodePortMark, priority=200,ip,nw_dst=172.17.0.1 actions=set_field:0x80000/0x80000->reg4
table=NodePortMark, priority=200,ip,nw_dst=169.254.0.252 actions=set_field:0x80000/0x80000->reg4
table=NodePortMark, priority=200,ipv6,ipv6_dst=2620:124:6020:1006:250:56ff:fea7:2787 actions=set_field:0x80000/0x80000->reg4
table=NodePortMark, priority=200,ipv6,ipv6_dst=fd00::192:168:77:100 actions=set_field:0x80000/0x80000->reg4
table=NodePortMark, priority=200,ipv6,ipv6_dst=fc01::aabb:ccdd:eefe actions=set_field:0x80000/0x80000->reg4

table=SessionAffinity, priority=0 actions=set_field:0x10000/0x70000->reg4

table=ServiceLB, priority=200,tcp,reg4=0x10000/0x70000,nw_dst=10.96.0.1,tp_dst=443 actions=set_field:0x200/0x200->reg0,set_field:0x20000/0x70000->reg4,set_field:0x5->reg7,group:5
table=ServiceLB, priority=200,tcp,reg4=0x10000/0x70000,nw_dst=10.96.225.251,tp_dst=80 actions=set_field:0x200/0x200->reg0,set_field:0x30000/0x70000->reg4,set_field:0x6->reg7,group:6
table=ServiceLB, priority=0 actions=goto_table:DSRServiceMark

Here are the draft flows of the design to implement loadBalancerSourceRanges:

...
table=ConntrackState, priority=0 actions=goto_table:LoadBalancerSourceRange

table=LoadBalancerSourceRange, priority=210,tcp,nw_src=192.168.77.0/24,nw_dst=192.168.77.150,tp_dst=80 actions=goto_table:PreRoutingClassifier
table=LoadBalancerSourceRange, priority=210,tcp,nw_src=192.168.78.0/24,nw_dst=192.168.77.150,tp_dst=80 actions=goto_table:PreRoutingClassifier
table=LoadBalancerSourceRange, priority=200,tcp,nw_dst=192.168.77.150,tp_dst=80 actions=drop
table=LoadBalancerSourceRange, priority=0 actions=goto_table:NodePortMark

table=NodePortMark, priority=200,ip,nw_dst=169.254.0.252 actions=set_field:0x80000/0x80000->reg4, goto_table:PreRoutingClassifier
table=NodePortMark, priority=200,ipv6,ipv6_dst=fd00::192:168:77:100 actions=set_field:0x80000/0x80000->reg4, goto_table:PreRoutingClassifier
table=NodePortMark, priority=200,ipv6,ipv6_dst=fc01::aabb:ccdd:eefe actions=set_field:0x80000/0x80000->reg4, goto_table:PreRoutingClassifier
table=NodePortMark, priority=0 actions=goto_table:PreRoutingClassifier

table=PreRoutingClassifier, priority=0,ip actions=resubmit(,SessionAffinity),resubmit(,ServiceLB)
table=PreRoutingClassifier, priority=0,ipv6 actions=resubmit(,SessionAffinity),resubmit(,ServiceLB)

table=SessionAffinity, priority=0 actions=set_field:0x10000/0x70000->reg4

table=ServiceLB, priority=200,tcp,reg4=0x10000/0x70000,nw_dst=10.96.0.1,tp_dst=443 actions=set_field:0x200/0x200->reg0,set_field:0x20000/0x70000->reg4,set_field:0x5->reg7,group:5
table=ServiceLB, priority=200,tcp,reg4=0x10000/0x70000,nw_dst=10.96.225.251,tp_dst=80 actions=set_field:0x200/0x200->reg0,set_field:0x30000/0x70000->reg4,set_field:0x6->reg7,group:6
table=ServiceLB, priority=0 actions=goto_table:DSRServiceMark

We can see key changes:

• Add a new table LoadBalancerSourceRange (this is not a great name; we may come up with another name if we use this design finally).
  • Add flows to match the packets sourced from each allowed CIDR and destined for the LoadBalancer IP and port, and forward them to the table PreRoutingClassifier.
  • Add flow to match the packets not sourced from any allowed CIDR and drop them.
    • Add a default flow to match other packets and forward them to the table NodePortMark to do NodePort traffic pre-checks.
• Table NodePortMark is like before, marking the potential NodePort packets, and all packets will be forwarded to the table PreRoutingClassifier.
• Table PreRoutingClassifier (maybe we should change the name, since xxxClassifier is always the first table of a stage, according to the convention of flexible pipeline design. PreRoutingClassifier is not a proper name again as it is not the first table of stage PreRouting) is like before, sending the packets to tables SessionAffinity and ServiceLB to implement Service load-balancing and sessionAffinity.

Look forwarding to your suggestions @tnqn @antoninbas