Overhead compensation for wireguard tunnel over docis, ADSL, etc.

[xnoreq]

Wireguard has some overhead, pads to some block size. How can we deal with this in cake if combined with other overhead compensations such as cable?

[xnoreq]

Also, why doesn't the following setup work:

• cake on wg0 tunnel interface with unlimited bandwidth and flow isolation
• qdisc on eth0 (which is used to send/receive the wireguard packets) with limited bandwidth

Shouldn't the queues fill up in the cake qdisc despite unlimited bandwidth because the outer interface/qdisc only dequeues at a limited rate?

[tohojo]

No, there's no backpressure through the wg interface unless the crypto can't keep up. And that is rare for wg.

Also, you shouldn't need the inner qdisc. Wireguard will (in recent versions) preserve the flow hash across encryption, so the outer cake instance can correctly queue the flows despite them being encrypted. You can check this by looking at the counters from 'tc -s qdisc'.

[xnoreq]

No backpressure? So where do the packets end up if the inner rate exceeds the outer rate? Are they simply dropped? What buffers are involved here?

On the second point: can other qdiscs besides cake also "see" wg flows? And how does this work anyway? I assume cake processes the final encrypted packet, that is transferred through eth0, but uses the in-tunnel addresses (such as 10.0.0.1 and 10.0.0.2) for flow isolation. So the "connection" between eth0 IP and the other peer's external IP itself becomes invisible to cake?

[tohojo]

On 30 August 2020 19:49:58 CEST, xnoreq notifications@github.com wrote:

No backpressure? So where do the packets end up if the inner rate exceeds the outer rate? Are they simply dropped? What buffers are involved here?

They'll end up queued in the outer qdisc. The stats should also show you this...

On the second point: can other qdiscs besides cake also "see" wg flows?

Yeah, at least fq_codel and fq, iirc.

And how does this work anyway? I assume cake processes the final encrypted packet, that is transferred through eth0, but uses the in-tunnel addresses (such as 10.0.0.1 and 10.0.0.2) for flow isolation. So the "connection" between eth0 IP and the other peer's external IP itself becomes invisible to cake?

The packets are sorted into flows by hashing on the packet header. Wireguard will make sure this happens prior to encryption, and that the result (the hash) is kept with the packet even after it is encrypted...

[lingfish]

Wireguard will (in recent versions) preserve the flow hash across encryption

Sorry to comment on a closed ticket, but can you clarify specifically from what version(s) of Wireguard, kernel etc that this happens on? As I'm not seeing it occur on tunnelled traffic.

[tohojo]

This was added in commit: c78a0b4a7883 ("wireguard: queueing: preserve flow hash across packet scrubbing") which went into Linux 5.7...

[lingfish]

Interesting, thanks for that. I see where your patch made it into kernel releases. I see where the scrubbing one made it too.

What I don't see is those making it into Debian bullseye changelog, nor do I see your related patch as mentioned here.

Upon doing some test pings using ping -Q 192 to aim for diffserv8 tin 7, I see counters increase on the wg interface, but not the outer interface.

Description:    Debian GNU/Linux 11 (bullseye)
Release:        11
Codename:       bullseye

Linux node 5.10.0-8-amd64 #1 SMP Debian 5.10.46-5 (2021-09-23) x86_64 GNU/Linux

I'm rather confused

[tohojo]

Ah, but it doesn't actually propagate the mark, it just sets the skb->hash field. This is a purely internal field that qdiscs can use to distinguish flows, but it doesn't go unto the wire. Cake will use this for flow hashing if you don't enable host fairness, but it won't affect tin selection...

If you do want to propagate the actual diffserv field, I wrote a BPF program that uses the hash to do this, here: https://github.com/xdp-project/bpf-examples/tree/master/preserve-dscp

[lingfish]

Ah, but it doesn't actually propagate the mark

Right, so you mean that inner qdisc on a wg interfaces counters won't increase, unless using some BPF stuff?

But above you seem to indicate that the outer qdisc will queue (put into tin?) the flows as recognised DSCP-wise from inside the wg? And hence counters should reflect that?

If that's what you mean, I'm not seeing that happening for me.

[tohojo]

Nope. The DSCP does not propagate at all unless you do the BPF stuff. The only thing the wg preserving of the hash does, is that it makes the outer qdisc able to distinguish individual flows. It can't see anything about those flows, only that they are different. This is enough to get per-flow queueing and sparse flow optimisation, but no tin selection.

The way to see if it's working is to watch the flow counters in cake...

[lingfish]

Thanks heaps for answering questions here, much appreciated!