Closed gudmundur closed 1 month ago
Been trying to sort out what's up with the failing sbr
test and have been unable to reproduce. I see that it's also failing on another recent pull request (https://github.com/containernetworking/plugins/pull/1099, failing test). Happy to provide a fix if I had a way to reproduce the failure. 😭
Thanks for the PR. Have you seen #1097? Are these two PRs duplicate or complementary?
@squeed The second issue #1097 describes is somewhat related, but it misses the mark on setting it to zero meaning that the kernel will pick a default.
This can be illustrated as such:
$ ip link add dev vm1 type veth
$ ip link show vm1
... snip snip snip ...
20: vm1@vm2: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 76:8b:12:5d:e9:c0 brd ff:ff:ff:ff:ff:ff
Note the qlen 1000
on the interface. Now if we run the same command, but set txqueuelen 0
the following happens:
$ ip link add dev vm1 txqueuelen 0 type veth
$ ip link show vm1
24: vm1@veth0: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN mode DEFAULT group default
link/ether 6a:19:4e:0b:3a:5e brd ff:ff:ff:ff:ff:ff
Note the missing qlen 1000
. If you look at any regular device that is created by ip
they will always have a qlen 1000
. The netlink
library propagates the 0 to the syscall when creating the link. I remember before fixing this, that we would see a handful of these log lines in the kernel logs. In digging a little through the Linux codebase it seems that a value of zero is caught in some places. In Googling this a little more, I also found that the Cilium folks had this issue in the past.
What I did notice on our end when we worked around this issue, was that we saw less tx packet drops. Now I can't remember the specifics on whether that was on veth
or tap
.
(@LionelJouin found the cause of the flake, he should have a fix soon)
Rebased to pick up SBR fixes.
@squeed Took care of the squashing for you. Reworded the commit message for a single commit as well.
When diagnosing networking issues, I noticed that
txqueuelen
onveth
pairs andtap
devices was set to zero. In investigating the issue, I found that most usages ofnetlink.LinkAttrs
in this repository uses the struct literal, which in thenetlink
docs is advised against unlessTxQLen
is explicitly set.When using the
ip
command directly without specifyingtxqueuelen
, Linux will automatically to 1000:This change goes through every single usage of
netlink.LinkAttrs{
and replaces it with thenetlink.NewLinkAttrs
constructor, and then overrides the individual fields in the struct.I even noticed that in the
bridge
plugin, this is somewhat called out when using the literal. I've replaced this explicitly in d376c8df5088e62c3a993ce14e5653b7d37407a0 though.