Packet loss on machines on switch acbdc32

illinois-scicomp / machine-shop-maintenance

Scripts and Issues for the birds and the beers

10 stars 3 forks source link

Packet loss on machines on switch acbdc32 #30

Closed inducer closed 3 years ago

inducer commented 3 years ago

Consider the following flood ping experiment:

root@porter:/# ping -f lager.cs.illinois.edu
PING lager.cs.illinois.edu (192.17.58.137) 56(84) bytes of data.
...........................................................................................................................^C
--- lager.cs.illinois.edu ping statistics ---
2240 packets transmitted, 2117 received, 5.49107% packet loss, time 2163ms
rtt min/avg/max/mdev = 0.050/0.076/0.325/0.020 ms, ipg/ewma 0.965/0.087 ms

More data from similar experiments:

lager -> porter: also ~5% packet loss
lager -> stout: no packet loss
stout -> porter: no packet loss
stout -> lager: no packet loss
tripel -> porter: ~5% packet loss
tripel -> stout: no packet loss
tripel -> dunkel: no packet loss

inducer commented 3 years ago

Also impacted:

dunkel <-> bock
koelsch <-> bock
Failing CI job on dunkel: https://gitlab.tiker.net/inducer/meshmode/-/jobs/189591 (indicating network generally has issues)

inducer commented 3 years ago

Reboot (on dunkel) does not help.

Kernel versions:

Dunkel 5.8.0-3-amd64
Porter 5.9.0-1-amd64

inducer commented 3 years ago

Some more observations:

Packet loss also occurs without flood ping. mtr -t (apt install mtr) also exhibits packet loss (at much lower load)
Packet loss occurs when pinging the gateway of 0017-engserv1-net, 192.17.58.129 (aka 0017-engserv1-net.gw.uiuc.edu), from: porter, stout, koelsch, dunkel, but not from lager, tripel, bock.

inducer commented 3 years ago

Submitted ticket to EngrIT last night, and tech services mid-day today.

inducer commented 3 years ago

Resolved by Tech Services:

Manual notification
Hi
I think the cause of your packet loss. We had a failing SFP on side of the switch uplink. I replaced this and the packet loss we we're seeing completely went away.
Please let me know if your still seeing issues.

Thanks
Lance

Request R5310399 Manual Notify.
Assigned to: Schickedanz, Lance A
Customer: Kloeckner, Andreas Paul Eberhard
Summary: Packet loss on machines in the ACB data center

Click on the following URL to view Request:
https://support.uillinois.edu/CAisd/pdmweb.exe?OP=SEARCH+FACTORY=cr+SKIPLIST=1+QBE.EQ.id=5474295

**************************************************************
DO NOT CHANGE OR REMOVE THE SECTION BELOW OR CHANGE
THE SUBJECT OR REPLIES WILL NOT UPDATE TICKETS.
AS A COURTESY, PLEASE REMOVE ORIGINAL MESSAGE FROM YOUR REPLY.
**************************************************************
%REQUEST_ID=R5310399
%STATUS=Client Updated