Symptoms:

Packets going into the wg interface on the supernode stop arriving on the node at the mesh_wg interface. (tcpdump -ni mesh_wg inbound).
Packets are still visible on br-wan.
wg handshakes are renewed.
CPU load drops significantly (to <20% on ER-X).
Memory load increases by a small but stable base amount.

Resulting symptoms:

Batman VPN Neighbors goes down to 0.
TX Link Quality to Supernodes (aka. TQ) no longer gets telemetry.
RX Link Quality from Supernodes (aka. reversed TQ) goes down to <5%.
Forward TX Traffic drops to 0.

See from:

Mailthread: "Failure Check"
Grafana: https://stats.ffh.zone/d/EGysRPsMz/router-fur-meshviewer?var-node=74acb9a77737&from=1685753778880&to=1685786777470&orgId=1
debug_info: https://gist.github.com/1977er/7ed4ba1028b903a8957be8bdf61beecb
debug_info: https://gist.github.com/1977er/3c52bf5ffba9273ef54766f286a87c6f

Actions taken so far:

Monitoring checks if vpn_neighbors=0 (only works for sites with multiple uplinks).

(Edit by @lemoer: translated using deepl to english)

Debug output from today:

The Situation

The two peers are UFU-FWH-E106-Woermannstr-Technik1 and sn10.
They have an established connection since approx 17 days.
Since 5 days (2023-06-17 15:44), the connection is broken.
Broken means:
- Packets entering the wg interface on UFU-FWH-E106-Woermannstr-Technik1 are dropped somewhere before they come out of the wg interface on sn10.
- The other way around seems to be ok.

UFU-FWH-E106-Woermannstr-Technik1

wg show on UFU-FWH-E106-Woermannstr-Technik1

root@UFU-FWH-E106-Woermannstr-Technik1:/tmp# wg show
interface: wg_mesh
  public key: lp7KvqqBam9boZyfJJgLIjd8jWF7nREqGA53WW5pfSE=
  private key: (hidden)
  listening port: 52231
  fwmark: 0x1

peer: SN10jGGoMekUFtCenlM1RMbnFh3fqJnhjTXnpNWqJ1A=
  endpoint: 81.3.6.91:51816
  allowed ips: fe80::1/128
  latest handshake: 2 seconds ago
  transfer: 1.75 TiB received, 192.12 GiB sent

Outbound on UFU-FWH-E106-Woermannstr-Technik1 wireguard interface

root@UFU-FWH-E106-Woermannstr-Technik1:/tmp# tcpdump -n -i wg_mesh -c 5 outbound 
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on wg_mesh, link-type RAW (Raw IP), capture size 262144 bytes
21:02:22.415229 IP6 fe80::277:2bff:fe2a:2f28.58294 > fe80::1.4789: VXLAN, flags [I] (0x08), vni 16317534
02:3e:b3:3a:4a:fa > ff:ff:ff:ff:ff:ff, ethertype Unknown (0x4305), length 162: 
    0x0000:  000f 2c00 11d1 0c62 d65e 5062 8583 be18  ..,....b.^Pb....
    0x0010:  0305 c0b0 0093 0038 0401 0028 013e 0003  .......8...(.>..
    0x0020:  a145 edb7 8001 0000 e879 9090 8000 0000  .E.......y......
    0x0030:  618f fa0b 0000 0000 1300 0000 3007 4de1  a...........0.M.
    0x0040:  3f73 0000 0602 0004 0100 0000 0201 0000  ?s..............
    0x0050:  000f 2a00 fe08 c294 a6ef 7a86 2ed3 be18  ..*.......z.....
    0x0060:  0305 c0b0 0068 002c 0401 001c 0197 0003  .....h.,........
    0x0070:  041e 320d 8001 0000 4d22 4f2a 8000 0000  ..2.....M"O*....
    0x0080:  d6e4 041b 0000 0000 0602 0004 0100 0000  ................
    0x0090:  0201 0000                                ....
21:02:22.425599 IP6 fe80::277:2bff:fe2a:2f28.58294 > fe80::1.4789: VXLAN, flags [I] (0x08), vni 16317534
02:3e:b3:3a:4a:fa > ff:ff:ff:ff:ff:ff, ethertype Unknown (0x4305), length 150: 
    0x0000:  000f 3100 ba50 2c2a be18 0305 c0b3 be18  ..1..P,*........
    0x0010:  0305 c0b0 00f0 002c 0401 001c 0141 0003  .......,.....A..
    0x0020:  b244 398b 8001 0000 fb78 44ac 8000 0000  .D9......xD.....
    0x0030:  7e27 3918 0000 0000 0602 0004 0100 0000  ~'9.............
    0x0040:  0201 0000 000f 2b00 e9ad 07e2 be2f 2c4b  ......+....../,K
    0x0050:  bbc3 be18 0305 c0b0 0081 002c 0401 001c  ...........,....
    0x0060:  016b 0003 65ad 73c8 8001 0000 2c91 0eef  .k..e.s.....,...
    0x0070:  8000 0000 0944 f095 0000 0000 0602 0004  .....D..........
    0x0080:  0100 0000 0201 0000                      ........
21:02:22.535510 IP6 fe80::277:2bff:fe2a:2f28.58294 > fe80::1.4789: VXLAN, flags [I] (0x08), vni 16317534
02:3e:b3:3a:4a:fa > ff:ff:ff:ff:ff:ff, ethertype Unknown (0x4305), length 218: 
    0x0000:  000f 2c00 f565 8fdf 5e87 400e b7d3 be18  ..,..e..^.@.....
    0x0010:  0305 c0b0 0089 002c 0401 001c 01ba 0003  .......,........
    0x0020:  d01b 8003 8001 0000 9927 fd24 8000 0000  .........'.$....
    0x0030:  568f 2ea2 0000 0000 0602 0004 0100 0000  V...............
    0x0040:  0201 0000 000f 2c00 5386 3a55 4e27 fe8a  ......,.S.:UN'..
    0x0050:  718b be18 0305 c0b0 0089 002c 0401 001c  q..........,....
    0x0060:  0128 0003 37ae 48e3 8001 0000 7e92 35c4  .(..7.H.....~.5.
    0x0070:  8000 0000 b682 98ae 0000 0000 0602 0004  ................
    0x0080:  0100 0000 0201 0000 000f 2b00 8021 66f5  ..........+..!f.
    0x0090:  8253 adf4 2fab be18 0305 c0b0 0070 002c  .S../........p.,
    0x00a0:  0401 001c 01a6 0003 2dd5 ff67 8001 0000  ........-..g....
    0x00b0:  64e9 8240 8000 0000 ccd7 5534 0000 0000  d..@......U4....
    0x00c0:  0602 0004 0100 0000 0201 0000            ............
21:02:22.635524 IP6 fe80::277:2bff:fe2a:2f28.58294 > fe80::1.4789: VXLAN, flags [I] (0x08), vni 16317534
02:3e:b3:3a:4a:fa > ff:ff:ff:ff:ff:ff, ethertype Unknown (0x4305), length 218: 
    0x0000:  000f 2b00 4dff 2907 3e39 c135 b7fb be18  ..+.M.).>9.5....
    0x0010:  0305 c0b0 0070 002c 0401 001c 013f 0003  .....p.,.....?..
    0x0020:  7dba 5544 8001 0000 3486 2863 8000 0000  }.UD....4.(c....
    0x0030:  1d10 2714 0000 0000 0602 0004 0100 0000  ..'.............
    0x0040:  0201 0000 000f 2c00 7854 616e 323d f939  ......,.xTan2=.9
    0x0050:  9dfb be18 0305 c0b0 0089 002c 0401 001c  ...........,....
    0x0060:  01ad 0003 d10e 6764 8001 0000 9832 1a43  ......gd.....2.C
    0x0070:  8000 0000 be01 52f0 0000 0000 0602 0004  ......R.........
    0x0080:  0100 0000 0201 0000 000f 2c00 cce9 3c46  ..........,...<F
    0x0090:  ce87 9658 284b be18 0305 c0b0 0089 002c  ...X(K.........,
    0x00a0:  0401 001c 0114 0003 7512 562b 8001 0000  ........u.V+....
    0x00b0:  3c2e 2b0c 8000 0000 6ecd 2eec 0000 0000  <.+.....n.......
    0x00c0:  0602 0004 0100 0000 0201 0000            ............
21:02:22.775439 IP6 fe80::277:2bff:fe2a:2f28.58294 > fe80::1.4789: VXLAN, flags [I] (0x08), vni 16317534
02:3e:b3:3a:4a:fa > ff:ff:ff:ff:ff:ff, ethertype Unknown (0x4305), length 366: 
    0x0000:  000f 2c00 a645 b3e6 9272 c34b f293 be18  ..,..E...r.K....
    0x0010:  0305 c0b0 0093 0038 0401 0028 01f5 0003  .......8...(....
    0x0020:  2602 c729 8001 0000 6f3e ba0e 8000 0000  &..)....o>......
    0x0030:  5d6d 6c89 0000 0000 1300 0000 8a70 8924  ]ml..........p.$
    0x0040:  48bf 0000 0602 0004 0100 0000 0201 0000  H...............
    0x0050:  000f 2c00 8151 cf97 96ea 7c9a 6603 be18  ..,..Q....|.f...
    0x0060:  0305 c0b0 0089 002c 0401 001c 013e 0003  .......,.....>..
    0x0070:  20db 4c0d 8001 0000 69e7 312a 8000 0000  ..L.....i.1*....
    0x0080:  f32a 46b0 0000 0000 0602 0004 0100 0000  .*F.............
    0x0090:  0201 0000 000f 2d00 5823 c893 5e0e 5730  ......-.X#..^.W0
    0x00a0:  b823 be18 0305 c0b0 009d 002c 0401 001c  .#.........,....
    0x00b0:  01d3 0003 1a00 6fd0 8001 0000 533c 12f7  ......o.....S<..
    0x00c0:  8000 0000 2c79 3702 0000 0000 0602 0004  ....,y7.........
    0x00d0:  0100 0000 0201 0000 000f 2b00 7dcd 93d0  ..........+.}...
    0x00e0:  2e69 250d 193b be18 0305 c0b0 007f 002c  .i%..;.........,
    0x00f0:  0401 001c 010a 0003 bd43 6cc4 8001 0000  .........Cl.....
    0x0100:  f47f 11e3 8000 0000 d20b 9440 0000 0000  ...........@....
    0x0110:  0602 0004 0100 0000 0201 0000 000f 2c00  ..............,.
    0x0120:  6390 8721 5270 2e88 d863 be18 0305 c0b0  c..!Rp...c......
    0x0130:  0089 002c 0401 001c 0102 0003 03df 976f  ...,...........o
    0x0140:  8001 0000 4ae3 ea48 8000 0000 f230 5043  ....J..H.....0PC
    0x0150:  0000 0000 0602 0004 0100 0000 0201 0000  ................
5 packets captured
6 packets received by filter
0 packets dropped by kernel

Inbound on UFU-FWH-E106-Woermannstr-Technik1 wireguard interface

root@UFU-FWH-E106-Woermannstr-Technik1:/tmp# tcpdump -n -i wg_mesh -c 5 inbound
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on wg_mesh, link-type RAW (Raw IP), capture size 262144 bytes
21:03:23.618209 IP6 fe80::1.55409 > fe80::277:2bff:fe2a:2f28.4789: VXLAN, flags [I] (0x08), vni 16317534
02:a1:71:04:10:16 > ff:ff:ff:ff:ff:ff, ethertype Unknown (0x4305), length 114: 
    0x0000:  010f 2d00 0000 2ac5 a201 1cf5 7333 3333  ..-...*.....s333
    0x0010:  ff00 0001 6637 be3b 72af 86dd 6000 0000  ....f7.;r...`...
    0x0020:  0020 3aff fdca ffee 0008 0016 8512 352b  ..:...........5+
    0x0030:  9488 28c7 ff02 0000 0000 0000 0000 0001  ..(.............
    0x0040:  ff00 0001 8700 6f3d 0000 0000 fdca ffee  ......o=........
    0x0050:  0008 0016 0000 0000 0000 0001 0101 6637  ..............f7
    0x0060:  be3b 72af                                .;r.
21:03:23.653481 IP6 fe80::1.55409 > fe80::277:2bff:fe2a:2f28.4789: VXLAN, flags [I] (0x08), vni 16317534
02:a1:71:04:10:16 > ff:ff:ff:ff:ff:ff, ethertype Unknown (0x4305), length 490: 
    0x0000:  000f 3000 f837 fd3c a6d4 fd2d 9e03 0653  ..0..7.<...-...S
    0x0010:  09f0 eb74 00c6 002c 0401 001c 0160 0003  ...t...,.....`..
    0x0020:  f3d5 1bea 8001 0000 bae9 66cd 8000 0000  ..........f.....
    0x0030:  c5fc 7380 0000 0000 0602 0004 0100 0000  ..s.............
    0x0040:  0201 0000 000f 2f00 e2a6 5d06 6ef6 27c9  ....../...].n.'.
    0x0050:  27db b27c aa6e fb33 0086 002c 0401 001c  '..|.n.3...,....
    0x0060:  0152 0003 820d d32d 8001 0000 cb31 ae0a  .R.....-.....1..
    0x0070:  8000 0000 ef6d d0cc 0000 0000 0602 0004  .....m..........
    0x0080:  0100 0000 0201 0000 000f 3000 81ba 667e  ..........0...f~
    0x0090:  bec0 cdff e61b 7a83 2589 f298 00c6 002c  ......z.%......,
    0x00a0:  0401 001c 0103 0003 a69f 614b 8001 0000  ..........aK....
    0x00b0:  efa3 1c6c 8000 0000 8508 f04f 0000 0000  ...l.......O....
    0x00c0:  0602 0004 0100 0000 0201 0000 000f 2f00  ............../.
    0x00d0:  a5ab 2750 2a98 271f 0e9b 7a83 2589 f298  ..'P*.'...z.%...
    0x00e0:  00ba 002c 0401 001c 0140 0003 27f0 3775  ...,.....@..'.7u
    0x00f0:  8001 0000 6ecc 4a52 8000 0000 b72d c5a4  ....n.JR.....-..
    0x0100:  0000 0000 0602 0004 0100 0000 0201 0000  ................
    0x0110:  000f 2f00 75f4 7fce 9612 a6c8 a613 7a83  ../.u.........z.
    0x0120:  2589 f298 00ba 002c 0401 001c 0176 0003  %......,.....v..
    0x0130:  7574 0d83 8001 0000 3c48 70a4 8000 0000  ut......<Hp.....
    0x0140:  95d4 cbc1 0000 0000 0602 0004 0100 0000  ................
    0x0150:  0201 0000 000f 2d00 acca d432 bef1 c258  ......-....2...X
    0x0160:  b1f3 6667 fc1d ed22 0074 002c 0401 001c  ..fg...".t.,....
    0x0170:  017a 0003 c764 8e65 8001 0000 8e58 f342  .z...d.e.....X.B
    0x0180:  8000 0000 e359 687f 0000 0000 0602 0004  .....Yh.........
    0x0190:  0100 0000 0201 0000 000f 3000 8646 5e23  ..........0..F^#
    0x01a0:  7aac 9fb8 bc53 4685 c781 1018 00d0 002c  z....SF........,
    0x01b0:  0401 001c 0186 0003 313e 1d5a 8001 0000  ........1>.Z....
    0x01c0:  7802 607d 8000 0000 0bdc e5e7 0000 0000  x.`}............
    0x01d0:  0602 0004 0100 0000 0201 0000            ............
21:03:23.776182 IP6 fe80::1.55409 > fe80::277:2bff:fe2a:2f28.4789: VXLAN, flags [I] (0x08), vni 16317534
02:a1:71:04:10:16 > ff:ff:ff:ff:ff:ff, ethertype Unknown (0x4305), length 310: 
    0x0000:  000f 3000 64ef 5a0d da9e 7e27 9ef3 b27c  ..0.d.Z...~'...|
    0x0010:  aa6e fb33 00d3 002c 0401 001c 01d7 0003  .n.3...,........
    0x0020:  1e76 cfb6 8001 0000 574a b291 8000 0000  .v......WJ......
    0x0030:  539d 50d7 0000 0000 0602 0004 0100 0000  S.P.............
    0x0040:  0201 0000 000f 2f00 7710 04c8 5603 1486  ....../.w...V...
    0x0050:  5353 6667 fc1d ed22 00ba 0044 0401 0034  SSfg..."...D...4
    0x0060:  0142 0003 9d4e 74a0 8001 0000 d472 0987  .B...Nt......r..
    0x0070:  8000 0000 91e6 90b1 0000 0000 1300 0000  ................
    0x0080:  3007 4de1 3f73 0000 1100 0000 46a1 ad8f  0.M.?s......F...
    0x0090:  fe22 0000 0602 0004 0100 0000 0201 0000  ."..............
    0x00a0:  000f 2f00 0163 7fe2 6254 3fb7 0133 6667  ../..c..bT?..3fg
    0x00b0:  fc1d ed22 00ba 002c 0401 001c 01ac 0003  ..."...,........
    0x00c0:  f0b0 35ae 8001 0000 b98c 4889 8000 0000  ..5.......H.....
    0x00d0:  dbb4 fe5b 0000 0000 0602 0004 0100 0000  ...[............
    0x00e0:  0201 0000 000f 3000 f6a6 0519 127f 1db0  ......0.........
    0x00f0:  ba2b 6667 fc1d ed22 00c6 002c 0401 001c  .+fg..."...,....
    0x0100:  0121 0003 2ef5 89ec 8001 0000 67c9 f4cb  .!..........g...
    0x0110:  8000 0000 d20b 9440 0000 0000 0602 0004  .......@........
    0x0120:  0100 0000 0201 0000                      ........
21:03:23.788811 IP6 fe80::1.55409 > fe80::277:2bff:fe2a:2f28.4789: VXLAN, flags [I] (0x08), vni 16317534
02:a1:71:04:10:16 > ff:ff:ff:ff:ff:ff, ethertype Unknown (0x4305), length 70: 
    0x0000:  010f 3100 0015 92b2 72e3 f0ca 68fc ffff  ..1.....r...h...
    0x0010:  ffff ffff 88e6 40ba a016 0806 0001 0800  ......@.........
    0x0020:  0604 0001 88e6 40ba a016 0a10 6401 0000  ......@.....d...
    0x0030:  0000 0000 0a10 6726                      ......g&
21:03:23.874988 IP6 fe80::1.55409 > fe80::277:2bff:fe2a:2f28.4789: VXLAN, flags [I] (0x08), vni 16317534
02:a1:71:04:10:16 > ff:ff:ff:ff:ff:ff, ethertype Unknown (0x4305), length 390: 
    0x0000:  000f 3000 e243 f342 5232 e0d2 fcab 4afd  ..0..C.BR2....J.
    0x0010:  27db 26dc 00c6 0038 0401 0028 011f 0003  '.&....8...(....
    0x0020:  1a39 5905 8001 0000 5305 2422 8000 0000  .9Y.....S.$"....
    0x0030:  dac6 e871 0000 0000 1100 0000 56f2 6d23  ...q........V.m#
    0x0040:  e2a2 0000 0602 0004 0100 0000 0201 0000  ................
    0x0050:  000f 2f00 cd58 c365 02a6 da6c 641b 6667  ../..X.e...ld.fg
    0x0060:  fc1d ed22 00ba 002c 0401 001c 016b 0003  ..."...,.....k..
    0x0070:  adcd 2363 8001 0000 e4f1 5e44 8000 0000  ..#c......^D....
    0x0080:  ac42 52b0 0000 0000 0602 0004 0100 0000  .BR.............
    0x0090:  0201 0000 000f 3100 ab2e 0529 ce9e 4182  ......1....)..A.
    0x00a0:  b5b3 42ae 25e5 5274 00cf 0044 0401 0034  ..B.%.Rt...D...4
    0x00b0:  0110 0003 9a71 8cb2 8001 0000 d34d f195  .....q.......M..
    0x00c0:  8000 0000 7a4e b02a 0000 0000 0100 0000  ....zN.*........
    0x00d0:  3333 ff78 57d3 0000 0100 0000 3333 ff6c  33.xW.......33.l
    0x00e0:  2ab9 0000 0602 0004 0100 0000 0201 0000  *...............
    0x00f0:  000f 3000 cf81 3f35 2637 87d9 76fb 4afd  ..0...?5&7..v.J.
    0x0100:  27db 26dc 00c6 002c 0401 001c 01bb 0003  '.&....,........
    0x0110:  a669 7925 8001 0000 ef55 0402 8000 0000  .iy%.....U......
    0x0120:  f34d db32 0000 0000 0602 0004 0100 0000  .M.2............
    0x0130:  0201 0000 000f 3000 8230 55ee 5e03 c2cf  ......0..0U.^...
    0x0140:  9923 2aab 8a15 cdcd 00c3 002c 0401 001c  .#*........,....
    0x0150:  0125 0003 3192 e5ae 8001 0000 78ae 9889  .%..1.......x...
    0x0160:  8000 0000 3f6b 668e 0000 0000 0602 0004  ....?kf.........
    0x0170:  0100 0000 0201 0000                      ........
5 packets captured
6 packets received by filter
0 packets dropped by kernel

sn10

wg show on sn10

[root@sn10]:~ # wg show  wg-16 | grep lp7KvqqBam9boZyfJJgLIjd8jWF7nREqGA53WW5pfSE= -A 5
peer: lp7KvqqBam9boZyfJJgLIjd8jWF7nREqGA53WW5pfSE=
  endpoint: 89.183.250.69:52231
  allowed ips: fe80::277:2bff:fe2a:2f28/128
  latest handshake: 10 seconds ago
  transfer: 188.98 GiB received, 1.77 TiB sent

Inbound on wireguard interface of sn10

(No packets appear here.)

[root@sn10]:~ # timeout 10 tcpdump -n -i wg-16 host fe80::277:2bff:fe2a:2f28 and inbound
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on wg-16, link-type RAW (Raw IP), snapshot length 262144 bytes

0 packets captured
44 packets received by filter
0 packets dropped by kernel

Outbound on wireguard interface on sn10

[root@sn10]:~ # tcpdump -n -i wg-16 host fe80::277:2bff:fe2a:2f28 and outbound -c 5
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on wg-16, link-type RAW (Raw IP), snapshot length 262144 bytes
21:11:49.536700 IP6 fe80::1.55409 > fe80::277:2bff:fe2a:2f28.4789: VXLAN, flags [I] (0x08), vni 16317534
02:a1:71:04:10:16 > ff:ff:ff:ff:ff:ff, ethertype Unknown (0x4305), length 354: 
    0x0000:  000f 2f00 656b 5815 8239 d50c 8c33 6667  ../.ekX..9...3fg
    0x0010:  fc1d ed22 00b3 002c 0401 001c 01fb 0003  ..."...,........
    0x0020:  74ee b3f6 8001 0000 3dd2 ced1 8000 0000  t.......=.......
    0x0030:  80dd eb54 0000 0000 0602 0004 0100 0000  ...T............
    0x0040:  0201 0000 000f 3000 32dc 47a1 a2b5 e36f  ......0.2.G....o
    0x0050:  714b 2aab 8a15 cdcd 00c3 002c 0401 001c  qK*........,....
    0x0060:  012b 0003 4587 3e1d 8001 0000 0cbb 433a  .+..E.>.......C:
    0x0070:  8000 0000 b5f1 b26a 0000 0000 0602 0004  .......j........
    0x0080:  0100 0000 0201 0000 000f 2f00 0bef 4b93  ........../...K.
    0x0090:  8a35 5004 e96b 7a83 2589 f298 00ba 002c  .5P..kz.%......,
    0x00a0:  0401 001c 011b 0003 507a e980 8001 0000  ........Pz......
    0x00b0:  1946 94a7 8000 0000 71c3 b36c 0000 0000  .F......q..l....
    0x00c0:  0602 0004 0100 0000 0201 0000 000f 3000  ..............0.
    0x00d0:  933e 99b8 56aa 2418 4243 4685 c781 1018  .>..V.$.BCF.....
    0x00e0:  00d3 002c 0401 001c 0130 0003 aed4 43ea  ...,.....0....C.
    0x00f0:  8001 0000 e7e8 3ecd 8000 0000 150d 5812  ......>.......X.
    0x0100:  0000 0000 0602 0004 0100 0000 0201 0000  ................
    0x0110:  000f 2d00 6ef9 1b1c 9623 d51e 6b5b 6667  ..-.n....#..k[fg
    0x0120:  fc1d ed22 0087 002c 0401 001c 01dc 0003  ..."...,........
    0x0130:  76c5 732a 8001 0000 3ff9 0e0d 8000 0000  v.s*....?.......
    0x0140:  cda5 9b58 0000 0000 0602 0004 0100 0000  ...X............
    0x0150:  0201 0000                                ....
21:11:49.635886 IP6 fe80::1.55409 > fe80::277:2bff:fe2a:2f28.4789: VXLAN, flags [I] (0x08), vni 16317534
02:a1:71:04:10:16 > ff:ff:ff:ff:ff:ff, ethertype Unknown (0x4305), length 230: 
    0x0000:  000f 3000 1342 8ae9 7a0e 21c0 f8f3 2aab  ..0..B..z.!...*.
    0x0010:  8a15 cdcd 00c6 002c 0401 001c 0184 0003  .......,........
    0x0020:  0701 df09 8001 0000 4e3d a22e 8000 0000  ........N=......
    0x0030:  241b 49e2 0000 0000 0602 0004 0100 0000  $.I.............
    0x0040:  0201 0000 000f 3000 83de 3a5e e6be ec5f  ......0...:^..._
    0x0050:  508b 42ae 25e5 5274 00c5 0038 0401 0028  P.B.%.Rt...8...(
    0x0060:  0140 0003 17aa e8d4 8001 0000 5e96 95f3  .@..........^...
    0x0070:  8000 0000 a671 9ed1 0000 0000 1000 0000  .....q..........
    0x0080:  e25c 0513 1a4b 0000 0602 0004 0100 0000  .\...K..........
    0x0090:  0201 0000 000f 3000 c33b 8c8e b2ec 4f8b  ......0..;....O.
    0x00a0:  0f6b 0653 09f0 eb74 00c6 002c 0401 001c  .k.S...t...,....
    0x00b0:  0145 0003 32c4 9e53 8001 0000 7bf8 e374  .E..2..S....{..t
    0x00c0:  8000 0000 11bb e639 0000 0000 0602 0004  .......9........
    0x00d0:  0100 0000 0201 0000                      ........
21:11:49.672427 IP6 fe80::1.55409 > fe80::277:2bff:fe2a:2f28.4789: VXLAN, flags [I] (0x08), vni 16317534
02:a1:71:04:10:16 > ff:ff:ff:ff:ff:ff, ethertype Unknown (0x4305), length 114: 
    0x0000:  010f 2e00 0000 0a4f 5a51 32cd 36ab 3333  .......OZQ2.6.33
    0x0010:  ff00 0001 1ccc d695 a8e6 86dd 6000 0000  ............`...
    0x0020:  0020 3aff fdca ffee 0008 0016 f042 8924  ..:..........B.$
    0x0030:  9bd7 e6d0 ff02 0000 0000 0000 0000 0001  ................
    0x0040:  ff00 0001 8700 e594 0000 0000 fdca ffee  ................
    0x0050:  0008 0016 0000 0000 0000 0001 0101 1ccc  ................
    0x0060:  d695 a8e6                                ....
21:11:49.682309 IP6 fe80::1.55409 > fe80::277:2bff:fe2a:2f28.4789: VXLAN, flags [I] (0x08), vni 16317534
02:a1:71:04:10:16 > ff:ff:ff:ff:ff:ff, ethertype Unknown (0x4305), length 70: 
    0x0000:  010f 2c00 0000 6787 6254 3fb7 0133 ffff  ..,...g.bT?..3..
    0x0010:  ffff ffff e076 d040 3f17 0806 0001 0800  .....v.@?.......
    0x0020:  0604 0001 e076 d040 3f17 0a10 5a8f 0000  .....v.@?...Z...
    0x0030:  0000 0000 0a10 5ad8                      ......Z.
21:11:49.683157 IP6 fe80::1.55409 > fe80::277:2bff:fe2a:2f28.4789: VXLAN, flags [I] (0x08), vni 16317534
02:a1:71:04:10:16 > ff:ff:ff:ff:ff:ff, ethertype Unknown (0x4305), length 70: 
    0x0000:  010f 2c00 0000 6786 6254 3fb7 0133 ffff  ..,...g.bT?..3..
    0x0010:  ffff ffff e076 d040 3f17 0806 0001 0800  .....v.@?.......
    0x0020:  0604 0001 e076 d040 3f17 0a10 5a8f 0000  .....v.@?...Z...
    0x0030:  0000 0000 0a10 5a57                      ......ZW
5 packets captured
63 packets received by filter
0 packets dropped by kernel

Currently, it seems that only handshakes are sent out at UFU-FWH-E106-Woermannstr-Technik1:

root@UFU-FWH-E106-Woermannstr-Technik1:/tmp# tcpdump -n -i br-wan outbound and host 81.3.6.91
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br-wan, link-type EN10MB (Ethernet), capture size 262144 bytes
21:27:47.011731 IP 192.168.178.32.52231 > 81.3.6.91.51816: UDP, length 92
21:28:02.369570 IP 192.168.178.32.52231 > 81.3.6.91.51816: UDP, length 92
21:28:17.792572 IP 192.168.178.32.52231 > 81.3.6.91.51816: UDP, length 92
21:28:33.352853 IP 192.168.178.32.52231 > 81.3.6.91.51816: UDP, length 92
21:28:48.973022 IP 192.168.178.32.52231 > 81.3.6.91.51816: UDP, length 92

And all of them seem to appear on sn10:

[root@sn10]:~ # tcpdump -n -i any host 89.183.250.69 and inbound
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
21:27:47.020876 eth0  In  IP 89.183.250.69.52231 > 81.3.6.91.51816: UDP, length 92
21:28:02.381791 eth0  In  IP 89.183.250.69.52231 > 81.3.6.91.51816: UDP, length 92
21:28:17.833270 eth0  In  IP 89.183.250.69.52231 > 81.3.6.91.51816: UDP, length 92
21:28:33.362380 eth0  In  IP 89.183.250.69.52231 > 81.3.6.91.51816: UDP, length 92
21:28:48.982652 eth0  In  IP 89.183.250.69.52231 > 81.3.6.91.51816: UDP, length 92

It also doesn't seem to be related to the vxlan traffic that we are sending into the wg interface. If we do a normal ping, we can also just see the handshakes on br-wan:

root@UFU-FWH-E106-Woermannstr-Technik1:/tmp# ping fe80::1%wg_mesh
PING fe80::1%wg_mesh (fe80::1%16): 56 data bytes

root@UFU-FWH-E106-Woermannstr-Technik1:~# tcpdump -n -i wg_mesh outbound and icmp6 -v
tcpdump: listening on wg_mesh, link-type RAW (Raw IP), capture size 262144 bytes
21:41:55.055297 IP6 (flowlabel 0xd5ca2, hlim 64, next-header ICMPv6 (58) payload length: 64) fe80::277:2bff:fe2a:2f28 > fe80::1: [icmp6 sum ok] ICMP6, echo request, seq 161
21:41:56.065291 IP6 (flowlabel 0xd5ca2, hlim 64, next-header ICMPv6 (58) payload length: 64) fe80::277:2bff:fe2a:2f28 > fe80::1: [icmp6 sum ok] ICMP6, echo request, seq 162
21:41:57.075316 IP6 (flowlabel 0xd5ca2, hlim 64, next-header ICMPv6 (58) payload length: 64) fe80::277:2bff:fe2a:2f28 > fe80::1: [icmp6 sum ok] ICMP6, echo request, seq 163
21:41:58.085348 IP6 (flowlabel 0xd5ca2, hlim 64, next-header ICMPv6 (58) payload length: 64) fe80::277:2bff:fe2a:2f28 > fe80::1: [icmp6 sum ok] ICMP6, echo request, seq 164
21:41:59.095278 IP6 (flowlabel 0xd5ca2, hlim 64, next-header ICMPv6 (58) payload length: 64) fe80::277:2bff:fe2a:2f28 > fe80::1: [icmp6 sum ok] ICMP6, echo request, seq 165
21:42:00.105287 IP6 (flowlabel 0xd5ca2, hlim 64, next-header ICMPv6 (58) payload length: 64) fe80::277:2bff:fe2a:2f28 > fe80::1: [icmp6 sum ok] ICMP6, echo request, seq 166
21:42:01.115318 IP6 (flowlabel 0xd5ca2, hlim 64, next-header ICMPv6 (58) payload length: 64) fe80::277:2bff:fe2a:2f28 > fe80::1: [icmp6 sum ok] ICMP6, echo request, seq 167
21:42:02.125331 IP6 (flowlabel 0xd5ca2, hlim 64, next-header ICMPv6 (58) payload length: 64) fe80::277:2bff:fe2a:2f28 > fe80::1: [icmp6 sum ok] ICMP6, echo request, seq 168
21:42:03.135295 IP6 (flowlabel 0xd5ca2, hlim 64, next-header ICMPv6 (58) payload length: 64) fe80::277:2bff:fe2a:2f28 > fe80::1: [icmp6 sum ok] ICMP6, echo request, seq 169
21:42:04.145293 IP6 (flowlabel 0xd5ca2, hlim 64, next-header ICMPv6 (58) payload length: 64) fe80::277:2bff:fe2a:2f28 > fe80::1: [icmp6 sum ok] ICMP6, echo request, seq 170
21:42:05.155292 IP6 (flowlabel 0xd5ca2, hlim 64, next-header ICMPv6 (58) payload length: 64) fe80::277:2bff:fe2a:2f28 > fe80::1: [icmp6 sum ok] ICMP6, echo request, seq 171
21:42:06.165324 IP6 (flowlabel 0xd5ca2, hlim 64, next-header ICMPv6 (58) payload length: 64) fe80::277:2bff:fe2a:2f28 > fe80::1: [icmp6 sum ok] ICMP6, echo request, seq 172
21:42:07.175321 IP6 (flowlabel 0xd5ca2, hlim 64, next-header ICMPv6 (58) payload length: 64) fe80::277:2bff:fe2a:2f28 > fe80::1: [icmp6 sum ok] ICMP6, echo request, seq 173
21:42:08.185316 IP6 (flowlabel 0xd5ca2, hlim 64, next-header ICMPv6 (58) payload length: 64) fe80::277:2bff:fe2a:2f28 > fe80::1: [icmp6 sum ok] ICMP6, echo request, seq 174
21:42:09.195293 IP6 (flowlabel 0xd5ca2, hlim 64, next-header ICMPv6 (58) payload length: 64) fe80::277:2bff:fe2a:2f28 > fe80::1: [icmp6 sum ok] ICMP6, echo request, seq 175
21:42:10.205302 IP6 (flowlabel 0xd5ca2, hlim 64, next-header ICMPv6 (58) payload length: 64) fe80::277:2bff:fe2a:2f28 > fe80::1: [icmp6 sum ok] ICMP6, echo request, seq 176
21:42:11.215292 IP6 (flowlabel 0xd5ca2, hlim 64, next-header ICMPv6 (58) payload length: 64) fe80::277:2bff:fe2a:2f28 > fe80::1: [icmp6 sum ok] ICMP6, echo request, seq 177
21:42:12.225289 IP6 (flowlabel 0xd5ca2, hlim 64, next-header ICMPv6 (58) payload length: 64) fe80::277:2bff:fe2a:2f28 > fe80::1: [icmp6 sum ok] ICMP6, echo request, seq 178
21:42:13.235292 IP6 (flowlabel 0xd5ca2, hlim 64, next-header ICMPv6 (58) payload length: 64) fe80::277:2bff:fe2a:2f28 > fe80::1: [icmp6 sum ok] ICMP6, echo request, seq 179
21:42:14.245295 IP6 (flowlabel 0xd5ca2, hlim 64, next-header ICMPv6 (58) payload length: 64) fe80::277:2bff:fe2a:2f28 > fe80::1: [icmp6 sum ok] ICMP6, echo request, seq 180
21:42:15.255438 IP6 (flowlabel 0xd5ca2, hlim 64, next-header ICMPv6 (58) payload length: 64) fe80::277:2bff:fe2a:2f28 > fe80::1: [icmp6 sum ok] ICMP6, echo request, seq 181
21:42:16.265321 IP6 (flowlabel 0xd5ca2, hlim 64, next-header ICMPv6 (58) payload length: 64) fe80::277:2bff:fe2a:2f28 > fe80::1: [icmp6 sum ok] ICMP6, echo request, seq 182
21:42:17.275321 IP6 (flowlabel 0xd5ca2, hlim 64, next-header ICMPv6 (58) payload length: 64) fe80::277:2bff:fe2a:2f28 > fe80::1: [icmp6 sum ok] ICMP6, echo request, seq 183
21:42:18.285300 IP6 (flowlabel 0xd5ca2, hlim 64, next-header ICMPv6 (58) payload length: 64) fe80::277:2bff:fe2a:2f28 > fe80::1: [icmp6 sum ok] ICMP6, echo request, seq 184

root@UFU-FWH-E106-Woermannstr-Technik1:~# tcpdump -n -i br-wan outbound and host 81.3.6.91
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br-wan, link-type EN10MB (Ethernet), capture size 262144 bytes
21:41:55.388648 IP 192.168.178.32.52231 > 81.3.6.91.51816: UDP, length 92
21:42:10.527058 IP 192.168.178.32.52231 > 81.3.6.91.51816: UDP, length 92

wg show still shows a valid handshake:

root@UFU-FWH-E106-Woermannstr-Technik1:~# wg show
interface: wg_mesh
  public key: lp7KvqqBam9boZyfJJgLIjd8jWF7nREqGA53WW5pfSE=
  private key: (hidden)
  listening port: 52231
  fwmark: 0x1

peer: SN10jGGoMekUFtCenlM1RMbnFh3fqJnhjTXnpNWqJ1A=
  endpoint: 81.3.6.91:51816
  allowed ips: fe80::1/128
  latest handshake: 14 seconds ago
  transfer: 1.75 TiB received, 192.12 GiB sent

Here is the config dump from UFU-FWH-E106-Woermannstr-Technik1:

root@UFU-FWH-E106-Woermannstr-Technik1:~# wg showconf wg_mesh
[Interface]
ListenPort = 52231
FwMark = 0x1
PrivateKey = (redacted)

[Peer]
PublicKey = SN10jGGoMekUFtCenlM1RMbnFh3fqJnhjTXnpNWqJ1A=
AllowedIPs = fe80::1/128
Endpoint = 81.3.6.91:51816

Also, the TX counter of wireguard shows that packets are not sent out (second last column):

root@UFU-FWH-E106-Woermannstr-Technik1:~# wg show all dump; sleep 15; wg show all dump
wg_mesh (redacted)  lp7KvqqBam9boZyfJJgLIjd8jWF7nREqGA53WW5pfSE=    52231   0x1
wg_mesh SN10jGGoMekUFtCenlM1RMbnFh3fqJnhjTXnpNWqJ1A=    (none)  81.3.6.91:51816 fe80::1/128 1687464100  1920192174532   206288920592    off
wg_mesh (redacted)  lp7KvqqBam9boZyfJJgLIjd8jWF7nREqGA53WW5pfSE=    52231   0x1
wg_mesh SN10jGGoMekUFtCenlM1RMbnFh3fqJnhjTXnpNWqJ1A=    (none)  81.3.6.91:51816 fe80::1/128 1687464115  1920192233560   206288920684    off

Only 92 bytes are seen in 15 seconds only. This is excactly the size of one handshake packet.

Also an inbound trace on UFU-WFH-E106-Technik1 on br-wan:

root@UFU-FWH-E106-Woermannstr-Technik1:~# tcpdump -n -i any host 81.3.6.91 and inbound
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
22:12:53.556666 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 320
22:12:53.556714 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 320
22:12:53.592908 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 160
22:12:53.592959 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 160
22:12:53.651267 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 320
22:12:53.651317 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 320
22:12:53.723720 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 176
22:12:53.723768 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 176
22:12:53.723923 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 320
22:12:53.723942 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 320
22:12:53.753076 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 160
22:12:53.753132 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 160
22:12:53.856151 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 384
22:12:53.856198 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 384
22:12:53.856321 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 176
22:12:53.856346 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 176
22:12:53.951600 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 128
22:12:53.951648 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 128
22:12:54.042231 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 192
22:12:54.042275 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 192
22:12:54.042389 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 160
22:12:54.042404 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 160
22:12:54.042455 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 160
22:12:54.042478 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 160
22:12:54.059258 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 240
22:12:54.059304 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 240
22:12:54.204977 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 176
22:12:54.205059 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 176
22:12:54.335885 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 240
22:12:54.335933 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 240
22:12:54.340871 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 176
22:12:54.340925 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 176
22:12:54.519126 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 128
22:12:54.519190 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 128
22:12:54.519374 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 256
22:12:54.519394 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 256
22:12:54.519446 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 224
22:12:54.519461 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 224
22:12:54.538861 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 176
22:12:54.538910 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 176
22:12:54.580684 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 176
22:12:54.580730 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 176
22:12:54.670918 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 160
22:12:54.670967 IP 81.3.6.91.51816 > 192.168.178.32.52231: UDP, length 160
^C
44 packets captured
44 packets received by filter
0 packets dropped by kernel

I just found this in dmesg:

[373474.182938] ------------[ cut here ]------------
[373474.192353] WARNING: CPU: 0 PID: 0 at kernel/rcu/tree.c:624 rcu_eqs_enter.constprop.0+0xd8/0xe0
[373474.209825] Modules linked in: iptable_nat batman_adv xt_state xt_nat xt_conntrack xt_REDIRECT xt_MASQUERADE xt_CT wireguard nf_nat nf_conntrack libchacha20poly1305 ipt_REJECT ebtable_nat ebtable_filter ebtable_broute cfg80211 xt_time xt_tcpudp xt_quota xt_pkttype xt_owner xt_multiport xt_mark xt_mac xt_limit xt_comment xt_addrtype xt_TCPMSS xt_LOG ts_kmp ts_fsm ts_bm poly1305_mips nf_reject_ipv4 nf_log_ipv6 nf_log_ipv4 nf_log_common nf_defrag_ipv6 nf_defrag_ipv4 libcurve25519_generic iptable_mangle iptable_filter ip_tables ebtables ebt_vlan ebt_stp ebt_snat ebt_redirect ebt_pkttype ebt_mark_m ebt_mark ebt_limit ebt_ip6 ebt_ip ebt_dnat ebt_arpreply ebt_arp ebt_among ebt_802_3 compat chacha_mips sch_teql sch_sfq sch_multiq sch_gred sch_fq sch_dsmark sch_codel em_text em_nbyte em_meta em_cmp act_simple act_pedit act_csum libcrc32c sch_htb sch_hfsc em_u32 cls_u32 cls_tcindex cls_route cls_matchall cls_fw cls_flow act_skbedit act_mirred act_gact ip6table_mangle ip6table_filter ip6_tables
[373474.210230]  ip6t_REJECT x_tables nf_reject_ipv6 dummy ip_tunnel veth vxlan udp_tunnel ip6_udp_tunnel kpp leds_gpio cls_basic sch_tbf sch_ingress gpio_button_hotplug crc32c_generic
[373474.415982] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.10.161 #0
[373474.428266] Stack : 809a0000 00000000 00000000 8007c664 80830000 8072d760 00000000 00000000
[373474.445079]         807b3d24 80980000 806fdc74 807c7298 807c6dc7 00000001 807b3cc8 6f3afccb
[373474.461890]         00000000 00000000 806fdc74 807b3b68 ffffefff 00000000 ffffffea 00000000
[373474.478702]         807b3b74 0000017a 807cb9e0 ffffffff 00000000 00000000 00000000 80700000
[373474.495513]         00000009 3c016ed8 000153ac 00000000 00000000 803a42ec 00000000 80980000
[373474.512324]         ...
[373474.517357] Call Trace:
[373474.522404] [<80007fc0>] show_stack+0x30/0x100
[373474.531422] [<8031f7c8>] dump_stack+0x9c/0xcc
[373474.540272] [<8002fe64>] __warn+0xc0/0x12c
[373474.548587] [<8002ff2c>] warn_slowpath_fmt+0x5c/0xac
[373474.558629] [<80656d6c>] rcu_eqs_enter.constprop.0+0xd8/0xe0
[373474.570058] [<8045dde4>] cpuidle_enter_state+0x410/0x540
[373474.580790] [<8045dfac>] cpuidle_enter+0x84/0xac
[373474.590163] [<8005e860>] do_idle+0x26c/0x31c
[373474.598824] [<8005eb88>] cpu_startup_entry+0x2c/0x34
[373474.608887] [<80848d00>] start_kernel+0x570/0x598
[373474.618408] 
[373474.621532] ---[ end trace 74d97155b98095dd ]---

Not sure if this is related, since this happened 15 days ago.

Listening on any interface, also just shows the handshakes:

root@UFU-FWH-E106-Woermannstr-Technik1:~# tcpdump -ni any host 81.3.6.91 and outbound
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
22:53:01.899449 IP 192.168.178.32.52231 > 81.3.6.91.51816: UDP, length 92
22:53:01.899506 IP 192.168.178.32.52231 > 81.3.6.91.51816: UDP, length 92
22:53:01.899528 IP 192.168.178.32.52231 > 81.3.6.91.51816: UDP, length 92
22:53:17.252236 IP 192.168.178.32.52231 > 81.3.6.91.51816: UDP, length 92
22:53:17.252308 IP 192.168.178.32.52231 > 81.3.6.91.51816: UDP, length 92
22:53:17.252336 IP 192.168.178.32.52231 > 81.3.6.91.51816: UDP, length 92
22:53:32.634665 IP 192.168.178.32.52231 > 81.3.6.91.51816: UDP, length 92
22:53:32.634747 IP 192.168.178.32.52231 > 81.3.6.91.51816: UDP, length 92
22:53:32.634779 IP 192.168.178.32.52231 > 81.3.6.91.51816: UDP, length 92
22:53:48.233387 IP 192.168.178.32.52231 > 81.3.6.91.51816: UDP, length 92
22:53:48.233449 IP 192.168.178.32.52231 > 81.3.6.91.51816: UDP, length 92
22:53:48.233470 IP 192.168.178.32.52231 > 81.3.6.91.51816: UDP, length 92
22:54:03.588697 IP 192.168.178.32.52231 > 81.3.6.91.51816: UDP, length 92
22:54:03.588774 IP 192.168.178.32.52231 > 81.3.6.91.51816: UDP, length 92
22:54:03.588801 IP 192.168.178.32.52231 > 81.3.6.91.51816: UDP, length 92
22:54:18.947856 IP 192.168.178.32.52231 > 81.3.6.91.51816: UDP, length 92
22:54:18.947911 IP 192.168.178.32.52231 > 81.3.6.91.51816: UDP, length 92
22:54:18.947933 IP 192.168.178.32.52231 > 81.3.6.91.51816: UDP, length 92
^C
18 packets captured
18 packets received by filter
0 packets dropped by kernel

I suppose every handshake is shown three times, since I have a linux bridge (or so):

root@UFU-FWH-E106-Woermannstr-Technik1:~# brctl show
bridge name bridge id       STP enabled interfaces
br-mesh_other       7fff.eaf340edfbcc   no      eth3
                            eth1
                            eth4
                            eth2
br-client       7fff.74acb9a77737   no      bat0
                            local-port
br-wan      7fff.74acb9a77737   no      eth0

Unfortunately, the tcpdump doesn't show the interfaces where it saw the packets.

freifunkh / site

wg Tunnelblockade #100