thrnz
commented
2 weeks ago Here's the sanitized ICE debug logs showing the difference between the relayed peer connections after waking and the direct peer connections after running netbird down then netbird up
Describe the problem
Previously working direct peer connections sometimes fall back to being relayed, possibly after waking from sleep. When it happens,
netbird downthennetbird upis enough to get the peer connections to be direct again.To Reproduce
While I can't reliably reproduce it yet, logging seems to show it happening when waking from sleep. There may be a minimum sleep period before it happens, and it might not happen on every wake.
Expected behavior
Direct peer connections should be reestablished.
Are you using NetBird Cloud?
Yes
NetBird version
0.28.9on Debian BookwormNetBird status -dA output: Note the two relayed peers. After bringing netbird down and up again they go back to being direct.
Screenshots
n/a
Additional context
I'm trying to diagnose an issue where peer connections are made via a relay server when a direct connection should be possible using stun. Running
netbird downthennetbird upseems to be enough to get direct peer connections established again when it happens.Netbird is running inside a Debian Bookworm VM, and I think the issues occurs when the host machine wakes from sleep and peer connections need reestablishing.
I noticed that there doesn't appear to be any mention of
stun.netbird.ioin the debug log when one of these relayed connections is made, almost as if the stun server isn't being used when it happens. For example a successful direct connection shows something like:Whereas there's no mention of
stun.netbird.iowhen a connection between the same peers ends up being relayed:I've attached a sanitized debug client.log file that hopefully show whats happening. The two relevant remote peers are
uMGErvJAyflc7PkSVttAXkZB5Ga09qFKJ4ThQTLanduMGErvJAyflc7PkSVttAXkZB5Ga09qFKJ4ThQTL+5VE=. The previously quoted timestamps should hopefully show the difference between a successful (15:52:55) direct connection, and a relayed one (15:37:09) being used instead to the same peer.The relay log entry at
15:37above was after waking from sleep, and the direct connection at15:52happened after runningnetbird downthennetbird up.I've also got a
netbird.outdebug logfile with thePIONS_LOG_DEBUG=allenv var set that probably shows some useful info, though haven't attached it yet as its unsanitized. I'll try to snip out and sanitize the relevant time periods and will attach them later below, but briefly of note is the following when the15:37post-wake connection attempt happened:There are also a couple of dozen more 'Failed to send packet' errors such as the ones below that only appear on that connection attempt, and not on the successful direct one. The errors appear nowhere else in the logfile apart from connection attempts immediately after the system wakes from sleep:
It's almost as if access to the stun server or network connectivity is initially unavailable on waking, causing the relay to be used instead.
I don't think there's any otherwise noteworthy networking stuff going on between peers. They're on devices behind standard residential internet connections without any CGNAT. When it works, stun seems to be enough to get direct connections working (at least until the machine goes to sleep).
From what I can make out by reading the log files, Netbird doesn't try reconnecting once the relayed connection is established. I'm not sure if this is currently a thing, but perhaps occasionally trying to 'upgrade' to a direct connection from a relayed one might be worth a feature request in future. While it might not be the cause of this issue, it could possibly work around it.