Node crash - libdatachannel error while creating datachannel nbr failed to gather local ICE candidates

[crProductGuy]

Got a sudden node crash with the error line above on Win 11 built from 0.1.7 source.

See attached screenshots.

The code looks like a network error that lacks a resilient recovery, from looking at the traceback and some source files.

The mining process kept running, saying "not connected to a node" as you can see at the top of the first screenshot.

Environment: Windows 11 desktop with Ryzen 7 5700G, 8 mining threads on the 8 cores.

Context: Node & miner with 8 threads had been running well overnight at 1.1MH/s, then crashed with this error after several hours. I didn't see it until an hour or 2 after the crash, so have no specific evidence of any network errors or timeouts that might have caused it.

However, I have a Windows cmd window open that's doing ping -t google.com, which before and after was showing a nice short 7-8 mS ping time, and my VPN was active before and after. So if there were any transient network glitches, they resolved. Connectivity is GigEth to my main home router, a nice fast Asus Zen mesh base, to Gig fiber ISP service by Ziply.com

After I captured screenshots post-error, the system started up fine again (miner was not impacted and continued OK once node was running again).

Evidence:

1st screenshot is Main Node window showing error and slices of the mining window above and the Ping Google window below.

2nd screenshot is a longer lookback in the Node window showing blockseq additions, etc.

3rd screenshot is ironfish status later showing version 0.1.7.

4th screenshot is Code section where crash occurred:

In file ironfish/src/network/peers/connections/webRtcConnection.ts, line 128. (close channel on network error)

[dguenther]

Thanks for the detailed problem report! We shouldn't crash the node process in this case for sure.

It looks like the error was actually getting thrown from createDataChannel -- it becomes a little confusing because the line numbers in the stack trace map to the compiled webRtcConnection.js rather than webRtcConnection.ts. I wasn't aware that createDataChannel could throw an error, but we should be able to simply wrap it in a try/catch, log an error, and close the connection if this happens.

[Shekelme]

So please solve this issue.

SEND WebRtc sYNcykq: NewTransaction (8) (repeated 15 times)
SEND WebRtc sYNcykq: SignalRequest (12)
CONN WebRtc unidentified STATE DISCONNECTED -> CONNECTING
PEER 9UtG/RL (thepouez20) STATE DISCONNECTED -> CONNECTING
Attempted to establish a WebRTC connection to tidfc0N (dda), but couldn't find a peer to broker the connection.
CONN WebSocket unidentified STATE DISCONNECTED -> CONNECTING
PEER tidfc0N (dda) STATE DISCONNECTED -> CONNECTING
CONN WebRtc unidentified STATE DISCONNECTED -> CONNECTING
/usr/lib/node_modules/ironfish/node_modules/@ironfish/sdk/build/src/network/peers/connections/webRtcConnection.js:132
            this.initializeDataChannel(this.peer.createDataChannel(''));
                                                 ^

Error: libdatachannel error while creating datachannel: Failed to gather local ICE candidates
    at PeerConnection.createDataChannel (<anonymous>)
    at new WebRtcConnection (/usr/lib/node_modules/ironfish/node_modules/@ironfish/sdk/build/src/network/peers/connections/webRtcConnection.js:132:50)
    at PeerManager.initWebRtcConnection (/usr/lib/node_modules/ironfish/node_modules/@ironfish/sdk/build/src/network/peers/peerManager.js:190:28)
    at PeerManager.connectToWebRTC (/usr/lib/node_modules/ironfish/node_modules/@ironfish/sdk/build/src/network/peers/peerManager.js:145:18)
    at PeerConnectionManager.connectToEligiblePeers (/usr/lib/node_modules/ironfish/node_modules/@ironfish/sdk/build/src/network/peers/peerConnectionManager.js:77:38)
    at PeerConnectionManager.eventLoop (/usr/lib/node_modules/ironfish/node_modules/@ironfish/sdk/build/src/network/peers/peerConnectionManager.js:56:22)
    at Timeout._onTimeout (/usr/lib/node_modules/ironfish/node_modules/@ironfish/sdk/build/src/network/peers/peerConnectionManager.js:72:53)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc

I use NPM installation method, Node.js v16.15.1, Ubuntu server 20.04.4 LTS, AMD Ryzen Threadripper PRO 3955WX 16-Cores, 512GB RAM

[Shekelme]

v0.1.41, still the same issue:

RECV WebRtc j6k2QqS: NewTransaction (8)
RECV WebRtc j6k2QqS: NewTransaction (8)
RECV WebRtc j6k2QqS: NewTransaction (8)
RECV WebRtc j6k2QqS: NewTransaction (8)
RECV WebSocket 2Gput3E: PeerListRequest (10)
RECV WebSocket UKvls76: SignalRequest (12)
CONN WebRtc unidentified STATE DISCONNECTED -> CONNECTING
/usr/lib/node_modules/ironfish/node_modules/@ironfish/sdk/build/src/network/peers/connections/webRtcConnection.js:133
            this.initializeDataChannel(this.peer.createDataChannel(''));
                                                 ^

Error: libdatachannel error while creating datachannel: Failed to gather local ICE candidates
    at PeerConnection.createDataChannel (<anonymous>)
    at new WebRtcConnection (/usr/lib/node_modules/ironfish/node_modules/@ironfish/sdk/build/src/network/peers/connections/webRtcConnection.js:133:50)
    at PeerManager.initWebRtcConnection (/usr/lib/node_modules/ironfish/node_modules/@ironfish/sdk/build/src/network/peers/peerManager.js:190:28)
    at PeerManager.handleSignalRequestMessage (/usr/lib/node_modules/ironfish/node_modules/@ironfish/sdk/build/src/network/peers/peerManager.js:900:14)
    at PeerManager.handleMessage (/usr/lib/node_modules/ironfish/node_modules/@ironfish/sdk/build/src/network/peers/peerManager.js:624:18)
    at /usr/lib/node_modules/ironfish/node_modules/@ironfish/sdk/build/src/network/peers/peerManager.js:496:24
    at Event.emitAsync (/usr/lib/node_modules/ironfish/node_modules/@ironfish/sdk/build/src/event.js:61:39)
    at Event.emit (/usr/lib/node_modules/ironfish/node_modules/@ironfish/sdk/build/src/event.js:52:19)
    at messageHandler (/usr/lib/node_modules/ironfish/node_modules/@ironfish/sdk/build/src/network/peers/peer.js:419:32)
    at Event.emitAsync (/usr/lib/node_modules/ironfish/node_modules/@ironfish/sdk/build/src/event.js:61:39)

[QustomQure]

0.1.51 and still this issue, sadly C:\Users\fr0ze\AppData\Roaming\npm\node_modules\ironfish\node_modules\@ironfish\sdk\build\src\network\peers\connections\webRtcConnection.js:133 this.initializeDataChannel(this.peer.createDataChannel('')); ^

Error: libdatachannel error while creating datachannel: Failed to gather local ICE candidates at PeerConnection.createDataChannel () at new WebRtcConnection (C:\Users\fr0ze\AppData\Roaming\npm\node_modules\ironfish\node_modules\@ironfish\sdk\build\src\network\peers\connections\webRtcConnection.js:133:50) at PeerManager.initWebRtcConnection (C:\Users\fr0ze\AppData\Roaming\npm\node_modules\ironfish\node_modules\@ironfish\sdk\build\src\network\peers\peerManager.js:200:28) at PeerManager.handleSignalRequestMessage (C:\Users\fr0ze\AppData\Roaming\npm\node_modules\ironfish\node_modules\@ironfish\sdk\build\src\network\peers\peerManager.js:924:14) at PeerManager.handleMessage (C:\Users\fr0ze\AppData\Roaming\npm\node_modules\ironfish\node_modules\@ironfish\sdk\build\src\network\peers\peerManager.js:640:18) at C:\Users\fr0ze\AppData\Roaming\npm\node_modules\ironfish\node_modules\@ironfish\sdk\build\src\network\peers\peerManager.js:503:18 at Event.emitAsync (C:\Users\fr0ze\AppData\Roaming\npm\node_modules\ironfish\node_modules\@ironfish\sdk\build\src\event.js:61:39) at Event.emit (C:\Users\fr0ze\AppData\Roaming\npm\node_modules\ironfish\node_modules\@ironfish\sdk\build\src\event.js:52:19) at messageHandler (C:\Users\fr0ze\AppData\Roaming\npm\node_modules\ironfish\node_modules\@ironfish\sdk\build\src\network\peers\peer.js:410:32) at Event.emitAsync (C:\Users\fr0ze\AppData\Roaming\npm\node_modules\ironfish\node_modules\@ironfish\sdk\build\src\event.js:61:39) PS C:\Users\fr0ze> ironfish start

[QustomQure]

PS C:\Users\fr0ze> ironfish debug Iron Fish version 0.1.51 @ 67da183 Iron Fish library 0.0.28 @ 67da183 Operating system Windows_NT x64 CPU model(s) AMD Ryzen 9 5950X 16-Core Processor CPU threads 32 RAM total 63.92 GiB Heap total 4.00 GiB Node version v16.13.2 ironfish in PATH true Garbage Collector Exposed false Telemetry enabled true Node name Qualia Block graffiti Qualia Account 851d6b uuid 851d6b1a-8e93-40f9-900f-f52c341bccca Account 851d6b name qualia Account 851d6b head hash 0000000000001ba9b5f49babf7c073d1b4f0e059cf97b8dca51f6b8609aea81f Account 851d6b head in chain true Account 851d6b sequence 262178