Socket connection abruptly cut off: Network.Socket.connect: <socket: 13>: does not exist (Connection refused)`

[deepfire]

While running the cardano-tx-generator against a small Shelley cluster, the following error happens -- pretty well into the node-to-node-submission phase:

cardano-tx-generator-byron: Network.Socket.connect: <socket: 13>: does not exist (Connection refused)

Reproduction:

1. cardano-benchmarking commit cc89b8cd216a8da4d21e9185a8d5e2553bfb1d6e
2. ./benchmarks/cluster3nodes/start.sh --nix --shelley --no-rtview --no-delegate

[deepfire]

generator-20200616211005.json.txt node1-20200616210930.json.txt node1-20200616211140.json.txt node2-20200616210930.json.txt node3-20200616210930.json.txt

[coot]

node-1

~{"at":"2020-06-16T21:12:05.94Z","env":"1.13.0:00000","ns":["cardano.node.ErrorPolicy"],"data":{"event":"ErrorPolicySuspendPeer (Just (ApplicationExceptionTrace (MuxError {errorType = MuxIOException Network.Socket.recvBuf: resource vanished (Connection reset by peer), errorMsg = \"(recv errored)\", errorStack = []}))) 20s2 0s","kind":"ErrorPolicyTrace","address":"127.0.0.1:47014"},"app":[],"msg":"","pid":"7431","loc":null,"host":"andromed","sev":"Warning","thread":"72"}~

node-2

{"at":"2020-06-16T21:12:05.94Z","env":"1.13.0:00000","ns":["cardano.node.ErrorPolicy"],"data":{"event":"ErrorPolicySuspendPeer (Just (ApplicationExceptionTrace (MuxError {errorType = MuxIOException Network.Socket.recvBuf: resource vanished (Connection reset by peer), errorMsg = \"(recv errored)\", errorStack = []}))) 20s 20s","kind":"ErrorPolicyTrace","address":"127.0.0.1:58262"},"app":[],"msg":"","pid":"7399","loc":null,"host":"andromed","sev":"Warning","thread":"58"}

node-3

Nothing strange around 21:12:05

generators

Log ends before 21:12:05.94

[karknu]

@coot The log line pasted for node-1 is incorrect, that line is from node-2 too. node-1 log ends just before the node-2 recv error.