CI sporadically failing

[rahulghangas]

In PR #835 , the CI failed here. However, a simple change in e72183f that did not introduce any new logic, the CI passed with flying colors. Not sure what's happening there, looks like a timeout issue.

Wild idea, maybe we have a race condition somewhere that causes congestion?

--- FAIL: TestBroadcastTxForPeerStopsWhenReactorStops (10.07s)
    leaktest.go:132: leaktest: timed out checking goroutines
    leaktest.go:150: leaktest: leaked goroutine: goroutine 2410 [select]:
        github.com/tendermint/tendermint/p2p/conn.(*MConnection).sendRoutine(0xc000eae9a0)
            /home/runner/work/celestia-core/celestia-core/p2p/conn/connection.go:431 +0x3f1
        created by github.com/tendermint/tendermint/p2p/conn.(*MConnection).OnStart
            /home/runner/work/celestia-core/celestia-core/p2p/conn/connection.go:230 +0x3a6
    leaktest.go:150: leaktest: leaked goroutine: goroutine 2411 [select]:
        net.(*pipe).read(0xc001568b00, {0xc00010f000, 0x414, 0x64})
            /opt/hostedtoolcache/go/1.17.13/x64/src/net/pipe.go:159 +0x249
        net.(*pipe).Read(0x414, {0xc00010f000, 0x414, 0x800})
            /opt/hostedtoolcache/go/1.17.13/x64/src/net/pipe.go:142 +0x55
        io.ReadAtLeast({0x7f0a78cdfac8, 0xc001568b00}, {0xc00010f000, 0x414, 0x800}, 0x414)
            /opt/hostedtoolcache/go/1.17.13/x64/src/io/io.go:328 +0xde
        io.ReadFull(...)
            /opt/hostedtoolcache/go/1.17.13/x64/src/io/io.go:347
        github.com/tendermint/tendermint/p2p/conn.(*SecretConnection).Read(0xc001568c80, {0xc000113400, 0x400, 0xc00157abc8})

[evan-forbes]

Yeah, unfortunately the CI is flakey for tendermint v0.34.20 (and v0.35 for that matter) (see also #822)

There could definitely be a mutex blocking somewhere, and it gets hit a lot more often in a resource constrained environment such as CI.

We might want to bubble this up to the tendermint team