search

stellar / go

Stellar's public monorepo of go code
https://stellar.org/developers
Apache License 2.0
1.28k stars 497 forks source link

ingest/ledgerbackend: getLogLineWriter() leaks goroutines and pipes #5342

Closed 2opremio closed 1 week ago

2opremio commented 3 weeks ago

getLogLineWriter() creates a pipe and a goroutine, but the goroutine sometimes doesn't exit (and the pipe is never released).

Every time captive core is restarted this causes a leak.

The easiest fix is probably returning a CloseWriter and making sure that the writer is closed (releasing the pipe and the goroutine).

2opremio commented 3 weeks ago

Discovered while investigating failing tests at https://github.com/stellar/soroban-rpc/pull/207

2opremio commented 3 weeks ago

It turns out the goroutines are closed in many places but something must be amiss, since they are still left behind in some cases. All the (*pipe).read goroutines in this dump are from Closed() captive core backends (from different soroban-rpc tests):

Screenshot 2024-06-13 at 17 30 56
2opremio commented 3 weeks ago

I think stellarCoreRunner.handleExit() may not be closing the pipes in some cases

2opremio commented 3 weeks ago

Here's another dump:

Screenshot 2024-06-13 at 18 28 57
tamirms commented 3 weeks ago

It looks like there are a few createCmd calls where we do not close the log line writers:

https://github.com/stellar/go/blob/a4e5a3fc6664f7cf6b26fb3196b366910563636f/ingest/ledgerbackend/stellar_core_runner.go#L311

https://github.com/stellar/go/blob/a4e5a3fc6664f7cf6b26fb3196b366910563636f/ingest/ledgerbackend/stellar_core_runner.go#L400-L404

https://github.com/stellar/go/blob/a4e5a3fc6664f7cf6b26fb3196b366910563636f/ingest/ledgerbackend/stellar_core_runner.go#L311