search

streamingfast / node-manager

Wrapping process to operate blockchain nodes.
https://dfuse.io
Apache License 2.0
7 stars 9 forks source link

mindreader goes in infinite loop if geth stops #24

Closed matthewdarwin closed 2 years ago

matthewdarwin commented 2 years ago

There was a storage problem and geth stopped working (as expected), but mindreader just filled up the log with tons of messages...

2022-03-21T12:24:17.774Z (mindreader.geth) synchronisation failed, retrying         err="write tcp 172.22.27.120:46498->3.11.147.67:30303: use of closed network connection"
2022-03-21T12:24:17.775Z (mindreader.geth) synchronisation failed, retrying         err="shutting down"
2022-03-21T12:24:17.775Z (mindreader.geth) synchronisation failed, retrying         err="shutting down"
2022-03-21T12:24:17.775Z (mindreader.geth) synchronisation failed, retrying         err="shutting down"
2022-03-21T12:24:17.775Z (mindreader.geth) synchronisation failed, retrying         err="shutting down"
2022-03-21T12:24:17.775Z (mindreader.geth) synchronisation failed, retrying         err="shutting down"
2022-03-21T12:24:17.775Z (mindreader.geth) synchronisation failed, retrying         err="shutting down"
2022-03-21T12:24:17.775Z (mindreader.geth) synchronisation failed, retrying         err="shutting down"
[goes on forever]

stack trace: debug.txt

maoueh commented 2 years ago

Would have need Geth debug stack trace because it seems it's geth that is stuck, indirectly sfeth (since we are waiting for Geth to stop which will never come).

If it happens again, something that can be done is to pass --pprof --pprof.port 6065 to the geth arguments and you will be able to obtain a stack of go routine for the Geth process.

I'll close this one for now.