search

streamingfast / node-manager

Wrapping process to operate blockchain nodes.
https://dfuse.io
Apache License 2.0
7 stars 9 forks source link

mindreader gets stuck uploading blocks to s3 after network communication issue #26

Closed matthewdarwin closed 2 years ago

matthewdarwin commented 2 years ago

There are multiple reasons why s3 could be unreachable. Server overloaded, network fail, s3 unavailable etc.

With the latest dfuse / sf binaries, there is a problem where if s3 cluster becomes unavailable for some reason, mindreader will no longer upload one-block files. This was also seen on new jungle binary compiled back in November.

I managed to catch another case of the problem just now. Logs are available for your investigation (not attached here)

This was discussed a few months back. But the problem has occurred a few times now, so I'm pretty confident with the analysis. Binaries built before May 28 2021 do not have this problem. The problem was definitely there by November 26.

sduchesneau commented 2 years ago

@matthewdarwin please try tag v0.10.0-rc.5 (build incoming) and see if the issue is still there. This bug could also be the cause of https://github.com/streamingfast/sf-ethereum/issues/7

matthewdarwin commented 2 years ago

Yes, I'm pretty sure they are related. I have been paying attention to when the problem happens. I had no shutdown issues when s3 is happy.

matthewdarwin commented 2 years ago

This issue seems to be resolved. I am closing it. If I run into more problems, I will open a new issue.