search

streamingfast / merger

Apache License 2.0
4 stars 5 forks source link

Merger is slow to start when there are many one-block files #23

Closed matthewdarwin closed 2 years ago

matthewdarwin commented 2 years ago

Merger is slow to start when there are many one-block files piled up in s3 bucket. (reading list from s3 is not fast)

In older implementations of merger it would read the first 2000 files (approximately) and take action from that. There is no need to read all the files.

sduchesneau commented 2 years ago

https://github.com/streamingfast/merger/commit/99f16fc1461a70961f60fb8967f5ac429bb5e19a and https://github.com/streamingfast/merger/commit/fcb32d7ccdc0c2bbb02eacab3ce975163baf3d9a

should fix this.

pushed in https://github.com/streamingfast/firehose-ethereum/commit/59e97909a27e990111d17b7b86a57e81e259d2a4

sduchesneau commented 2 years ago

Also, your specific problem is probably related to https://github.com/streamingfast/go-ethereum/issues/1 if the one-block-files have a libnum stuck way in the past, it won't progress until that libnum moves up.

The workaround for the geth issue should get you going, and those two commits linked above should also improve performance.

matthewdarwin commented 2 years ago

Latest merger seems fine with lots of one-block files.