blutgang doesn't finalize blocks with reth and nethermind

nidhi-singh02 commented 4 months ago

Describe the bug I am running a custom network with multiple EL clients and a CL. Blutgang works with geth alone, but doesn't work with reth and nethermind even solo.

To Reproduce Steps to reproduce the behavior: Run a network with EL and CL clients and add blutgang as load balancer for JSON RPC endpoints.

Expected behavior My blocks are not getting finalised via testing when using solo reth and solo nethermind.

Specs:

OS: Running in docker containers via kurtosis
Kernel: [e.g. Linux 6.2.0-27-generic]
Blutgang version: Used docker image both makemake1337/blutgang:latest and makemake1337/blutgang:master
- Kurtosis version Blutgang options: Kept all the blutgang options by default as in kurtosis ethereum package. This is how config toml template looks like https://github.com/kurtosis-tech/ethereum-package/blob/main/static_files/blutgang-config/config.toml.tmpl
DB mode: HighThroughput
DB compression: no
Cache capacity: 1000000000
MA length: 100
RPCs used: reth and nethermind

Additional context I also tried playing around with some parameters of toml file such as setting readonly flag in admin section to false and increasing the expected_block_time, health_check_ttl and ttl, but no luck.

makemake-kbo commented 4 months ago

That shouldn't be happening. Internally Blutgang only uses methods supported by all EL clients.

Try the 0.3.5-debug-verbose docker image.

It should have more logging outputs that should help us debug this.

nidhi-singh02 commented 4 months ago

That shouldn't be happening. Internally Blutgang only uses methods supported by all EL clients.

Try the 0.3.5-debug-verbose docker image.

It should have more logging outputs that should help us debug this.

Sure, Let me run using this docker image. Thank you for pushing the image :)

Btw I am curious, is logging level option not available in the master image ?

makemake-kbo commented 4 months ago

That shouldn't be happening. Internally Blutgang only uses methods supported by all EL clients. Try the 0.3.5-debug-verbose docker image. It should have more logging outputs that should help us debug this.

Sure, Let me run using this docker image. Thank you for pushing the image :)

Btw I am curious, is logging level option not available in the master image ?

No, its a compile time feature flag.

nidhi-singh02 commented 4 months ago

I ran with the docker image, seems the chain head in blutgang is getting updated, not sure why BlockByNumber method from geth module does not shows the updated block, it is stuck at Block number 1 always, whereas BlockNumber works fine.

makemake-kbo commented 4 months ago

I ran with the docker image, seems the chain head in blutgang is getting updated, not sure why BlockByNumber method from geth module does not shows the updated block, it is stuck at Block number 1 always, whereas BlockNumber works fine.

smells like a cache issue, try clearing it and see if that fixes it.

nidhi-singh02 commented 3 months ago

Are you referring to do_clear for cleaning the cache ?

makemake-kbo commented 3 months ago

Are you referring to do_clear for cleaning the cache ? @nidhi-singh02

yes, either that or deleting the database folder.

nidhi-singh02 commented 3 months ago

I have set do_clear parameter to true in the toml file, still same issue with BlockByNumber using reth full node or nethermind full node. Only geth full node works.

makemake-kbo commented 2 months ago

please try the 0.4.0 branch, if possible with debug-verbose on, or the 0.3.5-debug-verbose docker image. its hard to debug this issue without this context.

rainshowerLabs / blutgang

blutgang doesn't finalize blocks with reth and nethermind #82