RPC method handler panic when querying eth_getBalance

[alecalve]

System information

This issue appeared after upgrading from 1.2.8 to 1.2.9

OS & Version: Linux

Erigon Command (with flags/config):

            - --datadir=/tank
            - --chain=bsc
            - --http.addr=0.0.0.0
            - --http.vhosts=*
            - --http.corsdomain=*
            - --http.port=8565
            - --port=30333
            - --torrent.download.rate=100mb
            - --http.api=eth,debug,net,web3,trace,erigon
             --rpc.batch.concurrency=4
             --rpc.batch.limit=1000
             --db.size.limit=8TB
             --txpool.disable
             --p2p.protocol=68

Consensus Layer:

Chain/Network: BSC Mainnet

Expected behaviour

Calls to eth_getBalance would return

Actual behaviour

[EROR] [05-27|08:47:42.308] RPC method eth_getBalance crashed: runtime error: invalid memory address or nil pointer dereference
[service.go:219 panic.go:914 panic.go:261 signal_unix.go:861 eth_accounts.go:35 value.go:596 value.go:380 service.go:224 handler.go:532 handler.go:482 handler.go:421 handler.go:204 asm_amd64.s:1650]
[WARN] [05-27|08:47:42.308] [rpc] served                             conn=[xxx]:45346 method=eth_getBalance reqid=128880254 t=154.379µs err="method handler crashed"
[EROR] [05-27|08:47:42.308] RPC method eth_getBalance crashed: runtime error: invalid memory address or nil pointer dereference
[service.go:219 panic.go:914 panic.go:261 signal_unix.go:861 eth_accounts.go:35 value.go:596 value.go:380 service.go:224 handler.go:532 handler.go:482 handler.go:421 handler.go:204 asm_amd64.s:1650]
[WARN] [05-27|08:47:42.308] [rpc] served                             conn=[xxx]:45346 method=eth_getBalance reqid=128880255 t=133.569µs err="method handler crashed"
[EROR] [05-27|08:47:42.308] RPC method eth_getBalance crashed: runtime error: invalid memory address or nil pointer dereference
[service.go:219 panic.go:914 panic.go:261 signal_unix.go:861 eth_accounts.go:35 value.go:596 value.go:380 service.go:224 handler.go:532 handler.go:482 handler.go:421 handler.go:204 asm_amd64.s:1650]
[WARN] [05-27|08:47:42.309] [rpc] served                             conn=[xxx]:45346 method=eth_getBalance reqid=128880256 t=264.053µs err="method handler crashed"

[alecalve]

The errors goes away when downgrading to 1.2.8

[blxdyx]

Show you command? Never reproduce it.

[MatusKysel]

@alecalve also, does it happens for one account or on all?

[alecalve]

Could it be because the node is running into MDBX_MAP_FULL: Environment mapsize limit reached errors?

[blxdyx]

8TB is may not enough, the default value 12TB

[alecalve]

I tried increasing db.size.limit to 12TB but I get this error message:

mdbx_env_open: MDBX_TOO_LARGE: Database is too large for current system, e.g. could NOT be mapped into RAM

[blxdyx]

Try with 10TB？

[alecalve]

Neither 10TB or 9TB worked.

[alecalve]

Running mdbx_stat, the db is at its max capacity with 8TB and 4096 page size.

[blxdyx]

You need reset the db pagesize to 16kb. The only way is: https://github.com/node-real/bsc-erigon/tree/main/cmd/integration#copy-data-to-another-db

[alecalve]

You need reset the db pagesize to 16kb. The only way is: https://github.com/node-real/bsc-erigon/tree/main/cmd/integration#copy-data-to-another-db

What goes into choosing 16kB as page size? What about larger ones?

[avinashbo]

Reproduced with 12TB and 16KB page size

- --db.pagesize=16384
- --db.size.limit=12TB

[blxdyx]

You need reset the db pagesize to 16kb. The only way is: https://github.com/node-real/bsc-erigon/tree/main/cmd/integration#copy-data-to-another-db

What goes into choosing 16kB as page size? What about larger ones?

If expecting DB-size > 8Tb then set pageSize >= 8Kb. 16KB is work fine with me