search

kroma-network / go-ethereum

go-ethereum for Kroma
GNU Lesser General Public License v3.0
48 stars 23 forks source link

Bug: Memory leak #117

Closed xinzhongyoumeng closed 1 month ago

xinzhongyoumeng commented 3 months ago

System information

Geth version: v0.5.3 CL client & version: op-node v1.4.3 OS & Version: Linux Ubuntu 22.04 and we run as docker container Commit hash : (if develop)

Expected behaviour

Memory usage stabilization

Actual behaviour

We give 64G is not enough to use. The geth always oom. Today It non-self-healing. And the log is :

/opt/kromain/supervisor/geth_command.sh: line 96:   234 Killed                  ${COMMAND}
[2024-08-15 06:51:28] [geth_command.sh] exec command [/opt/kromain/core/geth --networkid=255 --datadir=/mnt/kromain/node/geth --gcmode=full --syncmode=full --db.engine=leveldb --state.scheme=hash --snapshot=false --http --http.addr=0.0.0.0 --http.port=8545 --http.vhosts=* --http.corsdomain=* --http.api=admin,eth,net,web3,personal,debug,txpool,engine,kroma --ws --ws.addr=0.0.0.0 --ws.port=8546 --ws.origins=* --ws.api=admin,eth,net,web3,personal,debug,txpool,engine,kroma --rpc.gascap=0 --rpc.txfeecap=0 --txpool.globalslots=10000 --txpool.globalqueue=5000 --authrpc.addr=127.0.0.1 --authrpc.port=8551 --authrpc.vhosts=* --authrpc.jwtsecret=/mnt/kromain/node/jwt.hex --maxpeers=200 --port=30303 --discovery.port=30303 --bootnodes=enode://acf4af39e9d9f3bbdb700315fde6e909efc714bd9a012b0f6943c4f9110008e3da2cb6c2ac9d3b6d98184a5ead57c4409be4ae7b19da1f554ceee7ba86c1fc2e@p2p-0.kroma.network:30304?discport=30303,enode://a3b6a7087b3399eefdb1ce5870aa0e58a60bfeccf3f7f7c02f5e142b1a544d19c171f7688d08db786d5e9b3ba734482bf8cf29dab7d46917b9da7f61656fde39@p2p-1.kroma.network:30304?discport=30303]
INFO [08-15|06:51:28.783] Maximum peer count                       ETH=200 total=200
INFO [08-15|06:51:28.785] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [08-15|06:51:28.785] Using leveldb as db engine
INFO [08-15|06:51:28.791] Global gas cap disabled
INFO [08-15|06:51:28.791] Initializing the KZG library             backend=gokzg
INFO [08-15|06:51:28.821] Allocated trie memory caches             clean=256.00MiB dirty=256.00MiB
INFO [08-15|06:51:29.321] Using leveldb as the backing database
INFO [08-15|06:51:29.321] Allocated cache and file handles         database=/mnt/kromain/node/geth/geth/chaindata cache=512.00MiB handles=524,288
INFO [08-15|06:51:30.068] Using LevelDB as the backing database
INFO [08-15|06:51:30.080] Opened ancient database                  database=/mnt/kromain/node/geth/geth/chaindata/ancient/chain readonly=false
INFO [08-15|06:51:30.129] State scheme set by user                 scheme=hash
INFO [08-15|06:51:30.848]
INFO [08-15|06:51:30.848] ---------------------------------------------------------------------------------------------------------------------------------------------------------
INFO [08-15|06:51:30.848] Chain ID:  255 (KromaMainnet)
INFO [08-15|06:51:30.848] Consensus: Kroma
INFO [08-15|06:51:30.848]
INFO [08-15|06:51:30.848] Pre-Merge hard forks (block based):
INFO [08-15|06:51:30.848]  - Homestead:                   #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/homestead.md)
INFO [08-15|06:51:30.848]  - Tangerine Whistle (EIP 150): #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/tangerine-whistle.md)
INFO [08-15|06:51:30.848]  - Spurious Dragon/1 (EIP 155): #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
INFO [08-15|06:51:30.848]  - Spurious Dragon/2 (EIP 158): #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
INFO [08-15|06:51:30.848]  - Byzantium:                   #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/byzantium.md)
INFO [08-15|06:51:30.848]  - Constantinople:              #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/constantinople.md)
INFO [08-15|06:51:30.848]  - Petersburg:                  #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/petersburg.md)
INFO [08-15|06:51:30.848]  - Istanbul:                    #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/istanbul.md)
INFO [08-15|06:51:30.848]  - Muir Glacier:                #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/muir-glacier.md)
INFO [08-15|06:51:30.848]  - Berlin:                      #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/berlin.md)
INFO [08-15|06:51:30.848]  - London:                      #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/london.md)
INFO [08-15|06:51:30.848]  - Arrow Glacier:               #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/arrow-glacier.md)
INFO [08-15|06:51:30.848]  - Gray Glacier:                #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/gray-glacier.md)
INFO [08-15|06:51:30.848]
INFO [08-15|06:51:30.848] Merge configured:
INFO [08-15|06:51:30.848]  - Hard-fork specification:    https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/paris.md
INFO [08-15|06:51:30.848]  - Network known to be merged: true
INFO [08-15|06:51:30.848]  - Total terminal difficulty:  0
INFO [08-15|06:51:30.848]  - Merge netsplit block:       #0
INFO [08-15|06:51:30.848]
INFO [08-15|06:51:30.848] Post-Merge hard forks (timestamp based):
INFO [08-15|06:51:30.848]  - Shanghai:                    @1708502400 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/shanghai.md)
INFO [08-15|06:51:30.848]  - Cancun:                      @1714032001
INFO [08-15|06:51:30.848]  - Regolith:                    @0
INFO [08-15|06:51:30.848]  - Canyon:                      @1708502400
INFO [08-15|06:51:30.848]  - Ecotone:                     @1714032001
INFO [08-15|06:51:30.848]
INFO [08-15|06:51:30.848] ---------------------------------------------------------------------------------------------------------------------------------------------------------
INFO [08-15|06:51:30.848]
INFO [08-15|06:51:30.850] Loaded most recent local block           number=14,897,484 hash=1c36c0..f24c53 td=0 age=8m55s
INFO [08-15|06:51:30.850] Loaded most recent local finalized block number=14,897,191 hash=b31967..a8b6af td=0 age=18m41s
WARN [08-15|06:51:30.855] Head state missing, repairing            number=14,897,484 hash=1c36c0..f24c53
INFO [08-15|07:34:06.893] Loaded most recent local block           number=14,597,407 hash=ca5b01..db62ba td=0 age=6d23h34m
INFO [08-15|07:34:06.893] Loaded most recent local finalized block number=14,897,191 hash=b31967..a8b6af td=<nil> age=1h1m17s
INFO [08-15|07:34:06.894] Initialising Ethereum protocol           network=255 dbversion=8
INFO [08-15|07:34:06.894] Initialized transaction indexer          limit=2,350,000
INFO [08-15|07:34:06.895] Loaded local transaction journal         transactions=0 dropped=0
INFO [08-15|07:34:06.895] Regenerated local transaction journal    transactions=0 accounts=0
INFO [08-15|07:34:06.896] Chain post-merge, sync via beacon client
INFO [08-15|07:34:06.896] Gasprice oracle is ignoring threshold set threshold=2
WARN [08-15|07:34:06.897] Unclean shutdown detected                booted=2024-08-08T19:29:33+0800 age=6d12h4m
WARN [08-15|07:34:06.897] Unclean shutdown detected                booted=2024-08-08T20:50:36+0800 age=6d10h43m
WARN [08-15|07:34:06.897] Unclean shutdown detected                booted=2024-08-09T10:48:30+0800 age=5d20h45m
WARN [08-15|07:34:06.897] Unclean shutdown detected                booted=2024-08-09T11:36:13+0800 age=5d19h57m
WARN [08-15|07:34:06.897] Unclean shutdown detected                booted=2024-08-09T23:31:27+0800 age=5d8h2m
WARN [08-15|07:34:06.897] Unclean shutdown detected                booted=2024-08-15T06:48:58+0800 age=45m8s
WARN [08-15|07:34:06.898] Engine API enabled                       protocol=eth
INFO [08-15|07:34:06.898] Starting peer-to-peer node               instance=Geth/v0.5.3-stable-1209a8ec/linux-amd64/go1.21.10
INFO [08-15|07:34:06.970] New local node record                    seq=1,723,108,507,072 id=669c134b493b5d8c ip=127.0.0.1 udp=30303 tcp=30303
INFO [08-15|07:34:06.970] IPC endpoint opened                      url=/mnt/kromain/node/geth/geth.ipc
INFO [08-15|07:34:06.971] Loaded JWT secret file                   path=/mnt/kromain/node/jwt.hex crc32=0x2747ce35
INFO [08-15|07:34:06.972] HTTP server started                      endpoint=[::]:8545 auth=false prefix= cors=* vhosts=*
INFO [08-15|07:34:06.972] WebSocket enabled                        url=ws://[::]:8546
INFO [08-15|07:34:06.972] WebSocket enabled                        url=ws://127.0.0.1:8551
INFO [08-15|07:34:06.972] HTTP server started                      endpoint=127.0.0.1:8551 auth=true  prefix= cors=localhost vhosts=*
INFO [08-15|07:34:06.973] Started P2P networking                   self=enode://498b1c6c0980d6be5b8368377d7b528042d5ce8b680d216589f6b07ca18ebf86e8472d9648a026c028c73b3b115201bfdbf667ec76a3bb12d28861f62e1a51fb@127.0.0.1:30303
INFO [08-15|07:34:07.895] Indexing transactions                    blocks=36920 txs=43622 tail=12,510,565 total=300,077 elapsed=1.000s
WARN [08-15|07:34:11.615] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:34:11.616] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:34:13.798] Forkchoice requested unknown head        hash=1c36c0..f24c53
INFO [08-15|07:34:15.895] Indexing transactions                    blocks=158,287 txs=580,983 tail=12,389,198 total=300,077 elapsed=9.000s
WARN [08-15|07:34:17.914] Forkchoice requested unknown head        hash=1c36c0..f24c53
INFO [08-15|07:34:23.898] Indexing transactions                    blocks=238,921 txs=1,156,350 tail=12,308,564 total=300,077 elapsed=17.003s
WARN [08-15|07:34:25.926] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:34:36.134] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:34:46.311] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:34:56.394] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:35:06.595] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:35:16.665] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:35:26.752] Forkchoice requested unknown head        hash=1c36c0..f24c53
INFO [08-15|07:35:30.597] New local node record                    seq=1,723,108,507,073 id=669c134b493b5d8c ip=47.76.189.116 udp=30303 tcp=30303
WARN [08-15|07:35:36.921] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:35:46.995] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:35:57.245] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:36:07.308] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:36:17.349] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:36:27.473] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:36:37.478] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:36:47.518] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:36:57.685] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:37:07.732] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:37:17.913] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:37:28.018] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:37:38.196] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:37:48.327] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:37:58.461] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:38:08.678] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:38:18.917] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:38:29.134] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:38:39.180] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:38:49.191] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:38:59.437] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:39:09.535] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:39:19.608] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:39:29.691] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:39:39.730] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:39:49.860] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:40:00.070] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:40:10.127] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:40:20.280] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:40:30.456] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:40:40.630] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:40:50.796] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:41:00.942] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:41:11.025] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:41:21.214] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:41:31.351] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:41:41.412] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:41:51.590] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:42:01.652] Forkchoice requested unknown head        hash=1c36c0..f24c53

Steps to reproduce the behaviour

geth:

/opt/kromain/core/geth --networkid=255 --datadir=/mnt/kromain/node/geth --gcmode=full --syncmode=full --db.engine=leveldb --state.scheme=hash --snapshot=false --http --http.addr=0.0.0.0 --http.port=8545 --http.vhosts=* --http.corsdomain=* --http.api=admin,eth,net,web3,personal,debug,txpool,engine,kroma --ws --ws.addr=0.0.0.0 --ws.port=8546 --ws.origins=* --ws.api=admin,eth,net,web3,personal,debug,txpool,engine,kroma --rpc.gascap=0 --rpc.txfeecap=0 --txpool.globalslots=10000 --txpool.globalqueue=5000 --authrpc.addr=127.0.0.1 --authrpc.port=8551 --authrpc.vhosts=* --authrpc.jwtsecret=/mnt/kromain/node/jwt.hex --maxpeers=200 --port=30303 --discovery.port=30303 --bootnodes=enode://acf4af39e9d9f3bbdb700315fde6e909efc714bd9a012b0f6943c4f9110008e3da2cb6c2ac9d3b6d98184a5ead57c4409be4ae7b19da1f554ceee7ba86c1fc2e@p2p-0.kroma.network:30304?discport=30303,enode://a3b6a7087b3399eefdb1ce5870aa0e58a60bfeccf3f7f7c02f5e142b1a544d19c171f7688d08db786d5e9b3ba734482bf8cf29dab7d46917b9da7f61656fde39@p2p-1.kroma.network:30304?discport=30303

Backtrace

[backtrace]

When submitting logs: please submit them as text and not screenshots.

seolaoh commented 3 months ago

64G is enough to operate geth for Kroma Mainnet. Is the log you provided the full log until the shutdown?

xinzhongyoumeng commented 3 months ago 
INFO [08-15|06:51:19.761] Starting work on payload                 id=0xcc1b72030a9544bb
INFO [08-15|06:51:19.771] Imported new potential chain segment     number=14,897,474 hash=b83b08..126ca2 blocks=1     txs=2       mgas=0.076  elapsed=8.034ms       mgasps=9.406    age=9m4s      triedirty=29.93GiB
INFO [08-15|06:51:19.772] Chain head was updated                   number=14,897,474 hash=b83b08..126ca2 root=1d4b60..b71c39 elapsed="94.203µs"    age=9m4s
INFO [08-15|06:51:19.784] Starting work on payload                 id=0x910081c87a78fc1c
INFO [08-15|06:51:19.799] Imported new potential chain segment     number=14,897,475 hash=41f670..69fef5 blocks=1     txs=2       mgas=0.067  elapsed=11.402ms      mgasps=5.892    age=9m2s      triedirty=29.93GiB
INFO [08-15|06:51:19.800] Chain head was updated                   number=14,897,475 hash=41f670..69fef5 root=1aa5c0..ebbc23 elapsed="131.895µs"   age=9m2s
INFO [08-15|06:51:19.820] Starting work on payload                 id=0x824e3f15da2c8946
INFO [08-15|06:51:19.838] Imported new potential chain segment     number=14,897,476 hash=2613d2..6ba871 blocks=1     txs=6       mgas=0.151  elapsed=16.933ms      mgasps=8.928    age=9m        triedirty=29.93GiB
INFO [08-15|06:51:19.840] Chain head was updated                   number=14,897,476 hash=2613d2..6ba871 root=04a582..7de84f elapsed="183.304µs"   age=9m
INFO [08-15|06:51:19.855] Starting work on payload                 id=0x11cb249c89a48257
INFO [08-15|06:51:19.869] Imported new potential chain segment     number=14,897,477 hash=7932e8..87a4c7 blocks=1     txs=3       mgas=0.170  elapsed=12.740ms      mgasps=13.355   age=8m58s     triedirty=29.93GiB
INFO [08-15|06:51:19.870] Chain head was updated                   number=14,897,477 hash=7932e8..87a4c7 root=02bef1..b284b7 elapsed="130.334µs"   age=8m58s
INFO [08-15|06:51:19.882] Starting work on payload                 id=0xf18685e6694e7a12
INFO [08-15|06:51:19.891] Imported new potential chain segment     number=14,897,478 hash=d8c890..d17318 blocks=1     txs=2       mgas=0.067  elapsed=7.721ms       mgasps=8.701    age=8m56s     triedirty=29.93GiB
INFO [08-15|06:51:19.892] Chain head was updated                   number=14,897,478 hash=d8c890..d17318 root=03acb7..ae26f3 elapsed="90.342µs"    age=8m56s
INFO [08-15|06:51:19.902] Starting work on payload                 id=0x647fe4c22b125d43
INFO [08-15|06:51:19.911] Imported new potential chain segment     number=14,897,479 hash=83ec2e..67eeac blocks=1     txs=2       mgas=0.067  elapsed=7.471ms       mgasps=8.992    age=8m54s     triedirty=29.93GiB
INFO [08-15|06:51:19.912] Chain head was updated                   number=14,897,479 hash=83ec2e..67eeac root=116b50..4f05d7 elapsed="104.574µs"   age=8m54s
INFO [08-15|06:51:19.920] Starting work on payload                 id=0x6d89b7e8f929aaa3
INFO [08-15|06:51:19.928] Imported new potential chain segment     number=14,897,480 hash=2e3d59..5b7c0e blocks=1     txs=1       mgas=0.055  elapsed=6.587ms       mgasps=8.284    age=8m52s     triedirty=29.93GiB
INFO [08-15|06:51:19.929] Chain head was updated                   number=14,897,480 hash=2e3d59..5b7c0e root=2cbbc6..421556 elapsed="83.642µs"    age=8m52s
INFO [08-15|06:51:19.944] Starting work on payload                 id=0x68e84a8ac95df0b6
INFO [08-15|06:51:19.960] Imported new potential chain segment     number=14,897,481 hash=ba2088..f52ea0 blocks=1     txs=7       mgas=0.172  elapsed=13.804ms      mgasps=12.473   age=8m50s     triedirty=29.93GiB
INFO [08-15|06:51:19.961] Chain head was updated                   number=14,897,481 hash=ba2088..f52ea0 root=2b2476..64aa9b elapsed="210.655µs"   age=8m50s
INFO [08-15|06:51:19.981] Starting work on payload                 id=0xb9a5328093e6fa63
INFO [08-15|06:51:20.001] Imported new potential chain segment     number=14,897,482 hash=e3cebc..1d9411 blocks=1     txs=4       mgas=0.482  elapsed=18.249ms      mgasps=26.396   age=8m49s     triedirty=29.93GiB
INFO [08-15|06:51:20.002] Chain head was updated                   number=14,897,482 hash=e3cebc..1d9411 root=2ffed1..abf063 elapsed="166.445µs"   age=8m49s
INFO [08-15|06:51:20.020] Starting work on payload                 id=0x3247966b8b4db3df
INFO [08-15|06:51:20.029] Imported new potential chain segment     number=14,897,483 hash=3827d3..0871cb blocks=1     txs=2       mgas=0.098  elapsed=8.038ms       mgasps=12.133   age=8m47s     triedirty=29.93GiB
INFO [08-15|06:51:20.030] Chain head was updated                   number=14,897,483 hash=3827d3..0871cb root=0024a8..f644b2 elapsed="107.623µs"   age=8m47s
INFO [08-15|06:51:20.058] Starting work on payload                 id=0x3b607b0358fdc20f
INFO [08-15|06:51:20.079] Imported new potential chain segment     number=14,897,484 hash=1c36c0..f24c53 blocks=1     txs=2       mgas=0.488  elapsed=19.343ms      mgasps=25.211   age=8m45s     triedirty=29.93GiB
INFO [08-15|06:51:20.080] Chain head was updated                   number=14,897,484 hash=1c36c0..f24c53 root=04e7c1..e631c0 elapsed="179.574µs"   age=8m45s
INFO [08-15|06:51:20.096] Starting work on payload                 id=0x6dd468c65fe907ac
/opt/kromain/supervisor/geth_command.sh: line 96:   234 Killed                  ${COMMAND}
[2024-08-15 06:51:28] [geth_command.sh] exec command [/opt/kromain/core/geth --networkid=255 --datadir=/mnt/kromain/node/geth --gcmode=full --syncmode=full --db.engine=leveldb --state.scheme=hash --snapshot=false --http --http.addr=0.0.0.0 --http.port=8545 --http.vhosts=* --http.corsdomain=* --http.api=admin,eth,net,web3,personal,debug,txpool,engine,kroma --ws --ws.addr=0.0.0.0 --ws.port=8546 --ws.origins=* --ws.api=admin,eth,net,web3,personal,debug,txpool,engine,kroma --rpc.gascap=0 --rpc.txfeecap=0 --txpool.globalslots=10000 --txpool.globalqueue=5000 --authrpc.addr=127.0.0.1 --authrpc.port=8551 --authrpc.vhosts=* --authrpc.jwtsecret=/mnt/kromain/node/jwt.hex --maxpeers=200 --port=30303 --discovery.port=30303 --bootnodes=enode://acf4af39e9d9f3bbdb700315fde6e909efc714bd9a012b0f6943c4f9110008e3da2cb6c2ac9d3b6d98184a5ead57c4409be4ae7b19da1f554ceee7ba86c1fc2e@p2p-0.kroma.network:30304?discport=30303,enode://a3b6a7087b3399eefdb1ce5870aa0e58a60bfeccf3f7f7c02f5e142b1a544d19c171f7688d08db786d5e9b3ba734482bf8cf29dab7d46917b9da7f61656fde39@p2p-1.kroma.network:30304?discport=30303]
INFO [08-15|06:51:28.783] Maximum peer count                       ETH=200 total=200
INFO [08-15|06:51:28.785] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [08-15|06:51:28.785] Using leveldb as db engine
INFO [08-15|06:51:28.791] Global gas cap disabled
INFO [08-15|06:51:28.791] Initializing the KZG library             backend=gokzg
INFO [08-15|06:51:28.821] Allocated trie memory caches             clean=256.00MiB dirty=256.00MiB
INFO [08-15|06:51:29.321] Using leveldb as the backing database
INFO [08-15|06:51:29.321] Allocated cache and file handles         database=/mnt/kromain/node/geth/geth/chaindata cache=512.00MiB handles=524,288
INFO [08-15|06:51:30.068] Using LevelDB as the backing database
INFO [08-15|06:51:30.080] Opened ancient database                  database=/mnt/kromain/node/geth/geth/chaindata/ancient/chain readonly=false
INFO [08-15|06:51:30.129] State scheme set by user                 scheme=hash
INFO [08-15|06:51:30.848]
INFO [08-15|06:51:30.848] ---------------------------------------------------------------------------------------------------------------------------------------------------------
INFO [08-15|06:51:30.848] Chain ID:  255 (KromaMainnet)
INFO [08-15|06:51:30.848] Consensus: Kroma
INFO [08-15|06:51:30.848]
INFO [08-15|06:51:30.848] Pre-Merge hard forks (block based):
INFO [08-15|06:51:30.848]  - Homestead:                   #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/homestead.md)
INFO [08-15|06:51:30.848]  - Tangerine Whistle (EIP 150): #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/tangerine-whistle.md)
INFO [08-15|06:51:30.848]  - Spurious Dragon/1 (EIP 155): #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
INFO [08-15|06:51:30.848]  - Spurious Dragon/2 (EIP 158): #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
INFO [08-15|06:51:30.848]  - Byzantium:                   #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/byzantium.md)
INFO [08-15|06:51:30.848]  - Constantinople:              #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/constantinople.md)
INFO [08-15|06:51:30.848]  - Petersburg:                  #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/petersburg.md)
INFO [08-15|06:51:30.848]  - Istanbul:                    #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/istanbul.md)
INFO [08-15|06:51:30.848]  - Muir Glacier:                #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/muir-glacier.md)
INFO [08-15|06:51:30.848]  - Berlin:                      #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/berlin.md)
INFO [08-15|06:51:30.848]  - London:                      #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/london.md)
INFO [08-15|06:51:30.848]  - Arrow Glacier:               #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/arrow-glacier.md)
INFO [08-15|06:51:30.848]  - Gray Glacier:                #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/gray-glacier.md)
INFO [08-15|06:51:30.848]
INFO [08-15|06:51:30.848] Merge configured:
INFO [08-15|06:51:30.848]  - Hard-fork specification:    https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/paris.md
INFO [08-15|06:51:30.848]  - Network known to be merged: true
INFO [08-15|06:51:30.848]  - Total terminal difficulty:  0
INFO [08-15|06:51:30.848]  - Merge netsplit block:       #0
INFO [08-15|06:51:30.848]
INFO [08-15|06:51:30.848] Post-Merge hard forks (timestamp based):
INFO [08-15|06:51:30.848]  - Shanghai:                    @1708502400 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/shanghai.md)
INFO [08-15|06:51:30.848]  - Cancun:                      @1714032001
INFO [08-15|06:51:30.848]  - Regolith:                    @0
INFO [08-15|06:51:30.848]  - Canyon:                      @1708502400
INFO [08-15|06:51:30.848]  - Ecotone:                     @1714032001
INFO [08-15|06:51:30.848]
INFO [08-15|06:51:30.848] ---------------------------------------------------------------------------------------------------------------------------------------------------------
INFO [08-15|06:51:30.848]
INFO [08-15|06:51:30.850] Loaded most recent local block           number=14,897,484 hash=1c36c0..f24c53 td=0 age=8m55s
INFO [08-15|06:51:30.850] Loaded most recent local finalized block number=14,897,191 hash=b31967..a8b6af td=0 age=18m41s
WARN [08-15|06:51:30.855] Head state missing, repairing            number=14,897,484 hash=1c36c0..f24c53
INFO [08-15|07:34:06.893] Loaded most recent local block           number=14,597,407 hash=ca5b01..db62ba td=0 age=6d23h34m
INFO [08-15|07:34:06.893] Loaded most recent local finalized block number=14,897,191 hash=b31967..a8b6af td=<nil> age=1h1m17s
INFO [08-15|07:34:06.894] Initialising Ethereum protocol           network=255 dbversion=8
INFO [08-15|07:34:06.894] Initialized transaction indexer          limit=2,350,000
INFO [08-15|07:34:06.895] Loaded local transaction journal         transactions=0 dropped=0
INFO [08-15|07:34:06.895] Regenerated local transaction journal    transactions=0 accounts=0
INFO [08-15|07:34:06.896] Chain post-merge, sync via beacon client
INFO [08-15|07:34:06.896] Gasprice oracle is ignoring threshold set threshold=2
WARN [08-15|07:34:06.897] Unclean shutdown detected                booted=2024-08-08T19:29:33+0800 age=6d12h4m
WARN [08-15|07:34:06.897] Unclean shutdown detected                booted=2024-08-08T20:50:36+0800 age=6d10h43m
WARN [08-15|07:34:06.897] Unclean shutdown detected                booted=2024-08-09T10:48:30+0800 age=5d20h45m
WARN [08-15|07:34:06.897] Unclean shutdown detected                booted=2024-08-09T11:36:13+0800 age=5d19h57m
WARN [08-15|07:34:06.897] Unclean shutdown detected                booted=2024-08-09T23:31:27+0800 age=5d8h2m
WARN [08-15|07:34:06.897] Unclean shutdown detected                booted=2024-08-15T06:48:58+0800 age=45m8s
WARN [08-15|07:34:06.898] Engine API enabled                       protocol=eth
INFO [08-15|07:34:06.898] Starting peer-to-peer node               instance=Geth/v0.5.3-stable-1209a8ec/linux-amd64/go1.21.10
INFO [08-15|07:34:06.970] New local node record                    seq=1,723,108,507,072 id=669c134b493b5d8c ip=127.0.0.1 udp=30303 tcp=30303
INFO [08-15|07:34:06.970] IPC endpoint opened                      url=/mnt/kromain/node/geth/geth.ipc
INFO [08-15|07:34:06.971] Loaded JWT secret file                   path=/mnt/kromain/node/jwt.hex crc32=0x2747ce35
INFO [08-15|07:34:06.972] HTTP server started                      endpoint=[::]:8545 auth=false prefix= cors=* vhosts=*
INFO [08-15|07:34:06.972] WebSocket enabled                        url=ws://[::]:8546
INFO [08-15|07:34:06.972] WebSocket enabled                        url=ws://127.0.0.1:8551
INFO [08-15|07:34:06.972] HTTP server started                      endpoint=127.0.0.1:8551 auth=true  prefix= cors=localhost vhosts=*
INFO [08-15|07:34:06.973] Started P2P networking                   self=enode://498b1c6c0980d6be5b8368377d7b528042d5ce8b680d216589f6b07ca18ebf86e8472d9648a026c028c73b3b115201bfdbf667ec76a3bb12d28861f62e1a51fb@127.0.0.1:30303
INFO [08-15|07:34:07.895] Indexing transactions                    blocks=36920 txs=43622 tail=12,510,565 total=300,077 elapsed=1.000s
WARN [08-15|07:34:11.615] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:34:11.616] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:34:13.798] Forkchoice requested unknown head        hash=1c36c0..f24c53
INFO [08-15|07:34:15.895] Indexing transactions                    blocks=158,287 txs=580,983 tail=12,389,198 total=300,077 elapsed=9.000s
WARN [08-15|07:34:17.914] Forkchoice requested unknown head        hash=1c36c0..f24c53
INFO [08-15|07:34:23.898] Indexing transactions                    blocks=238,921 txs=1,156,350 tail=12,308,564 total=300,077 elapsed=17.003s
WARN [08-15|07:34:25.926] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:34:36.134] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:34:46.311] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:34:56.394] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:35:06.595] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:35:16.665] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:35:26.752] Forkchoice requested unknown head        hash=1c36c0..f24c53
INFO [08-15|07:35:30.597] New local node record                    seq=1,723,108,507,073 id=669c134b493b5d8c ip=47.76.189.116 udp=30303 tcp=30303
WARN [08-15|07:35:36.921] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:35:46.995] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:35:57.245] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:36:07.308] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:36:17.349] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:36:27.473] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:36:37.478] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:36:47.518] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:36:57.685] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:37:07.732] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:37:17.913] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:37:28.018] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:37:38.196] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:37:48.327] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:37:58.461] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:38:08.678] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:38:18.917] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:38:29.134] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:38:39.180] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:38:49.191] Forkchoice requested unknown head        hash=1c36c0..f24c53
WARN [08-15|07:38:59.437] Forkchoice requested unknown head        hash=1c36c0..f24c53
...
so on
xinzhongyoumeng commented 3 months ago

image

Pangssu commented 3 months ago

Try changing the gcmode flag to archive in the geth command.

xinzhongyoumeng commented 3 months ago

Try changing the gcmode flag to archive in the geth command.

Thank you, I will try it.

northwestnodes-eric commented 2 months ago

We have the same issue when running on gcmode full. The OOM goes away when switching to archive.

xinzhongyoumeng commented 1 month ago

在 gcmode full 上运行时，我们遇到了同样的问题。切换到存档时，OOM 消失。

ok, thank you, you are right.