search

ethereum / go-ethereum

Go implementation of the Ethereum protocol
https://geth.ethereum.org
GNU Lesser General Public License v3.0
47.61k stars 20.16k forks source link

DB increases despite scheme=path #28758

Closed tchaps6 closed 10 months ago

tchaps6 commented 10 months ago

System information

Geth version: 1.13.7-stable CL client & version: prysm@v4.1.1 OS & Version: Linux Commit hash : (if develop)

Expected behaviour

DB does not increase after resyncing with scheme=path setting

Actual behaviour

DB increased by 30GB+ since resync (7 days ago)

Service file: [Unit] Description=Geth Execution Client (Mainnet) After=network.target Wants=network.target [Service] User=geth Group=geth Type=simple Restart=always RestartSec=3 ExecStart=/usr/local/bin/geth \ --mainnet \ --metrics \ --datadir=/var/lib/geth \ --authrpc.jwtsecret /var/lib/jwtsecret/jwt.hex \ --state.scheme=path [Install] WantedBy=default.target

Startup log:

Jan 02 21:25:14 server-address systemd[1]: Started Geth Execution Client (Mainnet). Jan 02 21:25:14 server-address geth[2418330]: INFO [01-02|21:25:14.460] Starting Geth on Ethereum mainnet... Jan 02 21:25:14 server-address geth[2418330]: INFO [01-02|21:25:14.462] Bumping default cache on mainnet provided=1024 updated=4096 Jan 02 21:25:14 server-address geth[2418330]: INFO [01-02|21:25:14.462] Enabling metrics collection Jan 02 21:25:14 server-address geth[2418330]: INFO [01-02|21:25:14.468] Maximum peer count ETH=50 total=50 Jan 02 21:25:14 server-address geth[2418330]: INFO [01-02|21:25:14.472] Smartcard socket not found, disabling err="stat /run/pcscd/pcscd.comm: no such file or directory" Jan 02 21:25:14 server-address geth[2418330]: INFO [01-02|21:25:14.481] Set global gas cap cap=50,000,000 Jan 02 21:25:14 server-address geth[2418330]: INFO [01-02|21:25:14.523] Initializing the KZG library backend=gokzg Jan 02 21:25:14 server-address geth[2418330]: INFO [01-02|21:25:14.610] Allocated trie memory caches clean=614.00MiB dirty=1024.00MiB Jan 02 21:25:14 server-address geth[2418330]: INFO [01-02|21:25:14.827] Using pebble as the backing database Jan 02 21:25:14 server-address geth[2418330]: INFO [01-02|21:25:14.828] Allocated cache and file handles database=/var/lib/geth/geth/chaindata cache=2.00GiB handles=262,144 Jan 02 21:25:30 server-address geth[2418330]: INFO [01-02|21:25:30.660] Opened ancient database database=/var/lib/geth/geth/chaindata/ancient/chain readonly=false Jan 02 21:25:30 server-address geth[2418330]: INFO [01-02|21:25:30.667] State scheme set by user scheme=path Jan 02 21:25:30 server-address geth[2418330]: INFO [01-02|21:25:30.677] Initialising Ethereum protocol network=1 dbversion=8 Jan 02 21:25:30 server-address geth[2418330]: WARN [01-02|21:25:30.678] Sanitizing invalid node buffer size provided=1024.00MiB updated=256.00MiB Jan 02 21:25:34 server-address geth[2418330]: INFO [01-02|21:25:34.641] Opened ancient database database=/var/lib/geth/geth/chaindata/ancient/state readonly=false Jan 02 21:25:34 server-address geth[2418330]: WARN [01-02|21:25:34.682] Path-based state scheme is an experimental feature Jan 02 21:25:34 server-address geth[2418330]: INFO [01-02|21:25:34.995] Jan 02 21:25:34 server-address geth[2418330]: INFO [01-02|21:25:34.995] --------------------------------------------------------------------------------------------------------------------------------------------------------- Jan 02 21:25:34 server-address geth[2418330]: INFO [01-02|21:25:34.995] Chain ID: 1 (mainnet) Jan 02 21:25:34 server-address geth[2418330]: INFO [01-02|21:25:34.995] Consensus: Beacon (proof-of-stake), merged from Ethash (proof-of-work) Jan 02 21:25:34 server-address geth[2418330]: INFO [01-02|21:25:34.995] Jan 02 21:25:34 server-address geth[2418330]: INFO [01-02|21:25:34.995] Pre-Merge hard forks (block based): Jan 02 21:25:34 server-address geth[2418330]: INFO [01-02|21:25:34.995] - Homestead: #1150000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/homestead.md) Jan 02 21:25:34 server-address geth[2418330]: INFO [01-02|21:25:34.995] - DAO Fork: #1920000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/dao-fork.md) Jan 02 21:25:34 server-address geth[2418330]: INFO [01-02|21:25:34.995] - Tangerine Whistle (EIP 150): #2463000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/tangerine-whistle.md) Jan 02 21:25:34 server-address geth[2418330]: INFO [01-02|21:25:34.995] - Spurious Dragon/1 (EIP 155): #2675000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md) Jan 02 21:25:34 server-address geth[2418330]: INFO [01-02|21:25:34.995] - Spurious Dragon/2 (EIP 158): #2675000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md) Jan 02 21:25:34 server-address geth[2418330]: INFO [01-02|21:25:34.995] - Byzantium: #4370000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/byzantium.md) Jan 02 21:25:34 server-address geth[2418330]: INFO [01-02|21:25:34.995] - Constantinople: #7280000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/constantinople.md) Jan 02 21:25:34 server-address geth[2418330]: INFO [01-02|21:25:34.995] - Petersburg: #7280000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/petersburg.md) Jan 02 21:25:34 server-address geth[2418330]: INFO [01-02|21:25:34.995] - Istanbul: #9069000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/istanbul.md) Jan 02 21:25:35 server-address geth[2418330]: INFO [01-02|21:25:34.995] - Muir Glacier: #9200000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/muir-glacier.md) Jan 02 21:25:35 server-address geth[2418330]: INFO [01-02|21:25:34.995] - Berlin: #12244000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/berlin.md) Jan 02 21:25:35 server-address geth[2418330]: INFO [01-02|21:25:34.995] - London: #12965000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/london.md) Jan 02 21:25:35 server-address geth[2418330]: INFO [01-02|21:25:34.995] - Arrow Glacier: #13773000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/arrow-glacier.md) Jan 02 21:25:35 server-address geth[2418330]: INFO [01-02|21:25:34.995] - Gray Glacier: #15050000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/gray-glacier.md) Jan 02 21:25:35 server-address geth[2418330]: INFO [01-02|21:25:34.995] Jan 02 21:25:35 server-address geth[2418330]: INFO [01-02|21:25:34.995] Merge configured: Jan 02 21:25:35 server-address geth[2418330]: INFO [01-02|21:25:34.995] - Hard-fork specification: https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/paris.md Jan 02 21:25:35 server-address geth[2418330]: INFO [01-02|21:25:34.995] - Network known to be merged: true Jan 02 21:25:35 server-address geth[2418330]: INFO [01-02|21:25:34.995] - Total terminal difficulty: 58750000000000000000000 Jan 02 21:25:35 server-address geth[2418330]: INFO [01-02|21:25:34.995] Jan 02 21:25:35 server-address geth[2418330]: INFO [01-02|21:25:34.995] Post-Merge hard forks (timestamp based): Jan 02 21:25:35 server-address geth[2418330]: INFO [01-02|21:25:34.995] - Shanghai: @1681338455 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/shanghai.md) Jan 02 21:25:35 server-address geth[2418330]: INFO [01-02|21:25:34.995] Jan 02 21:25:35 server-address geth[2418330]: INFO [01-02|21:25:34.995] --------------------------------------------------------------------------------------------------------------------------------------------------------- Jan 02 21:25:35 server-address geth[2418330]: INFO [01-02|21:25:34.995] Jan 02 21:25:35 server-address geth[2418330]: INFO [01-02|21:25:35.000] Loaded most recent local block number=18,922,042 hash=980906..023ce1 td=58,750,003,716,598,352,816,469 age=3m36s Jan 02 21:25:35 server-address geth[2418330]: INFO [01-02|21:25:35.000] Loaded most recent local finalized block number=18,921,967 hash=3927f5..9e9664 td=58,750,003,716,598,352,816,469 age=18m48s Jan 02 21:25:35 server-address geth[2418330]: INFO [01-02|21:25:35.003] Loaded last snap-sync pivot marker number=18,835,242 Jan 02 21:25:35 server-address geth[2418330]: INFO [01-02|21:25:35.731] Initialized transaction indexer limit=2,350,000 Jan 02 21:25:35 server-address geth[2418330]: INFO [01-02|21:25:35.738] Loaded local transaction journal transactions=0 dropped=0 Jan 02 21:25:35 server-address geth[2418330]: INFO [01-02|21:25:35.738] Regenerated local transaction journal transactions=0 accounts=0 Jan 02 21:25:35 server-address geth[2418330]: WARN [01-02|21:25:35.761] Switch sync mode from snap sync to full sync reason="snap sync complete" Jan 02 21:25:35 server-address geth[2418330]: INFO [01-02|21:25:35.761] Chain post-merge, sync via beacon client Jan 02 21:25:35 server-address geth[2418330]: INFO [01-02|21:25:35.762] Gasprice oracle is ignoring threshold set threshold=2 Jan 02 21:25:35 server-address geth[2418330]: WARN [01-02|21:25:35.777] Engine API enabled protocol=eth Jan 02 21:25:35 server-address geth[2418330]: INFO [01-02|21:25:35.777] Starting peer-to-peer node instance=Geth/v1.13.7-stable/linux-amd64/go1.21.3 Jan 02 21:25:35 server-address geth[2418330]: INFO [01-02|21:25:35.917] New local node record seq=1,703,139,276,590 id=66325a6cadbb9768 ip=127.0.0.1 udp=30303 tcp=30303 Jan 02 21:25:35 server-address geth[2418330]: INFO [01-02|21:25:35.918] Started P2P networking self=enode://890787ae21fe01224ca3d87fc3bce735df6cf31e27364aaf998af97f4845a2d64c84c7f36bdcab89eef6cf40fce30c995914bb737bba8e324b9c2f91a2028b5f@127.0.0.1:30303 Jan 02 21:25:35 server-address geth[2418330]: INFO [01-02|21:25:35.920] IPC endpoint opened url=/var/lib/geth/geth.ipc Jan 02 21:25:35 server-address geth[2418330]: INFO [01-02|21:25:35.920] Loaded JWT secret file path=/var/lib/jwtsecret/jwt.hex crc32=0x402b4eee Jan 02 21:25:35 server-address geth[2418330]: INFO [01-02|21:25:35.922] WebSocket enabled url=ws://127.0.0.1:8551 Jan 02 21:25:35 server-address geth[2418330]: INFO [01-02|21:25:35.922] HTTP server started endpoint=127.0.0.1:8551 auth=true prefix= cors=localhost vhosts=localhost Jan 02 21:25:39 server-address geth[2418330]: INFO [01-02|21:25:39.425] New local node record seq=1,703,139,276,591 id=66325a6cadbb9768 ip=62.171.152.51 udp=30303 tcp=30303 Jan 02 21:25:39 server-address geth[2418330]: INFO [01-02|21:25:39.770] Imported new potential chain segment number=18,922,043 hash=e5757d..4d631b blocks=1 txs=142 mgas=9.528 elapsed=1.409s mgasps=6.758 age=3m28s snapdiffs=6.31MiB triediffs=180.85MiB triedirty=54.13MiB Jan 02 21:25:40 server-address geth[2418330]: INFO [01-02|21:25:40.191] Chain head was updated number=18,922,043 hash=e5757d..4d631b root=a13390..62a9fc elapsed=5.078837ms age=3m29s

rjl493456442 commented 10 months ago

DB increased by 30GB+ since resync (7 days ago)

The path scheme can effectively prune the stale state data, but the chain data(blocks, receipts, etc) is still growing as before. Besides, the extra 30GB may contain some uncompacted stale data in database as we are using the LSM storage engine which does the lazy compaction.

If you are really interested the storage growth, you can inspect your node by:

You can compare the result to see the real state growth!

tchaps6 commented 10 months ago

Okay thanks for the explanation!

Glad to know this might be expected behaviour then. In that case, might be time to upgrade my 1.6TB to 2TB given 94% of the disk is full now.

rjl493456442 commented 10 months ago

AFAIK, the current size of Geth node is roughly 1TB, less than the number you gave. Probably you also host something else? (e.g. beacon chain data)

tchaps6 commented 10 months ago

Here the result, you were right, its just under a TB!

I also host a prysm beacon and validator, could they account for 400 GB? Is there a similar command to check on those?

When running sudo du -h /var/lib/, I can see "486G /var/lib/prysm/beacon/beaconchaindata" - is this alarming? Shall I report this somewhere?

+-----------------------+-------------------------+------------+------------+
|       DATABASE        |        CATEGORY         |    SIZE    |   ITEMS    |
+-----------------------+-------------------------+------------+------------+
| Key-Value store       | Headers                 | 52.54 MiB  |      90437 |
| Key-Value store       | Bodies                  | 13.60 GiB  |      90437 |
| Key-Value store       | Receipt lists           | 5.80 GiB   |      90437 |
| Key-Value store       | Difficulties            | 4.57 MiB   |      90437 |
| Key-Value store       | Block number->hash      | 3.61 MiB   |      90004 |
| Key-Value store       | Block hash->number      | 740.03 MiB |   18926284 |
| Key-Value store       | Transaction index       | 12.37 GiB  |  360144851 |
| Key-Value store       | Bloombit index          | 3.75 GiB   |    9466381 |
| Key-Value store       | Contract codes          | 7.48 GiB   |    1150699 |
| Key-Value store       | Hash trie nodes         | 0.00 B     |          0 |
| Key-Value store       | Path trie state lookups | 3.52 MiB   |      90003 |
| Key-Value store       | Path trie account nodes | 36.05 GiB  |  310751967 |
| Key-Value store       | Path trie storage nodes | 148.57 GiB | 1483449624 |
| Key-Value store       | Trie preimages          | 0.00 B     |          0 |
| Key-Value store       | Account snapshot        | 10.59 GiB  |  229875581 |
| Key-Value store       | Storage snapshot        | 78.62 GiB  | 1095208137 |
| Key-Value store       | Beacon sync headers     | 575.00 B   |          1 |
| Key-Value store       | Clique snapshots        | 0.00 B     |          0 |
| Key-Value store       | Singleton metadata      | 207.85 MiB |         19 |
| Light client          | CHT trie nodes          | 0.00 B     |          0 |
| Light client          | Bloom trie nodes        | 0.00 B     |          0 |
| Ancient store (Chain) | Headers                 | 8.69 GiB   |   18835848 |
| Ancient store (Chain) | Hashes                  | 682.60 MiB |   18835848 |
| Ancient store (Chain) | Bodies                  | 465.00 GiB |   18835848 |
| Ancient store (Chain) | Receipts                | 171.48 GiB |   18835848 |
| Ancient store (Chain) | Diffs                   | 298.38 MiB |   18835848 |
| Ancient store (State) | Storage.Index           | 1.17 GiB   |      90000 |
| Ancient store (State) | Account.Data            | 915.60 MiB |      90000 |
| Ancient store (State) | Storage.Data            | 322.59 MiB |      90000 |
| Ancient store (State) | History.Meta            | 6.78 MiB   |      90000 |
| Ancient store (State) | Account.Index           | 809.88 MiB |      90000 |
+-----------------------+-------------------------+------------+------------+
|                                  TOTAL          | 967.11 GIB |            |
+-----------------------+-------------------------+------------+------------+
rjl493456442 commented 10 months ago

Yeah, please report to prysm repo!

I will close this ticket as the node size is expected.