Disk Usage fast growth

[jinglics]

• OpenEthereum version: Parity Ethereum Client. version Parity-Ethereum/v2.7.2-stable-2662d19-20200206/x86_64-unknown-linux-gnu/rustc1.41.0

• Operating system: Linux

• Installation: homebrew / one-line installer / built from source

• Fully synchronized: yes

• Network: ethereum

• Restarted: yes

I run parity to get the ethereum mainnet and it synced in more one day. Now it caught up with the mainnet. However, after I start to use the node, running json rpc request and so on, the node cost disk rapidly, almost grows 1GB in 5 mins.

Here is my command to run parity: parity --cache-size 4096 --db-compaction ssd --base-path "/.eth" --jsonrpc-apis "web3,eth,pubsub,net,parity,private,parity_pubsub,traces,rpc,parity_transactions_pool,personal" --jsonrpc-interface 0.0.0.0 --jsonrpc-cors * --enable-signing-queue true

Here is the log: May 15 13:10:22: 2020-05-15 13:10:22 UTC #3110666 18/25 peers 221 MiB chain 138 MiB db 0 bytes queue 36 MiB sync RPC: 0 conn, 4 req/s, 25 µs May 15 13:10:26: 2020-05-15 13:10:26 UTC 37 -> GetNodeData: item 192/384 – slow state fetch for hash 0xb8904769acb3fc0902eb03496c9329c5c4c319185754b2ecf3b7004cc0b4ed76; took Instant { tv_sec: 887827, tv_nsec: 236457208 } May 15 13:10:31: 2020-05-15 13:10:31 UTC Imported #10070924 0xd45c…22f8 (184 txs, 9.96 Mgas, 3307 ms, 35.58 KiB) May 15 13:10:33: 2020-05-15 13:10:33 UTC 37 -> GetNodeData: item 10/384 – slow state fetch for hash 0xda4f4ed2b1bb8db5d0e9bb55e8a6e70258e64fe8021228c5e8f295896f576402; took Instant { tv_sec: 887834, tv_nsec: 232453585 } May 15 13:10:42: 2020-05-15 13:10:42 UTC Imported #10070926 0x08e5…6965 (0 txs, 0.00 Mgas, 16 ms, 0.52 KiB) May 15 13:10:51: 2020-05-15 13:10:51 UTC Imported #10070921 0x52b7…0dc5 (143 txs, 9.97 Mgas, 3419 ms, 22.28 KiB) May 15 13:10:54: 2020-05-15 13:10:54 UTC #3116924 20/25 peers 221 MiB chain 138 MiB db 0 bytes queue 39 MiB sync RPC: 0 conn, 35 req/s, 37 µs May 15 13:10:59: 2020-05-15 13:10:59 UTC 37 -> GetNodeData: item 69/384 – slow state fetch for hash 0x7227bb7e5ee96e544a38736163de276d232b6afc8be0b1b7f42381110596dbed; took Instant { tv_sec: 887860, tv_nsec: 352783122 } May 15 13:11:04: 2020-05-15 13:11:04 UTC 37 -> GetNodeData: item 3/384 – slow state fetch for hash 0xb70290b58b5e1e8b80ca3ce87a65ba915e66273c190ba41b51e77d07d2d42e41; took Instant { tv_sec: 887865, tv_nsec: 124825721 } May 15 13:11:06: 2020-05-15 13:11:06 UTC Imported #10070927 0xe264…a28c (169 txs, 9.99 Mgas, 4350 ms, 25.15 KiB) May 15 13:11:13: 2020-05-15 13:11:13 UTC 37 -> GetNodeData: item 29/384 – slow state fetch for hash 0xd3caa42ae5b83d3b609f688f386648d4f5b441eae64865ebd2f45da52de6eff2; took Instant { tv_sec: 887874, tv_nsec: 680094813 } May 15 13:11:30: 2020-05-15 13:11:30 UTC Imported #10070928 0xbca1…98b2 (172 txs, 9.96 Mgas, 4374 ms, 25.99 KiB) May 15 13:11:31: 2020-05-15 13:11:31 UTC #3123816 17/25 peers 222 MiB chain 139 MiB db 0 bytes queue 36 MiB sync RPC: 0 conn, 4 req/s, 22 µs May 15 13:11:43: 2020-05-15 13:11:43 UTC Imported #10070929 0x5c6b…6fdc (137 txs, 9.98 Mgas, 4732 ms, 23.64 KiB) May 15 13:11:54: 2020-05-15 13:11:54 UTC Imported #10070930 0xcade…975f (100 txs, 9.91 Mgas, 6032 ms, 21.55 KiB) May 15 13:12:05: 2020-05-15 13:12:05 UTC Reorg to #10070932 0x08a0…0049 (0xb411…88b7 #10070930 0xcade…975f 0x1017…909b) May 15 13:12:13: 2020-05-15 13:12:13 UTC Imported #10070932 0x08a0…0049 (0 txs, 0.00 Mgas, 2731 ms, 0.52 KiB) + another 1 block(s) containing 94 tx(s) May 15 13:12:20: 2020-05-15 13:12:20 UTC #3129173 20/25 peers 226 MiB chain 138 MiB db 0 bytes queue 44 MiB sync RPC: 0 conn, 18 req/s, 19 µs May 15 13:12:21: 2020-05-15 13:12:21 UTC Imported #10070933 0xcd68…77e1 (170 txs, 10.00 Mgas, 4908 ms, 29.98 KiB) May 15 13:12:31: 2020-05-15 13:12:31 UTC Imported #10070934 0x6765…2b3f (147 txs, 9.97 Mgas, 3966 ms, 25.80 KiB) May 15 13:12:50: 2020-05-15 13:12:50 UTC #3136348 21/25 peers 188 MiB chain 138 MiB db 0 bytes queue 53 MiB sync RPC: 0 conn, 19 req/s, 28 µs May 15 13:13:02: 2020-05-15 13:13:02 UTC Imported #10070935 0x1941…b56c (98 txs, 9.99 Mgas, 6063 ms, 15.50 KiB) May 15 13:13:26: 2020-05-15 13:13:26 UTC #3153829 21/25 peers 189 MiB chain 138 MiB db 0 bytes queue 47 MiB sync RPC: 0 conn, 5 req/s, 20 µs May 15 13:13:35: 2020-05-15 13:13:35 UTC 37 -> GetNodeData: item 5/384 – slow state fetch for hash 0x3d9ae468bd18c2bce5b81904843be444c38a4d55724a01e7b04cc6c59ee7d34f; took Instant { tv_sec: 888016, tv_nsec: 789367584 } May 15 13:14:03: 2020-05-15 13:14:03 UTC #3182026 18/25 peers 193 MiB chain 138 MiB db 0 bytes queue 35 MiB sync RPC: 0 conn, 16 req/s, 22 µs May 15 13:14:03: 2020-05-15 13:14:03 UTC Imported #10070936 0x03bb…1478 (255 txs, 9.96 Mgas, 3328 ms, 41.42 KiB)

[jinglics]

What's the meaning of this log 2020-05-15 13:13:26 UTC #3153829 21/25 peers 189 MiB chain 138 MiB db 0 bytes queue 47 MiB sync RPC: 0 conn, 5 req/s, 20 µs. I'm doing request to the node.

A lot of files generated at overlayrecent, like 012493.sst

[SomethingUseful]

Same issue but only started growing rapidly on openeth 3.0. No change in config on upgrade from 2.7 to 3. Running latest 3.0.1... eating 4-5GB per day! Now up to 400+GB

OpenEthereum version: OpenEthereum/v3.0.1-stable-8ca8089-20200601/x86_64-unknown-linux-gnu/rustc1.43.1

Operating system: Linux

Installation: https://github.com/openethereum/openethereum/releases/download/v3.0.1/openethereum-linux-v3.0.1.zip

Fully synchronized: yes Network: ethereum mainnet Restarted: yes

Setup: 6CPU/16GB of ram. SSD.

EXAMPLE RUN: openethereum --chain mainnet --config .../config.toml

SNIPPET OF CONFIG: `[parity] mode = "last" no_persistent_txqueue = true chain = "mainnet" base_path = "/not/default/.ethereumparity/share/io.parity.ethereum" db_path = "/not/default/ethereum/chains" light = false

[network] discovery = true warp = true

tx_queue_size = 8192 tx_queue_per_sender = 8192

[footprint] tracing = "off" fat_db = "off" pruning = "fast" db_compaction = "ssd"

[snapshots] disable_periodic = true`

CHAIN DATA: 400+GB and growing fast at 5/GB/DAY 6700+ sst files...

ethereum/chains/ethereum/db/906a34e69aec8c0d/overlayrecent/db /.. 2.4 GiB [##########] /blooms 91.9 MiB [ ] 436793.sst 67.6 MiB [ ] 434030.sst 67.6 MiB [ ] 390882.sst 67.5 MiB [ ] 419861.sst 67.4 MiB [ ] 400297.sst 67.4 MiB [ ] 421432.sst 67.3 MiB [ ] 424576.sst 67.2 MiB [ ] 411739.sst 67.1 MiB [ ] 381813.sst 67.1 MiB [ ] 435272.sst 67.1 MiB [ ] 424770.sst 67.0 MiB [ ] 387992.sst

/ethereum/chains/ethereum/db/906a34e69aec8c0d /.. 402.5 GiB [##########] /overlayrecent 14.3 GiB [ ] /snapshot

[denisgranha]

thank you for your reports.

Can I ask what’s the content of /not/default/ethereum/chains ?

please paste:

ls /not/default/ethereum/chains

[SomethingUseful]

^ added to original comment above

[jinglics]

My node now is running normally, but the blockchain data keeps growing about 1GB per hour now. However, when I restart, the storage size will decrease to 290GB around and then grow again.

parity --cache-size 12000 --db-compaction ssd --base-path /root/.eth --jsonrpc-apis "web3,eth,pubsub,net,parity,private,parity_pubsub,traces,rpc,parity_transactions_pool,personal" --jsonrpc-interface 0.0.0.0 --jsonrpc-cors * --enable-signing-queue true

65M 176025.sst 27M 176026.sst 421M 176027.log 175M 176029.log 418M 176030.log 578M 176031.sst 172M 176032.log 247M 176033.log 16M 176034.sst 404M 176035.log 150M 176036.log 177M 176037.log 536K 176039.sst

[SomethingUseful]

mine has 6700 sst files and only 94 log files. all the log files are less than 2 days old and less than 65MB. still increasing in size at 4 to 5GB per day.

[kael-shipman]

Possibly related: #11705

[DubbaThony]

@kael-shipman I am fairly convinced it's related or same issue, but for me it apeared in 2.7, and upgrade to 3.0 simply didn't fix it. We still have crippling issues with that node, resyncing from scratch dosen't help, I tried many times. Only solution I am aware of is to fallback to some version around 2.4

[kael-shipman]

I just realized I linked that other issue but never actually mentioned that I'm having the same problem I wrote a stack exchange question about it here, but haven't done anything else to make progress.

I don't think I really have anything to add here, other than that I'm also experiencing the issue in all versions past 2.5.

[claberus]

The next release of OpenEthereum will be based on 2.5.13-stable and will include all of the bug fixes and support for hard forks since 2.7 and 3.0.