Nethermind no longer auto pruning?

[rosskevin]

This originally happened about three weeks ago. I ran out of space when auto-pruning had been maintaining on a 2TB drive. I accepted it, update the system and performed a resync. I enabled disk space notification on Beaconcha.in.

Fast forward to today, and I'm at 90% usage, received a notification and can't execute a prune.

kross@sauce:~/eth-docker$ ./ethd up
[+] Running 9/9
 ✔ Container eth-docker-blackbox-exporter-1          Running                                                                                                                                                                                                       0.0s 
 ✔ Container eth-docker-json-exporter-1              Running                                                                                                                                                                                                       0.0s 
 ✔ Container eth-docker-ethereum-metrics-exporter-1  Started                                                                                                                                                                                                      13.3s 
 ✔ Container eth-docker-consensus-1                  Running                                                                                                                                                                                                       0.0s 
 ✔ Container eth-docker-mev-boost-1                  Running                                                                                                                                                                                                       0.0s 
 ✔ Container eth-docker-validator-1                  Running                                                                                                                                                                                                       0.0s 
 ✔ Container eth-docker-execution-1                  Started                                                                                                                                                                                                      13.5s 
 ✔ Container eth-docker-prometheus-1                 Started                                                                                                                                                                                                      11.8s 
 ✔ Container eth-docker-grafana-1                    Started                                                                                                                                                                                                      11.3s 

You are running Nethermind and have less than 300 GiB of free disk space.
It should currently be auto-pruning, check logs with "./ethd logs -f --tail 500 execution | grep Full". Free space:

Here's total and used space on /var/lib/docker
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv  1.9T  1.6T  203G  89% /

Here are the Docker volumes used by this copy of Eth Docker and their space usage:
eth-docker_grafana-data                                            1         2.73MB
eth-docker_loki-data                                               0         162.8MB
eth-docker_nm-eth1-data                                            1         1.274TB
eth-docker_prom-data                                               1         2.934GB
eth-docker_grafana-config                                          1         728.8kB
eth-docker_promtail-data                                           0         2.576kB
eth-docker_jwtsecret                                               2         64B
eth-docker_lhvalidator-data                                        1         1.068GB
eth-docker_lhbeacon-data                                           1         414.4GB

If your Consensus Layer client takes more than 300 GiB, you can resync it with
"./ethd resync-consensus".

If there is some mystery space being taken up, try "sudo ncdu /".

kross@sauce:~/eth-docker$ ./ethd logs -f --tail 500 execution | grep Full
eth-docker-execution-1  | --Pruning.FullPruningMaxDegreeOfParallelism=2 --Pruning.FullPruningTrigger=VolumeFreeSpace --Pruning.FullPruningThresholdMb=375810 --Pruning.FullPruningMemoryBudgetMb=16384 --Init.StateDbKeyScheme=HalfPath

Startup:

$ ./ethd logs -f execution 
eth-docker-execution-1  | Using pruning parameters:
eth-docker-execution-1  | --Pruning.FullPruningMaxDegreeOfParallelism=2 --Pruning.FullPruningTrigger=VolumeFreeSpace --Pruning.FullPruningThresholdMb=375810 --Pruning.FullPruningMemoryBudgetMb=16384 --Init.StateDbKeyScheme=HalfPath
eth-docker-execution-1  | 2024-05-12 20-49-25.3053|Nethermind starting initialization.
eth-docker-execution-1  | 2024-05-12 20-49-25.3200|Client version: Nethermind/v1.26.0+0068729c/linux-x64/dotnet8.0.4
eth-docker-execution-1  | 2024-05-12 20-49-25.3316|Loading embedded plugins
eth-docker-execution-1  | 2024-05-12 20-49-25.3317|  Found plugin type Nethermind.Consensus.AuRa.AuRaPlugin
eth-docker-execution-1  | 2024-05-12 20-49-25.3317|  Found plugin type Nethermind.Consensus.Clique.CliquePlugin
eth-docker-execution-1  | 2024-05-12 20-49-25.3317|  Found plugin type Nethermind.Consensus.Ethash.EthashPlugin
eth-docker-execution-1  | 2024-05-12 20-49-25.3317|  Found plugin type Nethermind.Consensus.Ethash.NethDevPlugin
eth-docker-execution-1  | 2024-05-12 20-49-25.3318|  Found plugin type Nethermind.Hive.HivePlugin
eth-docker-execution-1  | 2024-05-12 20-49-25.3318|  Found plugin type Nethermind.UPnP.Plugin.UPnPPlugin
eth-docker-execution-1  | Resolved executing directory as /nethermind.
eth-docker-execution-1  | 2024-05-12 20-49-25.3458|Loading 12 assemblies from /nethermind/plugins
eth-docker-execution-1  | 2024-05-12 20-49-25.3458|Loading assembly Nethermind.Optimism
eth-docker-execution-1  | 2024-05-12 20-49-25.3536|  Found plugin type Nethermind.Optimism
eth-docker-execution-1  | 2024-05-12 20-49-25.3536|Loading assembly Nethermind.Mev
eth-docker-execution-1  | 2024-05-12 20-49-25.3560|  Found plugin type Nethermind.Mev
eth-docker-execution-1  | 2024-05-12 20-49-25.3560|Loading assembly Nethermind.Merge.AuRa
eth-docker-execution-1  | 2024-05-12 20-49-25.3565|  Found plugin type Nethermind.Merge.AuRa
eth-docker-execution-1  | 2024-05-12 20-49-25.3565|Loading assembly Nethermind.Init
eth-docker-execution-1  | 2024-05-12 20-49-25.3575|Loading assembly Nethermind.Merge.Plugin
eth-docker-execution-1  | 2024-05-12 20-49-25.3599|  Found plugin type Nethermind.Merge.Plugin
eth-docker-execution-1  | 2024-05-12 20-49-25.3599|Loading assembly Nethermind.JsonRpc.TraceStore
eth-docker-execution-1  | 2024-05-12 20-49-25.3605|  Found plugin type Nethermind.JsonRpc.TraceStore
eth-docker-execution-1  | 2024-05-12 20-49-25.3605|Loading assembly Nethermind.AccountAbstraction
eth-docker-execution-1  | 2024-05-12 20-49-25.3650|  Found plugin type Nethermind.AccountAbstraction
eth-docker-execution-1  | 2024-05-12 20-49-25.3650|Loading assembly Nethermind.Api
eth-docker-execution-1  | 2024-05-12 20-49-25.3657|Loading assembly Nethermind.EthStats
eth-docker-execution-1  | 2024-05-12 20-49-25.3662|  Found plugin type Nethermind.EthStats
eth-docker-execution-1  | 2024-05-12 20-49-25.3663|Loading assembly Nethermind.HealthChecks
eth-docker-execution-1  | 2024-05-12 20-49-25.3677|  Found plugin type Nethermind.HealthChecks
eth-docker-execution-1  | 2024-05-12 20-49-25.3677|Loading assembly Nethermind.Init.Snapshot
eth-docker-execution-1  | 2024-05-12 20-49-25.3681|  Found plugin type Nethermind.Init.Snapshot
eth-docker-execution-1  | 2024-05-12 20-49-25.3681|Loading assembly Nethermind.Consensus.AuRa
eth-docker-execution-1  | 2024-05-12 20-49-25.4288|Loading standard NLog.config file from /nethermind/NLog.config.
eth-docker-execution-1  | 2024-05-12 20-49-25.5399|NLog.config loaded in 110ms.
eth-docker-execution-1  | Enabling log level override: INFO
eth-docker-execution-1  | file-async TEST
eth-docker-execution-1  | auto-colored-console-async TEST
eth-docker-execution-1  | 2024-05-12 20-49-25.5424|Reading config file from /nethermind/configs/mainnet.cfg
eth-docker-execution-1  | 2024-05-12 20-49-25.5661|Incorrect config settings found:
eth-docker-execution-1  | ConfigType:JsonConfigFile|Category:SyncConfig|Name:FastBlocks
eth-docker-execution-1  | 2024-05-12 20-49-25.5661|Configuration initialized.
eth-docker-execution-1  | 2024-05-12 20-49-25.5670|Setting BaseDbPath to: /var/lib/nethermind/nethermind_db/mainnet, from: nethermind_db/mainnet
eth-docker-execution-1  | 2024-05-12 20-49-25.5670|Setting KeyStoreDirectory to: /var/lib/nethermind/keystore, from: keystore
eth-docker-execution-1  | 2024-05-12 20-49-25.5670|Setting LogDirectory to: /var/lib/nethermind/logs, from: logs
eth-docker-execution-1  | 12 May 20:49:25 | RocksDb Version: 8.10.0 
eth-docker-execution-1  | 12 May 20:49:25 | ChainSpecPath matched an embedded resource inside the binary. Loading chainspec from embedded resources instead of file! 
eth-docker-execution-1  | 12 May 20:49:25 | Loading Ckzg trusted setup from file /nethermind/kzg_trusted_setup.txt 
eth-docker-execution-1  | 12 May 20:49:25 | CPU:  (CT) 
eth-docker-execution-1  | 12 May 20:49:25 | Using http://ipv4.icanhazip.com to get external ip 
eth-docker-execution-1  | 12 May 20:49:25 | Setting up memory allowances 
eth-docker-execution-1  | 12 May 20:49:25 |   Memory hint:         2048 MB 
eth-docker-execution-1  | 12 May 20:49:25 |   General memory:        32 MB 
eth-docker-execution-1  | 12 May 20:49:25 |   Peers memory:          50 MB 
eth-docker-execution-1  | 12 May 20:49:25 |   Netty memory:         268 MB 
eth-docker-execution-1  | 12 May 20:49:25 |   Mempool memory:       149 MB 
eth-docker-execution-1  | 12 May 20:49:25 |   Fast blocks memory:   154 MB 
eth-docker-execution-1  | 12 May 20:49:25 |   Trie memory:          278 MB 
eth-docker-execution-1  | 12 May 20:49:25 |   DB memory:           1114 MB 
eth-docker-execution-1  | 12 May 20:49:33 | Start block loaded from reorg boundary - 19856363 - 19856363 (0xe962f1...932797) 
eth-docker-execution-1  | 12 May 20:49:33 | Numbers resolved, level = Max(19856363, 19856428), header = Max(19856363, 19856428), body = Max(19856363, 19856428) 
eth-docker-execution-1  | 12 May 20:49:33 | Beacon Numbers resolved, level = 0, header = 0, body = 0 
eth-docker-execution-1  | 12 May 20:49:33 | Loading fork choice info 
eth-docker-execution-1  | 12 May 20:49:33 | Block tree initialized, last processed is 19856363 (0xe962f1...932797), best queued is 19856428, best known is 19856428, lowest inserted header 1, body 0, lowest sync inserted block number 19856425 
eth-docker-execution-1  | 12 May 20:49:33 | Initializing 15 plugins 
eth-docker-execution-1  | 12 May 20:49:33 |   Clique by Nethermind 
eth-docker-execution-1  | 12 May 20:49:33 |   Clique by Nethermind initialized in 0ms 
eth-docker-execution-1  | 12 May 20:49:33 |   AuRa by Nethermind 
eth-docker-execution-1  | 12 May 20:49:33 |   AuRa by Nethermind initialized in 0ms 
eth-docker-execution-1  | 12 May 20:49:33 |   Ethash by Nethermind 
eth-docker-execution-1  | 12 May 20:49:33 |   Ethash by Nethermind initialized in 2ms 
eth-docker-execution-1  | 12 May 20:49:33 |   Optimism by Nethermind 
eth-docker-execution-1  | 12 May 20:49:33 |   Optimism by Nethermind initialized in 0ms 
eth-docker-execution-1  | 12 May 20:49:33 |   NethDev by Nethermind 
eth-docker-execution-1  | 12 May 20:49:33 |   NethDev by Nethermind initialized in 0ms 
eth-docker-execution-1  | 12 May 20:49:33 |   AuRaMerge by Nethermind 
eth-docker-execution-1  | 12 May 20:49:33 |   AuRaMerge by Nethermind initialized in 0ms 
eth-docker-execution-1  | 12 May 20:49:33 |   Merge by Nethermind 
eth-docker-execution-1  | 12 May 20:49:33 | Client started with TTD: 58750000000000000000000, TTD reached: True, Terminal Block Number , FinalTotalDifficulty: 58750003716598352816469 
eth-docker-execution-1  | 12 May 20:49:33 |   Merge by Nethermind initialized in 7ms 
eth-docker-execution-1  | 12 May 20:49:33 |   MEV by Nethermind 
eth-docker-execution-1  | 12 May 20:49:33 |   MEV by Nethermind initialized in 0ms 
eth-docker-execution-1  | 12 May 20:49:33 |   HealthChecks by Nethermind 
eth-docker-execution-1  | 12 May 20:49:33 |   HealthChecks by Nethermind initialized in 1ms 
eth-docker-execution-1  | 12 May 20:49:33 |   Hive by Nethermind 
eth-docker-execution-1  | 12 May 20:49:33 |   Hive by Nethermind initialized in 0ms 
eth-docker-execution-1  | 12 May 20:49:33 |   Account Abstraction by Nethermind 
eth-docker-execution-1  | 12 May 20:49:33 |   Account Abstraction Plugin: User Operation Mining Disabled 
eth-docker-execution-1  | 12 May 20:49:33 |   Account Abstraction by Nethermind initialized in 0ms 
eth-docker-execution-1  | 12 May 20:49:33 |   EthStats by Nethermind 
eth-docker-execution-1  | 12 May 20:49:33 |   EthStats by Nethermind initialized in 0ms 
eth-docker-execution-1  | 12 May 20:49:33 |   Snapshot by Nethermind 
eth-docker-execution-1  | 12 May 20:49:33 |   Snapshot by Nethermind initialized in 0ms 
eth-docker-execution-1  | 12 May 20:49:33 |   TraceStore by Nethermind 
eth-docker-execution-1  | 12 May 20:49:33 |   TraceStore by Nethermind initialized in 0ms 
eth-docker-execution-1  | 12 May 20:49:33 |   UPnP by Nethermind 
eth-docker-execution-1  | 12 May 20:49:33 |   UPnP by Nethermind initialized in 0ms 
eth-docker-execution-1  | 12 May 20:49:33 | Detected Hash key scheme. 
eth-docker-execution-1  | 12 May 20:49:33 | Loaded 1 blob txs from persistent db, containing 1 blobs, in 3ms 
eth-docker-execution-1  | 12 May 20:49:33 | Snap - State Ranges (Phase 1) is finished. 
eth-docker-execution-1  | 12 May 20:49:33 | Pivot block has been set based on data from db. Pivot block number: 19697597, hash: 0xd3884dc61ee7bd395e9d3e43ec7c01a52d3b2b9e3387a96f6f87c3577c736079 
eth-docker-execution-1  | 12 May 20:49:33 | Loaded 0 static nodes from file: /nethermind/Data/static-nodes.json 
eth-docker-execution-1  | 12 May 20:49:33 | Skipping Account Abstraction network protocol 
eth-docker-execution-1  | 12 May 20:49:33 | Now syncing nodes starting from root of block 0 
eth-docker-execution-1  | 12 May 20:49:33 | Connected to 0 bootnodes, 4 trusted/persisted nodes 
eth-docker-execution-1  | 12 May 20:49:33 | Found 65 block tree levels to review for fixes starting from 19856364 
eth-docker-execution-1  | 12 May 20:49:33 | Reviewed 0 blocks out of 65 
eth-docker-execution-1  | 12 May 20:49:33 | Rerunning block after reorg or pruning: 19856364 (0x8002b0...acc5ec) 
eth-docker-execution-1  | 12 May 20:49:33 | Numbers resolved, level = Max(19856363, 19856428), header = Max(19856363, 19856428), body = Max(19856363, 19856428) 
eth-docker-execution-1  | 12 May 20:49:33 | Beacon Numbers resolved, level = 0, header = 0, body = 0 
eth-docker-execution-1  | 12 May 20:49:33 | Loading fork choice info 
eth-docker-execution-1  | Hosting environment: Production
eth-docker-execution-1  | Content root path: /nethermind/
eth-docker-execution-1  | Now listening on: http://0.0.0.0:6060
eth-docker-execution-1  | 12 May 20:49:33 | Started monitoring for the group: nethermind, instance: Mainnet 
eth-docker-execution-1  | 12 May 20:49:33 | System.Diagnostics.Metrics disabled 
eth-docker-execution-1  | 12 May 20:49:33 | Starting Merge block producer & sealer 

I'm going to force a resync (again) unless there is some information I can provide that would be helpful.

I'm not sure why auto prune is either not keeping up or not running.

[rosskevin]

As an FYI - I edited to adjust prune minimums to recommended nethermind minimums but failed because I only have 205G free:

kross@sauce:~/eth-docker$ ./ethd prune-nethermind
You do not have enough free disk space. Make sure this reads at least 250G free (Avail):

Here's total and used space on /var/lib/docker
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv  1.9T  1.6T  205G  89% /

Aborting.

[rosskevin]

I saw:

If your Consensus Layer client takes more than 300 GiB, you can resync it with
"./ethd resync-consensus".

So I executed that, which takes everything offline but removes the 414.4GB volume for consensus, which allowed me to execute ./ethd prune-nethermind.

Perhaps it is unexpected for my lighthouse consensus to grow so big which is impacting nethermind auto prune success?

[yorickdowne]

That Lighthouse grows that big is expected. A periodic resync of Lighthouse helps.

Most likely you didn’t have enough space for Nethermind to actually prune. It was below 350 GiB to start the prune, but not above the 270 or so it needs at a minimum. That minimum may increase with time.

This isn’t an Eth Docker issue.

In the near term both Lighthouse tree states (not ready yet) and Nethermind Path or Paprika (also not ready yet) can extend the usefulness of a 2TB drive.

In the medium term you’ll want to get a 4TB drive, see the drive gist, and CloneZilla your setup over. Remy has a good guide.

2TB should run out sometime early mid 2025.