search

anza-xyz / agave

Web-Scale Blockchain for fast, secure, scalable, decentralized apps and marketplaces.
https://www.anza.xyz/
Apache License 2.0
461 stars 227 forks source link

ledger-tool v1.18.26 clears the snapshot folder #3198

Closed diman-io closed 1 day ago

diman-io commented 1 month ago

I understand that it most likely won't be fixed, but maybe, if there's another restart, it would be worth just specifying to use v2 of ledger-tool. Today's instructions in the first point require installing v1.18.26.

~/.local/share/solana/install/releases/1.18.26/solana-release/bin/agave-ledger-tool -l /mnt/disk2/ledger create-snapshot --incremental --snapshot-archive-path /mnt/disk3/validator/snapshots/ --incremental-snapshot-archive-path /mnt/disk3/validator/incremental_snapshots/ --hard-fork 296877183 --deactivate-feature-gate 7uZBkJXJ1HkuP6R3MJfZs7mLwymBcDbKdqbF51ZWLier tvcF6b1TRz353zKuhBjinZkKzjmihXmBAHJdjNYw1sQ decoMktMcnmiq6t3u7g5BfgcQu91nKZr6RvMYf9z1Jb --accounts /mnt/disk3/validator/accounts --accounts-hash-cache-path /mnt/disk3/validator/accounts_hash_cache/ -- 296877183

[2024-10-17T07:31:45.212969291Z INFO  agave_ledger_tool] agave-ledger-tool 1.18.26 (src:c2b35002; feat:3241752014, client:Agave)
[2024-10-17T07:31:45.213265258Z INFO  solana_ledger::blockstore] Maximum open file descriptors: 1000000
[2024-10-17T07:31:45.213267800Z INFO  solana_ledger::blockstore] Opening blockstore at "/mnt/disk2/ledger/rocksdb"
[2024-10-17T07:31:45.214032395Z INFO  solana_ledger::blockstore_db] Opening Rocks with secondary (read only) access at: "/mnt/disk2/ledger/rocksdb/solana-secondary". This secondary access could temporarily degrade other accesses, such as by agave-validator
[2024-10-17T07:31:47.445610303Z INFO  solana_ledger::blockstore_db] Rocks's automatic compactions are disabled due to Secondary access
[2024-10-17T07:31:47.445692893Z INFO  solana_ledger::blockstore] Opening blockstore done; blockstore open took 2.2s
[2024-10-17T07:31:47.445730744Z INFO  agave_ledger_tool] Creating incremental snapshot of slot 296877183 in /mnt/disk3/validator/incremental_snapshots/
[2024-10-17T07:31:47.447783553Z INFO  agave_ledger_tool::ledger_utils] Checking if another process currently holding Primary access to "/mnt/disk2/ledger"
[2024-10-17T07:31:47.447792608Z INFO  solana_ledger::blockstore] Maximum open file descriptors: 1000000
[2024-10-17T07:31:47.447794223Z INFO  solana_ledger::blockstore] Opening blockstore at "/mnt/disk2/ledger/rocksdb"
[2024-10-17T07:31:47.719879852Z INFO  solana_ledger::blockstore_db] Rocks's automatic compactions are disabled due to PrimaryForMaintenance access
[2024-10-17T07:31:47.719950749Z INFO  solana_ledger::blockstore] Opening blockstore done; blockstore open took 272ms
[2024-10-17T07:31:47.747083422Z INFO  agave_ledger_tool::ledger_utils] Cleaning contents of account path: /mnt/disk3/validator/accounts/run
[2024-10-17T07:31:47.747149118Z INFO  agave_ledger_tool::ledger_utils] Cleaning account paths took 67us
[2024-10-17T07:31:47.747183705Z INFO  agave_ledger_tool::ledger_utils] Cleaning contents of account snapshot paths: ["/mnt/disk3/validator/accounts/snapshot"]
[2024-10-17T07:31:47.747235627Z INFO  solana_ledger::bank_forks_utils] Initializing bank snapshots dir: /mnt/disk2/ledger/ledger_tool/snapshot
[2024-10-17T07:31:47.747249207Z INFO  solana_runtime::snapshot_bank_utils] Loading bank from full snapshot archive: /mnt/disk3/validator/snapshots/snapshot-296863173-82pXRtPWvfGPzkkFMgzEUZPvvmJPVMrXB7jtShndjpcR.tar.zst, and incremental snapshot archive: Some("/mnt/disk3/validator/incremental_snapshots/incremental-snapshot-296863173-296876393-2gZEWR4vWo6fnmth7LxvMHXaC1UYJAfK79LeM7esxdwn.tar.zst")
[2024-10-17T07:31:55.124310455Z INFO  solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 249695/391142 slots with 0 collisions
[2024-10-17T07:31:57.125610279Z INFO  solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 323472/391142 slots with 0 collisions
[2024-10-17T07:31:58.914078308Z INFO  solana_accounts_db::shared_buffer_reader] reading entire decompressed file took: 11166640 us, bytes: 32611800576, read_us: 11143925, waiting_for_buffer_us: 0, largest fetch: 11665408, error: Ok(0)
[2024-10-17T07:31:58.927255350Z INFO  solana_accounts_db::hardened_unpack] unpacked 97787 entries total
[2024-10-17T07:31:58.928138191Z INFO  solana_accounts_db::hardened_unpack] unpacked 97786 entries total
[2024-10-17T07:31:58.928225244Z INFO  solana_accounts_db::hardened_unpack] unpacked 97787 entries total
[2024-10-17T07:31:58.969247137Z INFO  solana_accounts_db::hardened_unpack] unpacked 97787 entries total
[2024-10-17T07:31:59.006004942Z INFO  solana_runtime::snapshot_utils] snapshot untar took 11.3s
[2024-10-17T07:31:59.656572444Z INFO  solana_accounts_db::shared_buffer_reader] reading entire decompressed file took: 648106 us, bytes: 1766320640, read_us: 646539, waiting_for_buffer_us: 0, largest fetch: 8388608, error: Ok(0)
[2024-10-17T07:31:59.666058486Z INFO  solana_accounts_db::hardened_unpack] unpacked 3001 entries total
[2024-10-17T07:31:59.666053713Z INFO  solana_accounts_db::hardened_unpack] unpacked 3001 entries total
[2024-10-17T07:31:59.666066167Z INFO  solana_accounts_db::hardened_unpack] unpacked 3002 entries total
[2024-10-17T07:31:59.666064879Z INFO  solana_accounts_db::hardened_unpack] unpacked 3002 entries total
[2024-10-17T07:32:02.763201228Z INFO  solana_runtime::snapshot_utils] incremental snapshot untar took 3.8s
[2024-10-17T07:32:02.764875594Z INFO  solana_runtime::snapshot_utils] snapshot version: 1.2.0
[2024-10-17T07:32:02.764914556Z INFO  solana_runtime::snapshot_utils] snapshot version: 1.2.0
[2024-10-17T07:32:02.764932650Z INFO  solana_runtime::snapshot_bank_utils] Rebuilding bank from full snapshot /mnt/disk2/ledger/ledger_tool/snapshot/tmp-snapshot-archive-SRrRkd/snapshots/296863173/296863173 and incremental snapshot Some("/mnt/disk2/ledger/ledger_tool/snapshot/tmp-snapshot-archive-CQXApQ/snapshots/296876393/296876393")
[2024-10-17T07:32:08.518465791Z INFO  solana_accounts_db::accounts_db] Background account hasher has started
[2024-10-17T07:32:10.500210331Z INFO  solana_accounts_db::accounts_db] rent_collector: RentCollector { epoch: 700, epoch_schedule: EpochSchedule { slots_per_epoch: 432000, leader_schedule_slot_offset: 432000, warmup: true, first_normal_epoch: 14, first_normal_slot: 524256 }, slots_per_year: 78892314.984, rent: Rent { lamports_per_byte_year: 3480, exemption_threshold: 2.0, burn_percent: 100 } }
[2024-10-17T07:32:21.054549928Z INFO  solana_accounts_db::accounts_db] accounts data len: 19620152963
[2024-10-17T07:32:23.074823189Z INFO  solana_runtime::bank] Rebuilding skipped rewrites of 183 accounts took 416us
[2024-10-17T07:32:23.117157846Z INFO  solana_runtime::serde_snapshot] rent_collector: RentCollector { epoch: 700, epoch_schedule: EpochSchedule { slots_per_epoch: 432000, leader_schedule_slot_offset: 432000, warmup: true, first_normal_epoch: 14, first_normal_slot: 524256 }, slots_per_year: 78892314.984, rent: Rent { lamports_per_byte_year: 3480, exemption_threshold: 2.0, burn_percent: 50 } }
[2024-10-17T07:32:23.117192241Z INFO  solana_runtime::snapshot_bank_utils] Rebuilding status cache from /mnt/disk2/ledger/ledger_tool/snapshot/tmp-snapshot-archive-CQXApQ/snapshots/status_cache
[2024-10-17T07:32:23.534961814Z INFO  solana_runtime::snapshot_bank_utils] Rebuilt bank for slot: 296876393
[2024-10-17T07:32:23.540056003Z INFO  solana_runtime::snapshot_bank_utils] rebuild bank from snapshots took 20.8s
[2024-10-17T07:32:23.540066412Z INFO  solana_runtime::bank] Cleaning... Skipped.
[2024-10-17T07:32:23.540067935Z INFO  solana_runtime::bank] Shrinking... Skipped.
[2024-10-17T07:32:23.540068889Z INFO  solana_runtime::bank] Verifying accounts...
[2024-10-17T07:32:23.540097608Z INFO  solana_runtime::bank] Verifying accounts... In background.
[2024-10-17T07:32:23.540098866Z INFO  solana_runtime::bank] Verifying bank...
[2024-10-17T07:32:23.540250902Z INFO  solana_runtime::bank] Initial background accounts hash verification has started
[2024-10-17T07:32:23.540542120Z INFO  solana_accounts_db::accounts_db] skipped rewrite hashes 296876393 0
[2024-10-17T07:32:23.540908681Z INFO  solana_runtime::bank] bank frozen: 296876393 hash: AFic2vE3Y8y9L3wKahKQ2B6jhMg3PRychDFkMgeqroJT accounts_delta: EmBKnbJjhj7xxFrHF7UxL3AqVmP7z58NswkBVyzdccZL signature_count: 1670 last_blockhash: FF2aEZqshZmUg9HXgJJFVD8N2QwL5QNL5yGGCYwQ8341 capitalization: 1858608454709713878, stats: BankHashStats { num_updated_accounts: 3524, num_removed_accounts: 0, num_lamports_stored: 141533850365282086, total_data_len: 6451233, num_executable_accounts: 0 }
[2024-10-17T07:32:23.540939930Z INFO  solana_runtime::bank] Verifying bank... Done.
[2024-10-17T07:32:23.822112196Z INFO  agave_ledger_tool::ledger_utils] Using: block-verification-method: blockstore-processor
[2024-10-17T07:32:23.822122702Z INFO  agave_ledger_tool::ledger_utils] no scheduler pool is installed for block verification...
[2024-10-17T07:32:23.822639264Z INFO  solana_ledger::blockstore_processor] Processing ledger from slot 296876393...
[2024-10-17T07:32:23.822642986Z INFO  solana_ledger::blockstore_processor] Start slot 296876393 isn't a root, and won't be updated due to secondary blockstore access

...

[2024-10-17T07:33:36.250531209Z INFO  solana_runtime::bank] bank frozen: 296877184 hash: ABuvd5grMYhbrjeLmu3qCNqiD8DaDRw6oyboMgZfaQS5 accounts_delta: 4wcq3JUGR3HjPS78hXYheRWijVP1XmEDDW5FupsGMzSV signature_count: 0 last_blockhash: 5Jq6NhSQCWgh9GhMZJ3aJJhdZdALBoX2NWjqDUrh8VK5 capitalization: 1858608451526144441, stats: BankHashStats { num_updated_accounts: 184, num_removed_accounts: 3, num_lamports_stored: 11362993793, total_data_len: 170571, num_executable_accounts: 0 }
[2024-10-17T07:33:36.275312103Z INFO  solana_runtime::snapshot_bank_utils] Creating bank snapshot for slot 296877184, path: /mnt/disk2/ledger/.tmpPq3nES/296877184/296877184.pre
[2024-10-17T07:33:39.733868678Z INFO  solana_runtime::snapshot_bank_utils] bank serialize took 3.3s for slot 296877184 at /mnt/disk2/ledger/.tmpPq3nES/296877184/296877184.pre
[2024-10-17T07:33:39.734653596Z INFO  solana_runtime::snapshot_package] Package snapshot for bank 296877184 has 12555 account storage entries (snapshot kind: IncrementalSnapshot(296863173))
[2024-10-17T07:33:44.018628081Z INFO  solana_runtime::snapshot_utils] Generating snapshot archive for slot 296877184
[2024-10-17T07:33:47.682400135Z INFO  solana_runtime::snapshot_utils] Purging old full snapshot archives in /mnt/disk3/validator/incremental_snapshots/, retaining up to 18446744073709551615 full snapshots
[2024-10-17T07:33:47.686223564Z INFO  solana_runtime::snapshot_utils] Purging old incremental snapshot archives in /mnt/disk3/validator/incremental_snapshots/, retaining up to 18446744073709551615 incremental snapshots
[2024-10-17T07:33:47.714001474Z INFO  solana_runtime::snapshot_utils] Successfully created /mnt/disk3/validator/incremental_snapshots/incremental-snapshot-296863173-296877184-8EQhtn3ZghtLpmZgwhFyH6USAyAr3hwWNszZNcwe7zUc.tar.zst. slot: 296877184, elapsed ms: 3695, size: 369360216
Successfully created incremental snapshot for slot 296877184, hash ABuvd5grMYhbrjeLmu3qCNqiD8DaDRw6oyboMgZfaQS5, base slot: 296863173: /mnt/disk3/validator/incremental_snapshots/incremental-snapshot-296863173-296877184-8EQhtn3ZghtLpmZgwhFyH6USAyAr3hwWNszZNcwe7zUc.tar.zst
Shred version: 27799
[2024-10-17T07:33:47.783915459Z INFO  solana_runtime::accounts_background_service] bank DropCallback signal queue disconnected.
[2024-10-17T07:33:47.833827007Z INFO  agave_ledger_tool] ledger tool took 122.6s

After that, the incremental_snapshots folder contains only the snapshot directory. Before running ledger-tool, it had 4 incremental snapshots.

v2 worked fine; I only added the capitalization flag

steviez commented 1 month ago

Hmm, we specifically made it such that ledger-tool should NOT be destructive; note these logs that say we'll retain 2^64 - 1:

[2024-10-17T07:33:47.682400135Z ...] Purging old full snapshot archives in /mnt/disk3/validator/incremental_snapshots/, retaining up to 18446744073709551615 full snapshots
[2024-10-17T07:33:47.686223564Z ...] Purging old incremental snapshot archives in /mnt/disk3/validator/incremental_snapshots/, retaining up to 18446744073709551615 incremental snapshots

v2 worked fine; I only added the capitalization flag

To confirm, you're saying that v2.0 did not clear the other incrementals when you made the restart snap, but v1.18 did clear the others for this latest attempt ?

steviez commented 1 day ago

I'm going to close this one out since MNB has advanced to v2.0. Feel free to open an new issue if you're able to reproduce anything with newer versions