Capitalization overflow on test cluster (v1.18.2)

Problem

In setting up a new test cluster, @joeaba encountered a capitalization overflow while taking an accounts hash prior to creating a snapshot. The full backtrace / some relevant discussion here on Discord:

...
thread="solAccountsLo10" one=1i message="panicked at accounts-db/src/accounts_hash.rs:849:26:
    summing capitalization cannot overflow" location="accounts-db/src/accounts_hash.rs:849:26" version="1.18.2 (src:13656e30; feat:3352961542, client:SolanaLabs)"

The cluster was created with the following solana-genesis command:

solana-genesis --hashes-per-tick 12500 --bootstrap-validator genesis-keypair.json vote-pubkey.json stake-pubkey.json \
--ledger /home/sol/ledger --cluster-type development --faucet-pubkey faucet.json --faucet-lamports 100000000000000000 \
--bootstrap-validator-lamports 10000000000000 --bootstrap-validator-stake-lamports 250000000000000 \
--primordial-accounts-file accounts.json

The creation of the above genesis file yielded the following capitalization:

Capitalization: 600000000 SOL in 604 accounts

However, here are the lamports that we explicitly added:

18 446 744 073 709 551 615 <== U64::MAX
   100 000 000 000 000 000 <== --faucet-lamports
       250 000 000 000 000 <== --bootstrap-validator-stake-lamports
        10 000 000 000 000 <== --bootstrap-validator-lamports

Note that 500M of the 600M in capitalization stems from an item discovered in https://github.com/solana-labs/solana/pull/35266. Technically, we don't need/want those accounts in a development cluster, but even with 600M capitalization, I don't think we should have overflowed.

Proposed Solution

Recreate the problem and figure out why the overflow happened. Technically, minting too many lamports could cause an overflow. But, even with those extra accounts that got us up to 600M capitalization, we should still have been operating at about 1/30th of u64::MAX:

600M SOL * (1 billion lamports / 1 SOL) / u64::MAX ~= 1 / 30

Joe mentioned that the bootstrap validator was reproducing this. Steps to reproduce:

Use solana-keygen to create a bunch of keys to pass to above solana-genesis invocation

Create the following accounts.json file and pass it via --primordial-accounts-file:

{
"tvc2ZFRKhdDgGg3aDsmKGtUWoQdn8gDnhH48KmpEdoj": {
    "balance": 25000000000000000,
    "owner": "11111111111111111111111111111111",
    "data": "",
    "executable": false
},
"tvc3NTVL8WdHuUcXEoNcCheQh8S52wjcFEcCXN6AdBj": {
    "balance": 25000000000000000,
    "owner": "11111111111111111111111111111111",
    "data": "",
    "executable": false
},
"tvc4VQteaHCQA8hPEyZjfz31JNPR8ny8baNyQYSkt4C": {
    "balance": 25000000000000000,
    "owner": "11111111111111111111111111111111",
    "data": "",
    "executable": false
}
}

This issue showed on v1.18.2, but if trying to reproduce with master, back out the commit that was introduced with https://github.com/solana-labs/solana/pull/35266

Here were the bootstrap validator args; nothing special here but pasted for completeness:

Starting validator with: ArgsOs {
    inner: [
        "solana-validator",
        "--dynamic-port-range",
        "8000-8020",
        "--gossip-port",
        "8001",
        "--identity",
        "/home/sol/identity/genesis-keypair.json",
        "--ledger",
        "/home/sol/ledger",
        "--limit-ledger-size",
        "--log",
        "/home/sol/logs/solana-validator.log",
        "--rpc-port",
        "8899",
        "--expected-genesis-hash",
        "BEQ6Kc7BLfSUTh9nfMBW7fn5L8jfA22nCXuBna4NY3ue",
        "--wal-recovery-mode",
        "skip_any_corrupted_record",
        "--full-rpc-api",
        "--no-wait-for-vote-to-start-leader",
        "--enable-rpc-transaction-history",
        "--snapshot-interval-slots",
        "5000",
        "--vote-account",
        "/home/sol/identity/vote-pubkey.json",
        "--rpc-faucet-address",
        "127.0.0.1:9900",
        "--trusted-validator",
        "tvc2ZFRKhdDgGg3aDsmKGtUWoQdn8gDnhH48KmpEdoj",
        "--halt-on-trusted-validators-accounts-hash-mismatch",
        "--no-untrusted-rpc",
        "--expected-shred-version",
        "3541",
        "--entrypoint",
        "104.154.224.20:8001",
        "--no-genesis-fetch",
        "--no-snapshot-fetch",
        "--accounts",
        "/mnt/accounts",
        "--enable-cpi-and-log-storage",
    ],
}

anza-xyz / agave

Capitalization overflow on test cluster (v1.18.2) #19

Problem

Proposed Solution