search

solana-labs / solana

Web-Scale Blockchain for fast, secure, scalable, decentralized apps and marketplaces.
https://solanalabs.com
Apache License 2.0
13.17k stars 4.29k forks source link

Sol exit as soon as it starts #26856

Closed airstring closed 2 years ago

airstring commented 2 years ago

version

host:aws ec2 r6i.12xlarge os: ubuntu:20.4 disk: gp3 3T sol version: 1.10.32

start command

solana-validator --identity ${datadir}/validator-identity.json --log - --trusted-validator 7Np41oeYqPefeNQEHSv1UDhYrehxin3NStELsSKCT4K2 --trusted-validator GdnSyH3YtwcxFvQrVVJMm1JhTS4QVX7MFsX56uJLUfiZ --trusted-validator DE1bawNcRJB9rVm3buyMVfr8mBEoyyu73NBovf2oXJsJ --trusted-validator CakcnaRDHka2gXyfbEd2d3xsvkJkqsLw2akB3zsN1D2S --no-untrusted-rpc --ledger ${datadir} --enable-cpi-and-log-storage --enable-rpc-transaction-history --full-rpc-api --rpc-port 8899 --dynamic-port-range 8000-8020 --entrypoint mainnet-beta.solana.com:8001 --expected-genesis-hash 5eykt4UsFv8P8NJdTREpY1vzqKqZKvdpKuc147dw2N9d --snapshot-interval-slots 1000 --no-voting --limit-ledger-size 200000000 --accounts ${datadir}/accounts --account-index spl-token-owner --no-os-network-limits-test --maximum-local-snapshot-age 2500

Problem

... sol | 65.21.203.189 | 603 | B2GgVBSHbpFUhZu9HZYz6Z7ygG3yx6ECV7zZ7qUyCMm8 | - | 8899 | 8900 | 51382 sol | 34.243.229.224 | 650 | AadYphJfRNZ3oUJjeugTJD9vid7Zqh65gCwB5JEpr8Tv | 1.10.31 | 8899 | 8900 | 51382 sol | 157.90.0.2 | 577 | A15kwXzmUJTs2DzFFpJZdqqwXYxWFbDeK1Kf1zqtqs4T | - | 8899 | 8900 | 51382 sol | 35.242.237.75 | 603 | Aby4N1dW1rDzFeVEW1X4h7kJcAqjp1nqcS9iF6b28kcp | 1.10.31 | 8899 | 8900 | 51382 sol | 65.21.76.120 | 603 | 4dV7pEcdtsDsVdj2eNk2vegg654ah86sHippBeqTtKJY | - | 8899 | 8900 | 51382 sol | 85.10.207.158 | 603 | 2PawEGWYNpMF6a9daRSVbUgWZw3PkLeY5VR31sKu5MYx | - | 8899 | 8900 | 51382 sol | 35.75.106.100 | 603 | 99XwQrQUWd8AZkUAx8FukVGwMszmB2TEjbvS7TK7V9Bp | 1.10.29 | 8899 | 8900 | 51382 sol | RPC Enabled Nodes: 743 sol | [2022-07-30T02:08:37.547993343Z INFO solana_validator::bootstrap] Searching for an RPC service with shred version 51382 (Retrying: No snapshots available)... sol | [2022-07-30T02:08:37.548173955Z INFO solana_validator::bootstrap] Total 743 RPC nodes found. 4 known, 0 blacklisted sol | [2022-07-30T02:08:37.548207046Z INFO solana_validator::bootstrap::with_incremental_snapshots] Highest available snapshot slot is 143717318, available from 1 node: [7Np41oeYqPefeNQEHSv1UDhYrehxin3NStELsSKCT4K2] sol | [2022-07-30T02:08:37.548215535Z INFO solana_validator::bootstrap::with_incremental_snapshots] Using RPC service from node 7Np41oeYqPefeNQEHSv1UDhYrehxin3NStELsSKCT4K2: 145.40.67.83:80 sol | [2022-07-30T02:08:37.898167289Z INFO solana_validator::bootstrap::with_incremental_snapshots] RPC node version: 1.10.32 sol | [2022-07-30T02:08:38.523376305Z INFO solana_gossip::cluster_info] Saved contact info for 3552 nodes into /chain/data/contact-info.bin sol | [2022-07-30T02:08:38.883074328Z INFO solana_validator::bootstrap::with_incremental_snapshots] RPC node root slot: 143737562 sol | [2022-07-30T02:08:38.883144939Z INFO solana_validator::bootstrap::with_incremental_snapshots] Downloading a snapshot for slot 143717318 since there is not a local snapshot. sol | [2022-07-30T02:08:38.883162170Z INFO solana_runtime::snapshot_utils] Purging old snapshot archives in /chain/data, retaining up to 2 full snapshots and up to 4 incremental snapshots sol | State: Z (zombie) sol | stopping solana-validator... sol exited with code 143

or sol | [2022-07-30T04:00:01.468978727Z INFO solana_streamer::streamer] streamer send gossip hosts: count:121 [(142.132.149.180, SendStats { bytes: 1892404, count: 2882 }), (157.90.209.215, SendStats { bytes: 1314491, count: 1661 }), (74.118.143.105, SendStats { bytes: 1584710, count: 1721 }), (195.3.220.11, SendStats { bytes: 1269523, count: 2026 }), (141.94.139.59, SendStats { bytes: 1084680, count: 1026 })] sol | [2022-07-30T04:00:01.473742711Z INFO solana_metrics::metrics] datapoint: gossip streamer-send-sample_duration_ms=10012i streamer-send-host_count=121i streamer-send-bytes_total=65584635i streamer-send-pkt_count_total=66251i streamer-send-host_bytes_min=132i streamer-send-host_bytes_max=1892680i streamer-send-host_bytes_mean=542328i streamer-send-host_bytes_90pct=1085277i streamer-send-host_bytes_50pct=560464i streamer-send-host_bytes_10pct=264i sol | [2022-07-30T04:00:01.503098910Z INFO solana_metrics::metrics] datapoint: optimistic_slot_elapsed average_elapsed_ms=2071i sol | [2022-07-30T04:00:01.503116301Z INFO solana_metrics::metrics] datapoint: optimistic_slot slot=143748515i sol | [2022-07-30T04:00:01.507120941Z INFO solana_metrics::metrics] datapoint: cluster-slots-size cluster_slots_capacity=1938i pubkeys_capacity=13797717i validator_stakes_capacity=3584i validator_pubkeys_len=2069i sol | [2022-07-30T04:00:01.566390414Z INFO solana_core::replay_stage] bank frozen: 143746574 sol | [2022-07-30T04:00:01.567955196Z INFO solana_metrics::metrics] datapoint: load_to_collect_rent_eagerly_scan_elapsed total_elapsed=1441i latest_slot_elapsed=3i read_lock_elapsed=1i load_account_elapsed=134i iterator_elapsed=595i num_keys_iterated=534i sol | [2022-07-30T04:00:01.571041872Z INFO solana_metrics::metrics] datapoint: collect_rent_eagerly accounts=533i partitions=1i sol | [2022-07-30T04:00:01.593357596Z INFO solana_runtime::bank] bank frozen: 143746574 hash: ASoUxjvXrLSMjTfPtY8dd8zrqEqeuaW373J1XMeT77bv accounts_delta: Fiw8c3y7bnHNAhuZZ9Zy37VYw4e9iGed8Xpfp23XuP3N signature_count: 2775 last_blockhash: 5VxkkPZst9nHFNYj5SqG1T5yzx2hw9pntdqjReaMpRVC capitalization: 525837340039891485 sol | [2022-07-30T04:00:01.593384192Z INFO solana_runtime::bank] accounts hash slot: 143746574 stats: BankHashStats { num_updated_accounts: 8819, num_removed_accounts: 12, num_lamports_stored: 13002790692988870, total_data_len: 134978537, num_executable_accounts: 0 } sol | [2022-07-30T04:00:01.593415012Z INFO solana_metrics::metrics] datapoint: slot_repeated_writes same_account_writes=4075i same_account_writes_size=105619836i unique_account_writes_size=29357896i size=29357896i sol | [2022-07-30T04:00:01.593504374Z INFO solana_metrics::metrics] datapoint: accounts_cache_size num_roots=71i num_slots=105i total_unique_writes_size=3494015377i total_size=3494015377i sol | [2022-07-30T04:00:01.593516651Z INFO solana_core::cluster_slot_state_verifier] check_slot_agrees_with_cluster() sol | slot: 143746574, sol | root: 143746537, sol | slot_state_update: BankFrozen(BankFrozenState { frozen_hash: ASoUxjvXrLSMjTfPtY8dd8zrqEqeuaW373J1XMeT77bv, cluster_confirmed_hash: Some(DuplicateConfirmed(ASoUxjvXrLSMjTfPtY8dd8zrqEqeuaW373J1XMeT77bv)), is_slot_duplicate: false }) sol | [2022-07-30T04:00:01.593529257Z INFO solana_core::heaviest_subtree_fork_choice] marking fork starting at: (143746574, ASoUxjvXrLSMjTfPtY8dd8zrqEqeuaW373J1XMeT77bv) valid candidate sol | [2022-07-30T04:00:01.594330857Z INFO solana_metrics::metrics] datapoint: replay-slot-stats slot=143746574i fetch_entries_time=12867i fetch_entries_fail_time=0i entry_poh_verification_time=17701i entry_transaction_verification_time=16563i replay_time=776391i replay_total_elapsed=851474i total_entries=396i total_shreds=1153i check_us=2585i load_us=162117i execute_us=393467i store_us=416916iupdate_stakes_cache_us=35591i bank_complete_time_us=27924i total_batches_len=339i num_execute_batches=86i execute_details_serialize_us=77139i execute_details_create_vm_us=38134i execute_details_execute_inner_us=123601i execute_details_deserialize_us=90809i execute_details_get_or_create_executor_us=22i execute_details_changed_account_count=723i execute_details_total_account_count=18083i execute_details_total_data_size=417146946i execute_details_data_size_changed=18493847i execute_details_create_executor_register_syscalls_us=0i execute_details_create_executor_load_elf_us=0i execute_details_create_executor_verify_code_us=0i execute_details_create_executor_jit_compile_us=0i execute_accessories_feature_set_clone_us=0i execute_accessories_compute_budget_process_transaction_us=58i execute_accessories_get_executors_us=1019i execute_accessories_process_message_us=377102i execute_accessories_update_executors_us=136i execute_accessories_process_instructions_total_us=369841i execute_accessories_process_instructions_verify_caller_us=0i execute_accessories_process_instructions_process_executable_chain_us=351831i execute_accessories_process_instructions_verify_callee_us=6399i sol | [2022-07-30T04:00:01.594354646Z INFO solana_metrics::metrics] datapoint: per_program_timings slot=143746574i pubkey="all" execute_us=369841i accumulated_units=14347834i count=3779i errored_units=436348i errored_count=148i sol | [2022-07-30T04:00:01.598484823Z INFO solana_metrics::metrics] datapoint: shred_fetch_tvu_forwards index_overrun=0i shred_count=1879i slot_bad_deserialize=0i index_bad_deserialize=0i index_out_of_bounds=0i slot_out_of_range=0i duplicate_shred=1010i sol | [2022-07-30T04:00:01.605476996Z INFO solana_metrics::metrics] datapoint: retransmit-num_nodes count=3616i sol | [2022-07-30T04:00:01.605501922Z INFO solana_metrics::metrics] datapoint: retransmit-stage total_time=1257050i epoch_fetch=460i epoch_cache_update=12022i total_batches=610i num_nodes=5508i num_addrs_failed=0i num_shreds=8187i num_shreds_skipped=1063i retransmit_total=33479i compute_turbine=2861718i unknown_shred_slot_leader=0i sol | [2022-07-30T04:00:01.613484210Z INFO solana_metrics::metrics] datapoint: compute_bank_stats-best_slot computed_slot=143746574i overall_best_slot=143746574i overall_best_hash="ASoUxjvXrLSMjTfPtY8dd8zrqEqeuaW373J1XMeT77bv" elapsed=1919i sol | [2022-07-30T04:00:01.613509003Z INFO solana_metrics::metrics] datapoint: bank_weight slot=143746574i weight="0" sol | [2022-07-30T04:00:01.613483940Z INFO solana_core::replay_stage] Dh6gsqSZpf8J8SUtQAQLnAGXs9ceDiHZwoSyuTgDdBP8 slot_weight: 143746574 0 0 143746573 sol | [2022-07-30T04:00:01.613537466Z INFO solana_core::replay_stage] validator fork confirmed 143746572 3335ms sol | [2022-07-30T04:00:01.613550737Z INFO solana_metrics::metrics] datapoint: validator-confirmation duration_ms=3335i sol | [2022-07-30T04:00:01.613552836Z INFO solana_core::cluster_slot_state_verifier] check_slot_agrees_with_cluster() sol | slot: 143746572, sol | root: 143746537, sol | slot_state_update: DuplicateConfirmed(DuplicateConfirmedState { duplicate_confirmed_hash: BVXfyZRrp98AnNcYE4TSYH78PD2feMtFiRjdbWE6J8mH, bank_status: Frozen(BVXfyZRrp98AnNcYE4TSYH78PD2feMtFiRjdbWE6J8mH) }) sol | [2022-07-30T04:00:01.613572212Z INFO solana_core::replay_stage] voting: 143746574 0 sol | [2022-07-30T04:00:01.613599026Z INFO solana_metrics::metrics] datapoint: tower-vote latest=143746574i root=143746537i sol | [2022-07-30T04:00:01.613673821Z INFO solana_core::replay_stage] vote bank: Some((143746574, SameFork)) reset bank: 143746574 sol | [2022-07-30T04:00:01.613698465Z INFO solana_metrics::metrics] datapoint: blocks_produced num_blocks_on_fork=3804i num_dropped_blocks_on_fork=443i sol | [2022-07-30T04:00:01.613698842Z INFO solana_poh::poh_recorder] reset poh from: 5VxkkPZst9nHFNYj5SqG1T5yzx2hw9pntdqjReaMpRVC,9199780896,143746573 to: 5VxkkPZst9nHFNYj5SqG1T5yzx2hw9pntdqjReaMpRVC,143746574 sol | [2022-07-30T04:00:01.613746053Z INFO solana_core::replay_stage] 65fCXAJeBnm5mA4euswLknNSJnuu5jSh4hXz8rkAwfkW reset PoH to tick 9199780800 (within slot 143746574). I am not in the leader schedule yet sol | [2022-07-30T04:00:01.614564567Z INFO solana_core::replay_stage] new fork:143746575 parent:143746574 root:143746537 sol | [2022-07-30T04:00:01.616148349Z INFO solana_metrics::metrics] datapoint: bank-timestamp get_timestamp_estimate_us=1229i sol | [2022-07-30T04:00:01.616158598Z INFO solana_metrics::metrics] datapoint: bank-timestamp-correction slot=143746575i from_genesis=1641867570i corrected=1659152411i ancestor_timestamp=1659152410i sol | [2022-07-30T04:00:01.616423715Z INFO solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot=143746575i block_height=129776305i parent_slot=143746574i bank_rc_creation_us=3i total_elapsed_us=1822i status_cache_rc_us=0i fee_components_us=1i blockhash_queue_us=9i stakes_cache_us=2i epoch_stakes_time_us=0i builtin_programs_us=0i rewards_pool_pubkeys_us=0i cached_executors_us=25i transaction_debug_keys_us=0i transaction_log_collector_config_us=0i feature_set_us=0i ancestors_us=16i update_epoch_us=0i update_sysvars_us=1575i fill_sysvar_cache_us=174i sol | [2022-07-30T04:00:01.616433680Z INFO solana_metrics::metrics] datapoint: bank-executor-cache-stats slot=143746574i hits=1647i misses=0i evictions=0i insertions=0i replacements=0i one_hit_wonders=0i sol | [2022-07-30T04:00:01.634853919Z INFO solana_metrics::metrics] datapoint: block-commitment-cache aggregate-commitment-ms=21i highest-confirmed-root=143746536i highest-confirmed-slot=143746572i sol | [2022-07-30T04:00:01.795656587Z INFO solana_metrics::metrics] datapoint: spl_token_owner_index_stats num_secondary_keys=82982497i num_inner_keys=138060043i num_reverse_index_keys=137945486i sol | [2022-07-30T04:00:01.801652996Z INFO solana_core::repair_service] repair_stats: [] sol | [2022-07-30T04:00:01.801696443Z INFO solana_metrics::metrics] datapoint: repair_service-repair_timing set-root-elapsed=242i get-votes-elapsed=2966i add-votes-elapsed=36906i get-best-orphans-elapsed=850i get-best-shreds-elapsed=675774i get-unknown-last-index-elapsed=16866i get-closest-completion-elapsed=8242i send-repairs-elapsed=16i build-repairs-batch-elapsed=16i batch-send-repairs-elapsed=0i sol | [2022-07-30T04:00:01.801709465Z INFO solana_metrics::metrics] datapoint: serve_repair-best-repairs call-count=22i orphan-slots=0i orphan-repairs=0i best-shreds-slots=0i best-shreds-repairs=0i unknown-last-index-slots=0i unknown-last-index-repairs=0i closest-completion-slots=0i closest-completion-repairs=0i sol | [2022-07-30T04:00:01.835892859Z INFO solana_metrics::metrics] datapoint: memory-stats total=399415681024i swap_total=0i free_percent=2.2011772020216998 used_bytes=164456132608i avail_percent=58.8258197108395 buffers_percent=0.000018458964808537362 cached_percent=56.657734305229276 swap_free_percent=0 sol | [2022-07-30T04:00:01.836638191Z INFO solana_metrics::metrics] datapoint: cpu-stats cpu_num=48i cpu0_freq_mhz=3500i average_load_one_minute=39.88 average_load_five_minutes=39.36 average_load_fifteen_minutes=35.91 total_num_threads=1744i sol | [2022-07-30T04:00:01.868929580Z INFO solana_metrics::metrics] datapoint: tpu_receiver packets_count=0i packet_batches_count=0i full_packet_batches_count=0i channel_len=0i sol | [2022-07-30T04:00:01.868950844Z INFO solana_metrics::metrics] datapoint: tpu_vote_receiver packets_count=0i packet_batches_count=0i full_packet_batches_count=0i channel_len=0i sol | [2022-07-30T04:00:01.868954017Z INFO solana_metrics::metrics] datapoint: tpu_forwards_receiver packets_count=0i packet_batches_count=0i full_packet_batches_count=0i channel_len=0i sol | stopping solana-validator... sol | solana-validator stopped. sol exited with code 143

steviez commented 2 years ago

Assuming you figured this out; but if not, checkout #validator-support on our Discord: https://solana.com/discord