search

anza-xyz / agave

Web-Scale Blockchain for fast, secure, scalable, decentralized apps and marketplaces.
https://www.anza.xyz/
Apache License 2.0
413 stars 202 forks source link

validator crash after error "duplicate_confirmed_slot slot=XXX duplicate_confirmed_hash=XXX" #579

Closed iammelea closed 7 months ago

iammelea commented 7 months ago

validator crash after error "duplicate_confirmed_slot slot=258293789i duplicate_confirmed_hash" 2 times today, metrics open 1.17.28 (src:286fd575; feat:3746964731, client:JitoLabs 6TkKqq15wXjqEjNg9zqTKADwuVATR9dW3rkNnsYme1ea

[2024-04-04T10:41:08.900530321Z INFO  solana_streamer::streamer] streamer send Gossip hosts: count:1056 [(186.233.184.96, SendStats { bytes: 1556690, count: 1579 }), (74.118.139.40, SendStats { bytes: 1012513, count: 1038 }), (74.63.207.102, SendStats { bytes: 811896, count: 827 }), (216.18.206.178, SendStats { bytes: 392040, count: 398 }), (134.73.71.130, SendStats { bytes: 370372, count: 431 })]
[2024-04-04T10:41:08.906734867Z INFO  solana_metrics::metrics] datapoint: external_purge_slots_stats safety_checks_elapsed=0i remove_cache_elapsed=0i remove_storage_entries_elapsed=0i drop_storage_entries_elapsed=0i num_cached_slots_removed=0i num_stored_slots_removed=0i total_removed_storage_entries=0i total_removed_cached_bytes=0i total_removed_stored_bytes=0i recycle_stores_write_elapsed=0i scan_storages_elapsed=0i purge_accounts_index_elapsed=0i handle_reclaims_elapsed=0i
[2024-04-04T10:41:08.907216828Z INFO  solana_metrics::metrics] datapoint: block-commitment-cache aggregate-commitment-ms=15i highest-super-majority-root=258293756i highest-confirmed-slot=258293788i
[2024-04-04T10:41:08.917273550Z INFO  solana_metrics::metrics] datapoint: slot_stats_tracking_complete slot=258293487i last_index=602i num_repaired=0i num_recovered=295i min_turbine_fec_set_count=50i is_full=true is_rooted=true is_dead=false
[2024-04-04T10:41:08.921215719Z INFO  solana_core::replay_stage] new fork:258293790 parent:258293789 root:258293757
[2024-04-04T10:41:08.922452314Z INFO  solana_metrics::metrics] datapoint: bank-timestamp get_timestamp_estimate_us=959i
[2024-04-04T10:41:08.922461892Z INFO  solana_metrics::metrics] datapoint: bank-timestamp-correction slot=258293790i from_genesis=1687686456i corrected=1712227268i ancestor_timestamp=1712227267i
[2024-04-04T10:41:08.922556744Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot=258293790i block_height=238652968i parent_slot=258293789i bank_rc_creation_us=3i total_elapsed_us=1316i status_cache_us=0i fee_components_us=0i blockhash_queue_us=16i stakes_cache_us=1i epoch_stakes_time_us=0i builtin_programs_us=0i rewards_pool_pubkeys_us=0i executor_cache_us=0i transaction_debug_keys_us=0i transaction_log_collector_config_us=0i feature_set_us=0i ancestors_us=6i update_epoch_us=2i recompilation_time_us=0i update_sysvars_us=1183i fill_sysvar_cache_us=82i
[2024-04-04T10:41:08.922566663Z INFO  solana_metrics::metrics] datapoint: loaded-programs-cache-stats slot=258293789i hits=1758i misses=0i evictions=0i reloads=0i insertions=0i lost_insertions=0i replace_entry=0i one_hit_wonders=0i prunes_orphan=0i prunes_environment=0i empty_entries=0i
[2024-04-04T10:41:08.925907411Z INFO  solana_metrics::metrics] datapoint: retransmit-first-shred slot=258293790i
[2024-04-04T10:41:08.925917090Z INFO  solana_metrics::metrics] datapoint: retransmit-stage-slot-stats slot=258293013i outset_timestamp=1712226883992i elapsed_millis=410i num_shreds=1520i num_nodes=31144i num_shreds_received_root=11i num_shreds_received_1st_layer=1469i num_shreds_received_2nd_layer=40i num_shreds_received_3rd_layer=0i num_shreds_sent_root=2204i num_shreds_sent_1st_layer=28900i num_shreds_sent_2nd_layer=40i num_shreds_sent_3rd_layer=0i
[2024-04-04T10:41:09.003612640Z INFO  solana_metrics::metrics] datapoint: accounts_index estimate_mem_bytes=541233120i flush_should_evict_us=5914i count_in_mem=11275688i count=472951137i bg_waiting_percent=99.06593322753906 bg_throttling_wait_percent=82.80338287353516 slot_list_len=185i ref_count=1033617i slot_list_cached=0i min_in_bin_mem=1001i max_in_bin_mem=57839i count_from_bins_mem=11275688i median_from_bins_mem=1274i min_in_bin_disk=56865i max_in_bin_disk=213852i count_from_bins_disk=472873103i median_from_bins_disk=57684i gets_from_mem=677495i get_mem_us=70497i gets_missing=83093i get_missing_us=1755i entries_from_mem=0i entry_mem_us=0i load_disk_found_count=897i load_disk_found_us=116775i load_disk_missing_count=82196i load_disk_missing_us=159426i entries_missing=350i entry_missing_us=3i failed_to_evict=0i updates_in_mem=92967i get_range_us=46010i inserts=350i deletes=0i active_threads=1i items=252i keys=0i ms_per_age=400i buckets_scanned=804i flush_scan_us=414428i flush_update_us=437019i flush_grow_us=0i flush_evict_us=29595i disk_index_resizes=0i disk_index_failed_resizes=0i disk_index_max_size=0i disk_index_new_file_us=0i disk_index_resize_us=0i disk_index_flush_file_us=0i disk_index_index_file_size=1137198624i disk_index_data_file_size=3489760768i disk_index_data_file_count=60898i disk_index_find_index_entry_mut_us=100860i disk_index_flush_mmap_us=0i disk_data_resizes=0i disk_data_max_size=0i disk_data_new_file_us=0i disk_data_resize_us=0i disk_data_flush_file_us=0i disk_data_flush_mmap_us=0i flush_entries_updated_on_disk=76619i flush_entries_evicted_from_mem=77228i
[2024-04-04T10:41:09.060696565Z INFO  solana_metrics::metrics] datapoint: block_engine_stage-stats num_bundles=0i num_bundle_packets=0i num_packets=0i num_empty_packets=0i
[2024-04-04T10:41:09.116542031Z INFO  solana_core::repair::cluster_slot_state_verifier] check_slot_agrees_with_cluster() slot: 258293789, root: 258293757, slot_state_update: DuplicateConfirmed(DuplicateConfirmedState { duplicate_confirmed_hash: G5tE5F9nbxsuTpP2Zv1YoYKTveDzw8bqQZ8W1UwYkbpU, bank_status: Frozen(G5tE5F9nbxsuTpP2Zv1YoYKTveDzw8bqQZ8W1UwYkbpU) })
[2024-04-04T10:41:09.116587057Z INFO  solana_core::consensus::heaviest_subtree_fork_choice] marking fork starting at: (258293789, G5tE5F9nbxsuTpP2Zv1YoYKTveDzw8bqQZ8W1UwYkbpU) valid candidate
[2024-04-04T10:41:09.116601875Z INFO  solana_metrics::metrics] datapoint: duplicate_confirmed_slot slot=258293789i duplicate_confirmed_hash="G5tE5F9nbxsuTpP2Zv1YoYKTveDzw8bqQZ8W1UwYkbpU" my_hash="G5tE5F9nbxsuTpP2Zv1YoYKTveDzw8bqQZ8W1UwYkbpU"
[2024-04-04T10:41:09.132250170Z INFO  solana_metrics::metrics] datapoint: shred_fetch index_overrun=0i shred_count=4234i num_shreds_merkle_code=2156i num_shreds_merkle_data=2078i ping_count=0i ping_err_verify_count=0i slot_bad_deserialize=0i index_bad_deserialize=0i index_out_of_bounds=0i slot_out_of_range=0i bad_shred_type=0i shred_version_mismatch=0i bad_parent_offset=0i
   0: rust_begin_unwind
             at /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/std/src/panicking.rs:595:5
   1: core::panicking::panic_fmt
             at /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/core/src/panicking.rs:67:14
   2: core::result::unwrap_failed
             at /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/core/src/result.rs:1652:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
[2024-04-04T10:41:09.151040334Z ERROR solana_metrics::metrics] datapoint: panic program="validator" thread="solSnapshotPkgr" one=1i message="panicked at core/src/snapshot_packager_service.rs:81:26:
    failed to archive snapshot package: Io(Custom { kind: StorageFull, error: Error { kind: Write, source: Os { code: 28, kind: StorageFull, message: \"No space left on device\" }, path: \"/mnt/ramdiskb/snapsMB/tmp-snapshot-archive-258290602.tar.zst\" } })" location="core/src/snapshot_packager_service.rs:81:26" version="1.17.28 (src:286fd575; feat:3746964731, client:JitoLabs)"
[2024-04-04T10:41:09.412319329Z INFO  solana_runtime::bank] bank frozen: 258293790 hash: 7bTvubA8CPSak9iZhWzHJhBD8hXxN1RPM8N8fze3RpzZ accounts_delta: 6stwAPVJSpx2VHWnSZyFDvki5xyg6KdCyC1RPAPcVedz signature_count: 1963 last_blockhash: 8j3QMVz6uoS7W9irV4GLcCxEReHb9iKpco8htLsVVnjQ capitalization: 573194836699236245, stats: BankHashStats { num_updated_accounts: 5239, num_removed_accounts: 26, num_lamports_stored: 1188159997820504, total_data_len: 19809894, num_executable_accounts: 0 }
[2024-04-04T10:41:09.412454117Z INFO  solana_core::repair::cluster_slot_state_verifier] check_slot_agrees_with_cluster() slot: 258293790, root: 258293757, slot_state_update: BankFrozen(BankFrozenState { frozen_hash: 7bTvubA8CPSak9iZhWzHJhBD8hXxN1RPM8N8fze3RpzZ, cluster_confirmed_hash: None, is_slot_duplicate: false })
[2024-04-04T10:41:09.422603126Z INFO  solana_core::replay_stage] GvZEwtCHZ7YtCkQCaLRVEXsyVvQkRDhJhQgB6akPme1e slot_weight: 258293790 0 0 258293789
[2024-04-04T10:41:09.422628995Z INFO  solana_core::replay_stage] validator fork confirmed 258293789 783ms
[2024-04-04T10:41:09.422635127Z INFO  solana_core::repair::cluster_slot_state_verifier] check_slot_agrees_with_cluster() slot: 258293789, root: 258293757, slot_state_update: DuplicateConfirmed(DuplicateConfirmedState { duplicate_confirmed_hash: G5tE5F9nbxsuTpP2Zv1YoYKTveDzw8bqQZ8W1UwYkbpU, bank_status: Frozen(G5tE5F9nbxsuTpP2Zv1YoYKTveDzw8bqQZ8W1UwYkbpU) })
[2024-04-04T10:41:09.422647661Z INFO  solana_core::replay_stage] voting: 258293790 0
[2024-04-04T10:41:09.425896103Z INFO  solana_core::replay_stage] new root 258293758
[2024-04-04T10:41:09.426107207Z INFO  solana_core::replay_stage] vote bank: Some((258293790, SameFork)) reset bank: 258293790
[2024-04-04T10:41:09.427359501Z INFO  solana_poh::poh_recorder] reset poh from: 8j3QMVz6uoS7W9irV4GLcCxEReHb9iKpco8htLsVVnjQ,16530802655,258293789 to: 8j3QMVz6uoS7W9irV4GLcCxEReHb9iKpco8htLsVVnjQ,258293790
[2024-04-04T10:41:09.427386924Z INFO  solana_core::replay_stage] 6TkKqq15wXjqEjNg9zqTKADwuVATR9dW3rkNnsYme1ea reset PoH to tick 16530802624 (within slot 258293790). My next leader slot is 258293980
[2024-04-04T10:41:09.427590223Z INFO  solana_core::replay_stage] new fork:258293791 parent:258293790 root:258293758
[2024-04-04T10:41:09.547698461Z INFO  solana_streamer::streamer] streamer send Repair hosts: count:965 [(173.234.17.205, SendStats { bytes: 589016, count: 488 }), (64.74.162.138, SendStats { bytes: 537247, count: 446 }), (5.39.222.120, SendStats { bytes: 525045, count: 435 }), (52.77.188.37, SendStats { bytes: 517803, count: 429 }), (104.156.226.12, SendStats { bytes: 500905, count: 415 })]
[2024-04-04T10:41:09.621556183Z INFO  solana_core::repair::cluster_slot_state_verifier] check_slot_agrees_with_cluster() slot: 258293790, root: 258293758, slot_state_update: DuplicateConfirmed(DuplicateConfirmedState { duplicate_confirmed_hash: 7bTvubA8CPSak9iZhWzHJhBD8hXxN1RPM8N8fze3RpzZ, bank_status: Frozen(7bTvubA8CPSak9iZhWzHJhBD8hXxN1RPM8N8fze3RpzZ) })
[2024-04-04T10:41:09.621591861Z INFO  solana_core::consensus::heaviest_subtree_fork_choice] marking fork starting at: (258293790, 7bTvubA8CPSak9iZhWzHJhBD8hXxN1RPM8N8fze3RpzZ) valid candidate
[2024-04-04T10:41:09.682224071Z WARN  solana_metrics::metrics] submit response unsuccessful: 500 Internal Server Error {"error":"write failed: queue is full"}

[2024-04-04T10:41:09.685281568Z INFO  solana_metrics::metrics] datapoint: accounts_db_store_timings hash_accounts=0i store_accounts=26057i update_index=11612i handle_reclaims=14i append_accounts=0i stakes_cache_check_and_store_us=11423i num_accounts=12061i total_data=40106337i read_only_accounts_cache_entries=44240i read_only_accounts_cache_data_size=399986881i read_only_accounts_cache_hits=12651i read_only_accounts_cache_misses=1660i read_only_accounts_cache_evicts=1114i read_only_accounts_cache_load_us=3702i calc_stored_meta_us=7022i handle_dead_keys_us=0i purge_exact_us=0i purge_exact_count=0i
[2024-04-04T10:41:09.685302879Z INFO  solana_metrics::metrics] datapoint: accounts_db_store_timings2 recycle_store_count=0i current_recycle_store_count=1000i current_recycle_store_bytes=1862426624i create_store_count=0i store_get_slot_store=0i store_find_existing=0i dropped_stores=0i
[2024-04-04T10:41:09.685308018Z INFO  solana_metrics::metrics] datapoint: block_prioritization_fee slot=258293789i total_writable_accounts_count=64i relevant_writable_accounts_count=34i prioritized_transactions_count=7i non_prioritized_transactions_count=14i attempted_update_on_finalized_fee_count=0i total_prioritization_fee=5323816i min_prioritization_fee=1i max_prioritization_fee=3333333i total_update_elapsed_us=21i
[2024-04-04T10:41:09.685315072Z INFO  solana_metrics::metrics] datapoint: block_min_prioritization_fee slot=258293789i entity="block" min_prioritization_fee=0i
[2024-04-04T10:41:09.685319901Z INFO  solana_metrics::metrics] datapoint: block_prioritization_fee_counters slot=258293789i successful_transaction_update_count=237i purged_duplicated_bank_count=0i total_update_elapsed_us=1837i total_cache_lock_elapsed_us=41i total_entry_update_elapsed_us=534i total_block_finalize_elapsed_us=29i
[2024-04-04T10:41:09.685325021Z INFO  solana_metrics::metrics] datapoint: optimistic_slot_elapsed average_elapsed_ms=463i
[2024-04-04T10:41:09.685327786Z INFO  solana_metrics::metrics] datapoint: optimistic_slot slot=258293789i
[2024-04-04T10:41:09.685329770Z INFO  solana_metrics::metrics] datapoint: shred_insert_is_full total_time_ms=376i slot=258293790i last_index=893i num_repaired=0i num_recovered=429i
[2024-04-04T10:41:09.685333637Z INFO  solana_metrics::metrics] datapoint: vote-processing-timing vote_txn_processing_us=51911i slot_confirming_time_us=30071i
[2024-04-04T10:41:09.685337024Z INFO  solana_metrics::metrics] datapoint: tpu_receiver packets_count=0i packet_batches_count=0i full_packet_batches_count=0i channel_len=0i
[2024-04-04T10:41:09.685340330Z INFO  solana_metrics::metrics] datapoint: tpu_vote_receiver packets_count=0i packet_batches_count=0i full_packet_batches_count=0i channel_len=0i
[2024-04-04T10:41:09.685344208Z INFO  solana_metrics::metrics] datapoint: tpu_forwards_receiver packets_count=0i packet_batches_count=0i full_packet_batches_count=0i channel_len=0i
[2024-04-04T10:41:09.685346722Z INFO  solana_metrics::metrics] datapoint: poh_slot_timing slot=258293790i start_time=1712227268903i end_time=1712227269266i full_time=1712227269235i start_to_full_time_diff=332i full_to_end_time_diff=31i
[2024-04-04T10:41:09.685350500Z INFO  solana_metrics::metrics] datapoint: slot_stats_tracking_complete slot=258293488i last_index=1041i num_repaired=0i num_recovered=508i min_turbine_fec_set_count=17i is_full=true is_rooted=true is_dead=false
[2024-04-04T10:41:09.685354307Z INFO  solana_metrics::metrics] datapoint: retransmit-first-shred slot=258293791i
[2024-04-04T10:41:09.685357583Z INFO  solana_metrics::metrics] datapoint: retransmit-stage-slot-stats slot=258293014i outset_timestamp=1712226884362i elapsed_millis=399i num_shreds=1402i num_nodes=28511i num_shreds_received_root=8i num_shreds_received_1st_layer=1366i num_shreds_received_2nd_layer=28i num_shreds_received_3rd_layer=0i num_shreds_sent_root=1601i num_shreds_sent_1st_layer=26882i num_shreds_sent_2nd_layer=28i num_shreds_sent_3rd_layer=0i
[2024-04-04T10:41:09.685361811Z INFO  solana_metrics::metrics] datapoint: shred_fetch_repair index_overrun=0i shred_count=17i num_shreds_merkle_code=0i num_shreds_merkle_data=13i ping_count=4i ping_err_verify_count=0i slot_bad_deserialize=0i index_bad_deserialize=0i index_out_of_bounds=0i slot_out_of_range=0i bad_shred_type=0i shred_version_mismatch=0i bad_parent_offset=0i
[2024-04-04T10:41:09.685365709Z INFO  solana_metrics::metrics] datapoint: quic_streamer_tpu active_connections=0i active_streams=0i new_connections=0i new_streams=0i evictions=0i connection_added_from_staked_peer=0i connection_added_from_unstaked_peer=0i connection_add_failed=0i connection_add_failed_invalid_stream_count=0i connection_add_failed_staked_node=0i connection_add_failed_unstaked_node=0i connection_add_failed_on_pruning=0i connection_removed=0i connection_remove_failed=0i connection_setup_timeout=0i connection_setup_error=0i connection_setup_error_timed_out=0i connection_setup_error_closed=0i connection_setup_error_transport=0i connection_setup_error_app_closed=0i connection_setup_error_reset=0i connection_setup_error_locally_closed=0i invalid_chunk=0i invalid_chunk_size=0i packets_allocated=0i packet_batches_allocated=0i packets_sent_for_batching=0i bytes_sent_for_batching=0i chunks_sent_for_batching=0i packets_sent_to_consumer=0i bytes_sent_to_consumer=0i chunks_processed_by_batcher=0i chunks_received=0i staked_chunks_received=0i unstaked_chunks_received=0i packet_batch_send_error=0i handle_chunk_to_packet_batcher_send_error=0i packet_batches_sent=0i packet_batch_empty=0i stream_read_errors=0i stream_read_timeouts=0i throttled_streams=0i
[2024-04-04T10:41:09.685374045Z INFO  solana_metrics::metrics] datapoint: bundle_stage-loop_stats id=10000i num_bundles_received=0i num_packets_received=0i newly_buffered_bundles_count=0i current_buffered_bundles_count=0i current_buffered_packets_count=0i num_bundles_dropped=0i receive_and_buffer_bundles_elapsed_us=0i process_buffered_bundles_elapsed_us=0i
[2024-04-04T10:41:09.685377621Z INFO  solana_metrics::metrics] datapoint: collect_rent_eagerly accounts=1055i partitions=1i total_time_us=4977i hold_range_us=2093i load_us=17076i collect_us=0i hash_us=0i store_us=3751i
[2024-04-04T10:41:09.685381619Z INFO  solana_metrics::metrics] datapoint: slot_repeated_writes same_account_writes=533i same_account_writes_size=2028282i unique_account_writes_size=17779467i size=17779467i
[2024-04-04T10:41:09.685384244Z INFO  solana_metrics::metrics] datapoint: accounts_cache_size num_roots=34i num_slots=67i total_unique_writes_size=792613292i total_size=792610905i
[2024-04-04T10:41:09.685386779Z INFO  solana_metrics::metrics] datapoint: bank_frozen slot=258293790i hash="7bTvubA8CPSak9iZhWzHJhBD8hXxN1RPM8N8fze3RpzZ"
[2024-04-04T10:41:09.685388863Z INFO  solana_metrics::metrics] datapoint: replay-slot-stats slot=258293790i fetch_entries_time=3420i fetch_entries_fail_time=0i entry_poh_verification_time=116308i entry_transaction_verification_time=22729i confirmation_time_us=453345i replay_time=312578i execute_batches_us=298103i replay_total_elapsed=489893i bank_complete_time_us=7629i total_transactions=1940i total_entries=347i total_shreds=894i validate_transactions_us=3091i load_us=37505i execute_us=572673i collect_logs_us=98i store_us=30330i update_stakes_cache_us=14816i total_batches_len=611i num_execute_batches=154i update_transaction_statuses=17178i execute_details_serialize_us=97486i execute_details_create_vm_us=65517i execute_details_execute_inner_us=589440i execute_details_deserialize_us=16254i execute_details_get_or_create_executor_us=87i execute_details_changed_account_count=3281i execute_details_total_account_count=15538i execute_details_create_executor_register_syscalls_us=0i execute_details_create_executor_load_elf_us=0i execute_details_create_executor_verify_code_us=0i execute_details_create_executor_jit_compile_us=0i execute_accessories_feature_set_clone_us=0i execute_accessories_compute_budget_process_transaction_us=169i execute_accessories_get_executors_us=0i execute_accessories_process_message_us=560055i execute_accessories_update_executors_us=175366i execute_accessories_process_instructions_total_us=555151i execute_accessories_process_instructions_verify_caller_us=0i execute_accessories_process_instructions_process_executable_chain_us=550637i execute_accessories_process_instructions_verify_callee_us=0i
[2024-04-04T10:41:09.685397339Z INFO  solana_metrics::metrics] datapoint: replay-slot-end-to-end-stats slot=258293790i total_thread_us=288073i total_transactions_executed=373i validate_transactions_us=528i load_us=8398i execute_us=157945i collect_logs_us=12i store_us=4911i update_stakes_cache_us=2704i total_batches_len=1i num_execute_batches=154i update_transaction_statuses=3932i execute_details_serialize_us=26867i execute_details_create_vm_us=20165i execute_details_execute_inner_us=180892i execute_details_deserialize_us=3859i execute_details_get_or_create_executor_us=18i execute_details_changed_account_count=822i execute_details_total_account_count=4319i execute_details_create_executor_register_syscalls_us=0i execute_details_create_executor_load_elf_us=0i execute_details_create_executor_verify_code_us=0i execute_details_create_executor_jit_compile_us=0i execute_accessories_feature_set_clone_us=0i execute_accessories_compute_budget_process_transaction_us=30i execute_accessories_get_executors_us=0i execute_accessories_process_message_us=155663i execute_accessories_update_executors_us=8287i execute_accessories_process_instructions_total_us=154528i execute_accessories_process_instructions_verify_caller_us=0i execute_accessories_process_instructions_process_executable_chain_us=153564i execute_accessories_process_instructions_verify_callee_us=0i
[2024-04-04T10:41:09.685403842Z INFO  solana_metrics::metrics] datapoint: per_program_timings slot=258293790i pubkey="all" execute_us=555151i accumulated_units=17762562i count=2906i errored_units=17103369i errored_count=453i
[2024-04-04T10:41:09.685407168Z INFO  solana_metrics::metrics] datapoint: tower-observed slot=258293789i root=258293757i
[2024-04-04T10:41:09.685411025Z INFO  solana_metrics::metrics] datapoint: compute_bank_stats-best_slot computed_slot=258293790i overall_best_slot=258293790i overall_best_hash="7bTvubA8CPSak9iZhWzHJhBD8hXxN1RPM8N8fze3RpzZ" elapsed=1024i
[2024-04-04T10:41:09.685415624Z INFO  solana_metrics::metrics] datapoint: bank_weight slot=258293790i weight="0"
[2024-04-04T10:41:09.685418740Z INFO  solana_metrics::metrics] datapoint: validator-confirmation duration_ms=783i
[2024-04-04T10:41:09.685421325Z INFO  solana_metrics::metrics] datapoint: tower-vote latest=258293790i root=258293758i
[2024-04-04T10:41:09.685423990Z INFO  solana_metrics::metrics] datapoint: bank-forks_set_root elapsed_ms=0i slot=258293758i total_parent_banks=2i total_banks=35i total_squash_cache_ms=0i total_squash_accounts_ms=0i total_squash_accounts_index_ms=0i total_squash_accounts_cache_ms=0i total_squash_accounts_store_ms=0i total_snapshot_ms=0i tx_count=1244i prune_non_rooted_ms=0i drop_parent_banks_ms=0i prune_slots_ms=0i prune_remove_ms=0i program_cache_prune_ms=0i dropped_banks_len=1i accounts_data_len=162732462030i
[2024-04-04T10:41:09.685429020Z INFO  solana_metrics::metrics] datapoint: blocks_produced num_blocks_on_fork=1579619i num_dropped_blocks_on_fork=78121i
[2024-04-04T10:41:09.685433098Z INFO  solana_metrics::metrics] datapoint: bank-timestamp get_timestamp_estimate_us=767i
[2024-04-04T10:41:09.685436815Z INFO  solana_metrics::metrics] datapoint: bank-timestamp-correction slot=258293791i from_genesis=1687686456i corrected=1712227268i ancestor_timestamp=1712227268i
[2024-04-04T10:41:09.685440622Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot=258293791i block_height=238652969i parent_slot=258293790i bank_rc_creation_us=2i total_elapsed_us=1131i status_cache_us=0i fee_components_us=0i blockhash_queue_us=10i stakes_cache_us=0i epoch_stakes_time_us=0i builtin_programs_us=0i rewards_pool_pubkeys_us=0i executor_cache_us=0i transaction_debug_keys_us=0i transaction_log_collector_config_us=0i feature_set_us=0i ancestors_us=8i update_epoch_us=1i recompilation_time_us=0i update_sysvars_us=1004i fill_sysvar_cache_us=90i
[2024-04-04T10:41:09.685446062Z INFO  solana_metrics::metrics] datapoint: loaded-programs-cache-stats slot=258293790i hits=6924i misses=6i evictions=6i reloads=6i insertions=0i lost_insertions=0i replace_entry=0i one_hit_wonders=0i prunes_orphan=0i prunes_environment=0i empty_entries=0i
[2024-04-04T10:41:09.685451483Z INFO  solana_metrics::metrics] datapoint: block-commitment-cache aggregate-commitment-ms=14i highest-super-majority-root=258293757i highest-confirmed-slot=258293789i
[2024-04-04T10:41:09.685455771Z INFO  solana_metrics::metrics] datapoint: Repair streamer-send-sample_duration_ms=10001i streamer-send-host_count=965i streamer-send-bytes_total=40819983i streamer-send-pkt_count_total=33969i streamer-send-host_bytes_min=132i streamer-send-host_bytes_max=589300i streamer-send-host_bytes_mean=42316i streamer-send-host_bytes_90pct=236585i streamer-send-host_bytes_50pct=1208i streamer-send-host_bytes_10pct=1208i
[2024-04-04T10:41:09.685460290Z INFO  solana_metrics::metrics] datapoint: ancestor_hashes_responses total_packets=0i processed=0i dropped_packets=0i invalid_packets=0i ping_count=0i ping_err_verify_count=0i
[2024-04-04T10:41:09.685463806Z INFO  solana_metrics::metrics] datapoint: shred_insert_is_full total_time_ms=366i slot=258293791i last_index=767i num_repaired=0i num_recovered=385i
[2024-04-04T10:41:09.685467203Z INFO  solana_metrics::metrics] datapoint: duplicate_confirmed_slot slot=258293790i duplicate_confirmed_hash="7bTvubA8CPSak9iZhWzHJhBD8hXxN1RPM8N8fze3RpzZ" my_hash="7bTvubA8CPSak9iZhWzHJhBD8hXxN1RPM8N8fze3RpzZ"
[2024-04-04T10:41:09.685470479Z INFO  solana_metrics::metrics] datapoint: replay-loop-voting-stats vote_push_us=0i vote_send_us=0i generate_vote_us=332i update_commitment_cache_us=17i
[2024-04-04T10:41:09.685473625Z INFO  solana_metrics::metrics] datapoint: replay-loop-timing-stats total_elapsed_us=1150000i collect_frozen_banks_elapsed=291i compute_bank_stats_elapsed=37356i select_vote_and_reset_forks_elapsed=46i start_leader_elapsed=299i reset_bank_elapsed=11535i voting_elapsed=15098i select_forks_elapsed=28i compute_slot_stats_elapsed=139i generate_new_bank_forks_elapsed=7197i replay_active_banks_elapsed=964441i process_ancestor_hashes_duplicate_slots_elapsed=17i process_gossip_duplicate_confirmed_slots_elapsed=507i process_unfrozen_gossip_verified_vote_hashes_elapsed=856i process_popular_pruned_forks_elapsed=11i wait_receive_elapsed=111750i heaviest_fork_failures_elapsed=1i bank_count=3i process_duplicate_slots_elapsed=23i repair_correct_slots_elapsed=3i retransmit_not_propagated_elapsed=1i generate_new_bank_forks_read_lock_us=0i generate_new_bank_forks_get_slots_since_us=3411i generate_new_bank_forks_loop_us=3523i generate_new_bank_forks_write_lock_us=38i replay_blockstore_us=938583i
[2024-04-04T10:41:09.685480438Z INFO  solana_metrics::metrics] datapoint: block_prioritization_fee slot=258293790i total_writable_accounts_count=1990i relevant_writable_accounts_count=1898i prioritized_transactions_count=548i non_prioritized_transactions_count=31i attempted_update_on_finalized_fee_count=0i total_prioritization_fee=269007451i min_prioritization_fee=1i max_prioritization_fee=43673844i total_update_elapsed_us=873i
[2024-04-04T10:41:09.685484596Z INFO  solana_metrics::metrics] datapoint: block_min_prioritization_fee slot=258293790i entity="block" min_prioritization_fee=0i
[2024-04-04T10:41:09.685486961Z INFO  solana_metrics::metrics] datapoint: block_prioritization_fee_counters slot=258293790i successful_transaction_update_count=640i purged_duplicated_bank_count=0i total_update_elapsed_us=3513i total_cache_lock_elapsed_us=27i total_entry_update_elapsed_us=1103i total_block_finalize_elapsed_us=179i
[2024-04-04T10:41:09.685490237Z INFO  solana_metrics::metrics] datapoint: optimistic_slot_elapsed average_elapsed_ms=479i
[2024-04-04T10:41:09.685492351Z INFO  solana_metrics::metrics] datapoint: optimistic_slot slot=258293790i
[2024-04-04T10:41:09.695577628Z INFO  solana_metrics::metrics] datapoint: relayer_stage-stats num_empty_messages=0i num_packets=0i num_heartbeats=2i
[2024-04-04T10:41:09.699515979Z INFO  solana_metrics::metrics] datapoint: Gossip streamer-send-sample_duration_ms=799i streamer-send-host_count=1083i streamer-send-bytes_total=22248744i streamer-send-pkt_count_total=24348i streamer-send-host_bytes_min=132i streamer-send-host_bytes_max=4047504i streamer-send-host_bytes_mean=20554i streamer-send-host_bytes_90pct=9135i streamer-send-host_bytes_50pct=1060i streamer-send-host_bytes_10pct=1050i
[2024-04-04T10:41:09.699527170Z INFO  solana_streamer::streamer] streamer send Gossip hosts: count:1083 [(186.233.184.96, SendStats { bytes: 4046442, count: 4010 }), (74.118.139.40, SendStats { bytes: 2445670, count: 2433 }), (74.63.207.102, SendStats { bytes: 2156061, count: 2146 }), (216.18.206.178, SendStats { bytes: 970480, count: 973 }), (160.202.128.123, SendStats { bytes: 925437, count: 926 })]
[2024-04-04T10:41:09.703398273Z INFO  solana_metrics::metrics] datapoint: cluster_info_stats entrypoint=331i entrypoint2=0i push_vote_read=114i get_votes=8096i get_votes_count=5947i get_accounts_hash=0i all_tvu_peers=0i tvu_peers=1931i new_push_requests_num=39982i table_size=721669i purged_values_size=898045i failed_inserts_size=2359i num_nodes=3922i num_nodes_staked=1791i num_pubkeys=3923i
[2024-04-04T10:41:09.703410667Z INFO  solana_metrics::metrics] datapoint: cluster_info_stats2 gossip_packets_dropped_count=0i repair_peers=3788i new_push_requests=133676i new_push_requests2=5424i purge=40350i purge_count=110i process_gossip_packets_time=1378959i verify_gossip_packets_time=1085176i handle_batch_ping_messages_time=1499i handle_batch_pong_messages_time=110i handle_batch_prune_messages_time=1701i handle_batch_pull_requests_time=392818i handle_batch_pull_responses_time=796i handle_batch_push_messages_time=890207i process_pull_resp=184i filter_pull_resp=142i filter_crds_values_dropped_requests=6i filter_crds_values_dropped_values=25351i process_pull_resp_count=33i pull_response_fail_insert=17i pull_response_fail_timeout=0i pull_response_success=9i process_pull_resp_timeout=0i push_response_count=431i
[2024-04-04T10:41:09.703421478Z INFO  solana_metrics::metrics] datapoint: cluster_info_stats3 process_pull_resp_len=37i process_pull_requests=83988i pull_request_ping_pong_check_failed_count=7i new_pull_requests_pings_count=48i generate_pull_responses=236810i process_prune=1520i prune_message_timeout=1i bad_prune_destination=0i process_push_message=472740i prune_received_cache=3551i epoch_slots_lookup=1021i new_pull_requests=214099i gossip_pull_request_no_budget=0i gossip_pull_request_sent_requests=1900i gossip_pull_request_dropped_requests=0i gossip_transmit_loop_time=255670i gossip_transmit_loop_iterations_since_last_report=20i gossip_listen_loop_time=2003413i gossip_listen_loop_iterations_since_last_report=1323i
[2024-04-04T10:41:09.703428611Z INFO  solana_metrics::metrics] datapoint: cluster_info_stats4 skip_push_message_shred_version=8i skip_pull_response_shred_version=0i skip_pull_shred_version=0i push_message_count=38822i push_fanout_num_entries=21868i push_fanout_num_nodes=47729i push_message_pushes=37099i push_message_value_count=47272i new_pull_requests_count=2560i pull_from_entrypoint_count=0i prune_message_count=939i prune_message_len=1103i epoch_slots_filled=0i window_request_loopback=0i get_epoch_duration_no_working_bank=0i
[2024-04-04T10:41:09.703434593Z INFO  solana_metrics::metrics] datapoint: cluster_info_stats5 pull_requests_count=4201i packets_received_count=43937i packets_received_ping_messages_count=35i packets_received_pong_messages_count=34i packets_received_prune_messages_count=936i packets_received_pull_requests_count=4194i packets_received_pull_responses_count=37i packets_received_push_messages_count=38701i packets_received_unknown_count=0i packets_received_verified_count=43937i packets_sent_gossip_requests_count=5491i packets_sent_prune_messages_count=431i packets_sent_pull_requests_count=2560i packets_sent_pull_responses_count=1904i packets_sent_push_messages_count=39982i require_stake_for_gossip_unknown_stakes=0i trim_crds_table=0i trim_crds_table_failed=0i trim_crds_table_purged_values_count=0i gossip_pull_request_verify_fail=0i gossip_pull_response_verify_fail=0i gossip_push_msg_verify_fail=0i gossip_prune_msg_verify_fail=0i gossip_ping_msg_verify_fail=0i gossip_pong_msg_verify_fail=0i
[2024-04-04T10:41:09.703445854Z INFO  solana_metrics::metrics] datapoint: cluster_info_crds_stats LegacyContactInfo-push=1103i LegacyContactInfo-pull=0i Vote-push=5998i Vote-pull=5i LowestSlot-push=1i LowestSlot-pull=0i LegacySnapshotHashes-push=0i LegacySnapshotHashes-pull=0i AccountsHashes-push=9i AccountsHashes-pull=0i EpochSlots-push=12134i EpochSlots-pull=14i LegacyVersion-push=0i LegacyVersion-pull=0i Version-push=5i Version-pull=0i NodeInstance-push=1016i NodeInstance-pull=1i DuplicateShred-push=0i DuplicateShred-pull=0i SnapshotHashes-push=27i SnapshotHashes-pull=0i ContactInfo-push=1033i ContactInfo-pull=0i all-push=21326i all-pull=20i
[2024-04-04T10:41:09.703454601Z INFO  solana_metrics::metrics] datapoint: cluster_info_crds_stats_fails LegacyContactInfo-push=1647i LegacyContactInfo-pull=0i Vote-push=8562i Vote-pull=0i LowestSlot-push=2i LowestSlot-pull=0i LegacySnapshotHashes-push=0i LegacySnapshotHashes-pull=0i AccountsHashes-push=17i AccountsHashes-pull=0i EpochSlots-push=13362i EpochSlots-pull=0i LegacyVersion-push=0i LegacyVersion-pull=0i Version-push=4i Version-pull=0i NodeInstance-push=1154i NodeInstance-pull=0i DuplicateShred-push=0i DuplicateShred-pull=0i SnapshotHashes-push=46i SnapshotHashes-pull=0i ContactInfo-push=1152i ContactInfo-pull=0i all-push=25946i all-pull=0i
[2024-04-04T10:41:22.475612524Z INFO  solana_validator] solana-validator 1.17.28 (src:286fd575; feat:3746964731, client:JitoLabs)
[2024-04-04T10:41:22.475886809Z INFO  solana_validator] Starting validator with: ArgsOs {
iammelea commented 7 months ago

No space left on device

steviez commented 7 months ago

For completeless, the issue was this:

failed to archive snapshot package: Io(Custom { kind: StorageFull, error: Error { kind: Write, source: Os { code: 28, kind: StorageFull, message: \"No space left on device\" }, path: \"/mnt/ramdiskb/snapsMB/tmp-snapshot-archive-258290602.tar.zst\" } })"