search

MinaProtocol / mina

Mina is a cryptocurrency protocol with a constant size blockchain, improving scaling while maintaining decentralization and security.
https://minaprotocol.com
Apache License 2.0
1.99k stars 529 forks source link

Pause gossip message validation during block production #7446

Open deepthiskumar opened 3 years ago

deepthiskumar commented 3 years ago

Gossip message validation during block production can cause delay in broadcasting blocks to a point where it could be rejected with the invalid-time error

The following log shows this:

{"timestamp":"2021-01-05 05:53:30.109939Z","level":"Info","source":{"module":"Block_producer","location":"File \"src/lib/block_producer/block_producer.ml\", line 409, characters 12-23"},"message":"Producing new block with parent $breadcrumb","metadata":{"breadcrumb":{"validated_transition":{"data":{"protocol_state":{"previous_state_hash":"3NKvG2GVjvWCX5jBDErFhmt72QjXkhDDXPcK5WnNP4LSSNkug2qg","body":{"genesis_state_hash":"3NLxYrjb7zmHdoFgBrubCN8ijM8v7eT8kvLiPLc9DHt3M8XrDDEG","blockchain_state":{"staged_ledger_hash":{"non_snark":{"ledger_hash":"jxX9Dv7CuPNs5hibJcJkZQS6PLrXe4WbhWYJXGDQY3aW6dsgM8z","aux_hash":"UWFEBe94e2i27XbR19jxmUeJcFiLQ2knELtRWdJspkreZmf8pQ","pending_coinbase_aux":"WZn4Ymv28C5MwX1X34fZdZcZMWm2cPj7SgQsXEWw23sj4MNGNv"},"pending_coinbase_hash":"2mzkERX27NCrwAqtoZc8AjTJrE5S49SWnzXhQDnPYArjFFdAhyfd"},"snarked_ledger_hash":"jwoak7maT75PodXDX7By9NdCi6bBNXmkbGbEfefSD8VJJQVxoix","genesis_ledger_hash":"jxX6VJ84HaafrKozFRA4qjnni4aPXqXC2H5vQLKSryNpKTXuz1R","snarked_next_available_token":"2","timestamp":"1609825830000"},"consensus_state":{"blockchain_length":"488","epoch_count":"0","min_window_density":"1","sub_window_densities":["0","1","1","1","1","2","0","2","3","2","1"],"last_vrf_output":"VYO83YSQob7JJ7DLcWYmAb_T_I5RlW2iPKLDC8zUAQA=","total_currency":"166055000000001000","curr_global_slot":{"slot_number":"2612","slots_per_epoch":"7140"},"global_slot_since_genesis":"2612","staking_epoch_data":{"ledger":{"hash":"jxX6VJ84HaafrKozFRA4qjnni4aPXqXC2H5vQLKSryNpKTXuz1R","total_currency":"165950000000001000"},"seed":"2va9BGv9JrLTtrzZttiEMDYw1Zj6a6EHzXjmP9evHDTG3oEquURA","start_checkpoint":"3NK2tkzqqK5spR2sZ7tujjqPksL45M3UUrcA4WhCkeiPtnugyE2x","lock_checkpoint":"3NK2tkzqqK5spR2sZ7tujjqPksL45M3UUrcA4WhCkeiPtnugyE2x","epoch_length":"1"},"next_epoch_data":{"ledger":{"hash":"jxX6VJ84HaafrKozFRA4qjnni4aPXqXC2H5vQLKSryNpKTXuz1R","total_currency":"165950000000001000"},"seed":"2vbcBC53PXUeGbXriQWeHEy7HaTqajxjHktQyDzWjziavavvgWC5","start_checkpoint":"3NK2tkzqqK5spR2sZ7tujjqPksL45M3UUrcA4WhCkeiPtnugyE2x","lock_checkpoint":"3NKvG2GVjvWCX5jBDErFhmt72QjXkhDDXPcK5WnNP4LSSNkug2qg","epoch_length":"489"},"has_ancestor_in_same_checkpoint_window":true,"block_stake_winner":"B62qmfJdubus1nt9iUHeZcRbgresqKRw7aZt92XxUMJW3MGBcW6MyaS","block_creator":"B62qmfJdubus1nt9iUHeZcRbgresqKRw7aZt92XxUMJW3MGBcW6MyaS","coinbase_receiver":"B62qmfJdubus1nt9iUHeZcRbgresqKRw7aZt92XxUMJW3MGBcW6MyaS","supercharge_coinbase":true},"constants":{"k":"290","slots_per_epoch":"7140","slots_per_sub_window":"7","delta":"0","genesis_state_timestamp":"1609355670000"}}},"protocol_state_proof":"<opaque>","staged_ledger_diff":"<opaque>","delta_transition_chain_proof":"<opaque>","current_protocol_version":"1.1.0","proposed_protocol_version":"<None>"},"hash":"3NK8ei2Tks8aFRyUPC5bobEojq6NqYfEk1Wag7MmjQQEGg8XL4id"},"staged_ledger":"<opaque>","just_emitted_a_proof":true,"transition_receipt_time":"2021-01-05T05:52:51.194710Z"},"host":"165.227.15.94","peer_id":"12D3KooWHosL4GxDy4UPbTbKLJbYsMj5trjRuEDQdA3tmD9tVJuL","pid":8373,"port":8302}}
{"timestamp":"2021-01-05 05:53:30.115400Z","level":"Info","source":{"module":"Staged_ledger","location":"File \"src/lib/staged_ledger/staged_ledger.ml\", line 1738, characters 6-17"},"message":"No locked tokens in the delegator/delegatee account, applying supercharged coinbase","metadata":{"host":"165.227.15.94","peer_id":"12D3KooWHosL4GxDy4UPbTbKLJbYsMj5trjRuEDQdA3tmD9tVJuL","pid":8373,"port":8302}}
{"timestamp":"2021-01-05 05:54:27.841679Z","level":"Error","source":{"module":"Coda_net2","location":"File \"src/lib/coda_net2/coda_net2.ml\", line 730, characters 16-40"},"message":"error sending message on stream $idx: $error","metadata":{"error":{"string":"RPC #235185 failed: \"only wrote 0 out of 9 bytes error: libp2p error error: closed stream\""},"host":"165.227.15.94","idx":207720,"peer_id":"12D3KooWHosL4GxDy4UPbTbKLJbYsMj5trjRuEDQdA3tmD9tVJuL","pid":8373,"port":8302}}
{"timestamp":"2021-01-05 05:54:43.131974Z","level":"Warn","source":{"module":"Gossip_net__Libp2p","location":"File \"src/lib/gossip_net/libp2p.ml\", line 300, characters 26-52"},"message":"failed to reset stream (this means it was probably closed successfully): $error","metadata":{"error":{"string":"RPC #235195 failed: \"internal RPC error error: unknown stream_idx\""},"host":"165.227.15.94","peer_id":"12D3KooWHosL4GxDy4UPbTbKLJbYsMj5trjRuEDQdA3tmD9tVJuL","pid":8373,"port":8302}}
{"timestamp":"2021-01-05 05:55:18.369091Z","level":"Info","source":{"module":"Block_producer","location":"File \"src/lib/block_producer/block_producer.ml\", line 379, characters 2-13"},"message":"Protocol_state_proof proving time(ms)","metadata":{"host":"165.227.15.94","peer_id":"12D3KooWHosL4GxDy4UPbTbKLJbYsMj5trjRuEDQdA3tmD9tVJuL","pid":8373,"port":8302,"time":83143}}
{"timestamp":"2021-01-05 05:56:04.445359Z","level":"Error","source":{"module":"Coda_net2","location":"File \"src/lib/coda_net2/coda_net2.ml\", line 730, characters 16-40"},"message":"error sending message on stream $idx: $error","metadata":{"error":{"string":"RPC #235294 failed: \"only wrote 0 out of 9 bytes error: libp2p error error: closed stream\""},"host":"165.227.15.94","idx":207830,"peer_id":"12D3KooWHosL4GxDy4UPbTbKLJbYsMj5trjRuEDQdA3tmD9tVJuL","pid":8373,"port":8302}}
{"timestamp":"2021-01-05 05:56:06.249698Z","level":"Info","source":{"module":"Mina_networking","location":"File \"src/lib/mina_networking/mina_networking.ml\", line 1136, characters 14-29"},"message":"Received a block from $sender","metadata":{"external_transition":{"protocol_state":{"previous_state_hash":"3NLxugGnhVjKdfg7NSsM7bMgxxghLureZnwkpJGuHrut63HozT8k","body":{"genesis_state_hash":"3NLxYrjb7zmHdoFgBrubCN8ijM8v7eT8kvLiPLc9DHt3M8XrDDEG","blockchain_state":{"staged_ledger_hash":{"non_snark":{"ledger_hash":"jxV28r6Y3NtxY4h97bhFStthzX27YRvgyHopGve1Jc7eBmcyDra","aux_hash":"VYLbSFXXY91rsi3fm5PLw8NqYr6sXz54xVEGTA18bjEU7XivYX","pending_coinbase_aux":"Wx9uUcKqseqLLnwAfZhnzzu36uWPgGPZqtypMaJ5AZY1FxLpPa"},"pending_coinbase_hash":"2n1MC7yxUF4muZwjyA2NgAenQ3rNXCFkYWLrqWe2Dqbuz3bCymDu"},"snarked_ledger_hash":"jx1Ff7N5TG6ixF5BPHabiPYZBWjN2ZzdGfDf8tESJd3iQLtt2c3","genesis_ledger_hash":"jxX6VJ84HaafrKozFRA4qjnni4aPXqXC2H5vQLKSryNpKTXuz1R","snarked_next_available_token":"2","timestamp":"1609826010000"},"consensus_state":{"blockchain_length":"307","epoch_count":"0","min_window_density":"0","sub_window_densities":["0","0","0","0","0","0","0","0","0","0","1"],"last_vrf_output":"kVA-JSYccxniX8iuO9xj5Yz154Llou2vLwIozhQGAwA=","total_currency":"166013400000001000","curr_global_slot":{"slot_number":"2613","slots_per_epoch":"7140"},"global_slot_since_genesis":"2613","staking_epoch_data":{"ledger":{"hash":"jxX6VJ84HaafrKozFRA4qjnni4aPXqXC2H5vQLKSryNpKTXuz1R","total_currency":"165950000000001000"},"seed":"2va9BGv9JrLTtrzZttiEMDYw1Zj6a6EHzXjmP9evHDTG3oEquURA","start_checkpoint":"3NK2tkzqqK5spR2sZ7tujjqPksL45M3UUrcA4WhCkeiPtnugyE2x","lock_checkpoint":"3NK2tkzqqK5spR2sZ7tujjqPksL45M3UUrcA4WhCkeiPtnugyE2x","epoch_length":"1"},"next_epoch_data":{"ledger":{"hash":"jxX6VJ84HaafrKozFRA4qjnni4aPXqXC2H5vQLKSryNpKTXuz1R","total_currency":"165950000000001000"},"seed":"2vaN9ENReBEzGCvfyt2uLoFi9dDgt1wp9T4eYLtTzmBbCuVeyTYp","start_checkpoint":"3NK2tkzqqK5spR2sZ7tujjqPksL45M3UUrcA4WhCkeiPtnugyE2x","lock_checkpoint":"3NLxugGnhVjKdfg7NSsM7bMgxxghLureZnwkpJGuHrut63HozT8k","epoch_length":"308"},"has_ancestor_in_same_checkpoint_window":true,"block_stake_winner":"B62qn82Mfis2nG1c8CRSCUJMs4QYK39tvhXjVFDfACyELQ82PrJ29uS","block_creator":"B62qn82Mfis2nG1c8CRSCUJMs4QYK39tvhXjVFDfACyELQ82PrJ29uS","coinbase_receiver":"B62qn82Mfis2nG1c8CRSCUJMs4QYK39tvhXjVFDfACyELQ82PrJ29uS","supercharge_coinbase":true},"constants":{"k":"290","slots_per_epoch":"7140","slots_per_sub_window":"7","delta":"0","genesis_state_timestamp":"1609355670000"}}},"protocol_state_proof":"<opaque>","staged_ledger_diff":"<opaque>","delta_transition_chain_proof":"<opaque>","current_protocol_version":"1.1.0","proposed_protocol_version":"<None>"},"host":"165.227.15.94","peer_id":"12D3KooWHosL4GxDy4UPbTbKLJbYsMj5trjRuEDQdA3tmD9tVJuL","pid":8373,"port":8302,"sender":{"Remote":{"host":"13.233.169.93","peer_id":"12D3KooWDcCkkqRVM9zh3XANJZ34YoXSZDvoG565QEbUQTQUYWnx","libp2p_port":8302}},"state_hash":"3NLgxwnRYqs7pQgWowjZoBXkwMsTLzaco5hhxRRmyvD8hpwxU5vw"},"event_id":"b7fa8b10ff17b32b1aa2e2ef508ba811"}
{"timestamp":"2021-01-05 05:56:22.277849Z","level":"Error","source":{"module":"Coda_net2","location":"File \"src/lib/coda_net2/coda_net2.ml\", line 730, characters 16-40"},"message":"error sending message on stream $idx: $error","metadata":{"error":{"string":"RPC #235328 failed: \"only wrote 0 out of 9 bytes error: libp2p error error: closed stream\""},"host":"165.227.15.94","idx":207830,"peer_id":"12D3KooWHosL4GxDy4UPbTbKLJbYsMj5trjRuEDQdA3tmD9tVJuL","pid":8373,"port":8302}}
{"timestamp":"2021-01-05 05:56:22.520020Z","level":"Warn","source":{"module":"Gossip_net__Libp2p","location":"File \"src/lib/gossip_net/libp2p.ml\", line 300, characters 26-52"},"message":"failed to reset stream (this means it was probably closed successfully): $error","metadata":{"error":{"string":"RPC #235329 failed: \"internal RPC error error: unknown stream_idx\""},"host":"165.227.15.94","peer_id":"12D3KooWHosL4GxDy4UPbTbKLJbYsMj5trjRuEDQdA3tmD9tVJuL","pid":8373,"port":8302}}
{"timestamp":"2021-01-05 05:56:47.962805Z","level":"Warn","source":{"module":"Mina_lib","location":"File \"src/lib/mina_lib/mina_lib.ml\", line 1283, characters 26-52"},"message":"Not rebroadcasting block $state_hash because it was received $timing","metadata":{"external_transition":{"data":{"protocol_state":{"previous_state_hash":"3NLxugGnhVjKdfg7NSsM7bMgxxghLureZnwkpJGuHrut63HozT8k","body":{"genesis_state_hash":"3NLxYrjb7zmHdoFgBrubCN8ijM8v7eT8kvLiPLc9DHt3M8XrDDEG","blockchain_state":{"staged_ledger_hash":{"non_snark":{"ledger_hash":"jxV28r6Y3NtxY4h97bhFStthzX27YRvgyHopGve1Jc7eBmcyDra","aux_hash":"VYLbSFXXY91rsi3fm5PLw8NqYr6sXz54xVEGTA18bjEU7XivYX","pending_coinbase_aux":"Wx9uUcKqseqLLnwAfZhnzzu36uWPgGPZqtypMaJ5AZY1FxLpPa"},"pending_coinbase_hash":"2n1MC7yxUF4muZwjyA2NgAenQ3rNXCFkYWLrqWe2Dqbuz3bCymDu"},"snarked_ledger_hash":"jx1Ff7N5TG6ixF5BPHabiPYZBWjN2ZzdGfDf8tESJd3iQLtt2c3","genesis_ledger_hash":"jxX6VJ84HaafrKozFRA4qjnni4aPXqXC2H5vQLKSryNpKTXuz1R","snarked_next_available_token":"2","timestamp":"1609826010000"},"consensus_state":{"blockchain_length":"307","epoch_count":"0","min_window_density":"0","sub_window_densities":["0","0","0","0","0","0","0","0","0","0","1"],"last_vrf_output":"kVA-JSYccxniX8iuO9xj5Yz154Llou2vLwIozhQGAwA=","total_currency":"166013400000001000","curr_global_slot":{"slot_number":"2613","slots_per_epoch":"7140"},"global_slot_since_genesis":"2613","staking_epoch_data":{"ledger":{"hash":"jxX6VJ84HaafrKozFRA4qjnni4aPXqXC2H5vQLKSryNpKTXuz1R","total_currency":"165950000000001000"},"seed":"2va9BGv9JrLTtrzZttiEMDYw1Zj6a6EHzXjmP9evHDTG3oEquURA","start_checkpoint":"3NK2tkzqqK5spR2sZ7tujjqPksL45M3UUrcA4WhCkeiPtnugyE2x","lock_checkpoint":"3NK2tkzqqK5spR2sZ7tujjqPksL45M3UUrcA4WhCkeiPtnugyE2x","epoch_length":"1"},"next_epoch_data":{"ledger":{"hash":"jxX6VJ84HaafrKozFRA4qjnni4aPXqXC2H5vQLKSryNpKTXuz1R","total_currency":"165950000000001000"},"seed":"2vaN9ENReBEzGCvfyt2uLoFi9dDgt1wp9T4eYLtTzmBbCuVeyTYp","start_checkpoint":"3NK2tkzqqK5spR2sZ7tujjqPksL45M3UUrcA4WhCkeiPtnugyE2x","lock_checkpoint":"3NLxugGnhVjKdfg7NSsM7bMgxxghLureZnwkpJGuHrut63HozT8k","epoch_length":"308"},"has_ancestor_in_same_checkpoint_window":true,"block_stake_winner":"B62qn82Mfis2nG1c8CRSCUJMs4QYK39tvhXjVFDfACyELQ82PrJ29uS","block_creator":"B62qn82Mfis2nG1c8CRSCUJMs4QYK39tvhXjVFDfACyELQ82PrJ29uS","coinbase_receiver":"B62qn82Mfis2nG1c8CRSCUJMs4QYK39tvhXjVFDfACyELQ82PrJ29uS","supercharge_coinbase":true},"constants":{"k":"290","slots_per_epoch":"7140","slots_per_sub_window":"7","delta":"0","genesis_state_timestamp":"1609355670000"}}},"protocol_state_proof":"<opaque>","staged_ledger_diff":"<opaque>","delta_transition_chain_proof":"<opaque>","current_protocol_version":"1.1.0","proposed_protocol_version":"<None>"},"hash":"3NLgxwnRYqs7pQgWowjZoBXkwMsTLzaco5hhxRRmyvD8hpwxU5vw"},"host":"165.227.15.94","peer_id":"12D3KooWHosL4GxDy4UPbTbKLJbYsMj5trjRuEDQdA3tmD9tVJuL","pid":8373,"port":8302,"state_hash":"3NLgxwnRYqs7pQgWowjZoBXkwMsTLzaco5hhxRRmyvD8hpwxU5vw","timing":"1 slots too late"}}
{"timestamp":"2021-01-05 05:56:47.965145Z","level":"Info","source":{"module":"Mina_lib__Coda_subscriptions","location":"File \"src/lib/mina_lib/coda_subscriptions.ml\", line 192, characters 15-26"},"message":"Saw block $precomputed_block","metadata":{"host":"165.227.15.94","peer_id":"12D3KooWHosL4GxDy4UPbTbKLJbYsMj5trjRuEDQdA3tmD9tVJuL","pid":8373,"port":8302,"precomputed_block":{"scheduled_time":"1609826207964","protocol_state":{"previous_state_hash":"3NLxugGnhVjKdfg7NSsM7bMgxxghLureZnwkpJGuHrut63HozT8k","body":{"genesis_state_hash":"3NLxYrjb7zmHdoFgBrubCN8ijM8v7eT8kvLiPLc9DHt3M8XrDDEG","blockchain_state":{"staged_ledger_hash":{"non_snark":{"ledger_hash":"jxV28r6Y3NtxY4h97bhFStthzX27YRvgyHopGve1Jc7eBmcyDra","aux_hash":"VYLbSFXXY91rsi3fm5PLw8NqYr6sXz54xVEGTA18bjEU7XivYX","pending_coinbase_aux":"Wx9uUcKqseqLLnwAfZhnzzu36uWPgGPZqtypMaJ5AZY1FxLpPa"},"pending_coinbase_hash":"2n1MC7yxUF4muZwjyA2NgAenQ3rNXCFkYWLrqWe2Dqbuz3bCymDu"},"snarked_ledger_hash":"jx1Ff7N5TG6ixF5BPHabiPYZBWjN2ZzdGfDf8tESJd3iQLtt2c3","genesis_ledger_hash":"jxX6VJ84HaafrKozFRA4qjnni4aPXqXC2H5vQLKSryNpKTXuz1R","snarked_next_available_token":"2","timestamp":"1609826010000"},"consensus_state":{"blockchain_length":"307","epoch_count":"0","min_window_density":"0","sub_window_densities":["0","0","0","0","0","0","0","0","0","0","1"],"last_vrf_output":"kVA-JSYccxniX8iuO9xj5Yz154Llou2vLwIozhQGAwA=","total_currency":"166013400000001000","curr_global_slot":{"slot_number":"2613","slots_per_epoch":"7140"},"global_slot_since_genesis":"2613","staking_epoch_data":{"ledger":{"hash":"jxX6VJ84HaafrKozFRA4qjnni4aPXqXC2H5vQLKSryNpKTXuz1R","total_currency":"165950000000001000"},"seed":"2va9BGv9JrLTtrzZttiEMDYw1Zj6a6EHzXjmP9evHDTG3oEquURA","start_checkpoint":"3NK2tkzqqK5spR2sZ7tujjqPksL45M3UUrcA4WhCkeiPtnugyE2x","lock_checkpoint":"3NK2tkzqqK5spR2sZ7tujjqPksL45M3UUrcA4WhCkeiPtnugyE2x","epoch_length":"1"},"next_epoch_data":{"ledger":{"hash":"jxX6VJ84HaafrKozFRA4qjnni4aPXqXC2H5vQLKSryNpKTXuz1R","total_currency":"165950000000001000"},"seed":"2vaN9ENReBEzGCvfyt2uLoFi9dDgt1wp9T4eYLtTzmBbCuVeyTYp","start_checkpoint":"3NK2tkzqqK5spR2sZ7tujjqPksL45M3UUrcA4WhCkeiPtnugyE2x","lock_checkpoint":"3NLxugGnhVjKdfg7NSsM7bMgxxghLureZnwkpJGuHrut63HozT8k","epoch_length":"308"},"has_ancestor_in_same_checkpoint_window":true,"block_stake_winner":"B62qn82Mfis2nG1c8CRSCUJMs4QYK39tvhXjVFDfACyELQ82PrJ29uS","block_creator":"B62qn82Mfis2nG1c8CRSCUJMs4QYK39tvhXjVFDfACyELQ82PrJ29uS","coinbase_receiver":"B62qn82Mfis2nG1c8CRSCUJMs4QYK39tvhXjVFDfACyELQ82PrJ29uS","supercharge_coinbase":true},"constants":{"k":"290","slots_per_epoch":"7140","slots_per_sub_window":"7","delta":"0","genesis_state_timestamp":"1609355670000"}}},"protocol_state_proof":"AQEBAQD817h_B3CJGLr8rtou6DKvciUA_CYrP34N0L3y_B-9ht984Lr0APzz1Lhd78TvAPwgwzEMFIjilwABAPx48iPXBION4vzNf3sHKIkubgABAIiO6W4MCYv0opCA1nX1EMYjjdZdcLYwsOcuecBY-HsZAQB_adqgX4XOvvC1W-oE-zCaImWLyxT9kuKLAy37bm7gLQEA_MJ4AZH-PXOw_PM5Xw8YQgiqAAEBAPyZJoQ3vdqaRPxq7mF3rdl4GAABAQD8kwdx371Twmz8DDwCcwcq-KoAAQEA_Pq3alGj3rdU_HpZSAYNXnrkAAEBAPySWgqtcT78BvyI6TVRWSOiWgABAQD8la7Dh-grzuT80NmstF-_agEAAQEA_J6U67VRFwxg_JBuCeWN0NroAAEBAPyIASPGt_H6ZfyK1qB05ufJiAABAQD8ctY-zGQYZDz8t7ud8GqDodAAAQEA_IoRTICNuZpE_Fy0mtGz9R8FAAEBAPw9mFZtvmKwzfwH9ENiSllRnwABAQD8UYB3bDmiAPD8QIJIYJHYQYQAAQEA_AOgTNTXYatV_B3-It6Ix-FnAAEBAPyvPnr3Y-y6OPzXdzpuHR1DuAABAQD8nRTPw-HKWzr8SCsv8UexhrIAAQEA_GX9_ayOB-M4_PYfltJF1AbsAAEBAPxVI-FrbIxX4fzYdaE2Qa5kAgABAQD8O1bCao9-0Tb8A59dw4Ig-qAAAQEA_Gwrz308GQQs_EbVfIqgNS2-AAABAPxVFU_yKlJqPfw6P2a_INve-Pw0vGuCQfkDqPww_SRlMGVHIwD8vU628eJyHOpE2mSqiPGMSnX5v_Db5z7d1rhDI4igNdxA0PZSkqjQe7xVCd-X3KyjvAt7rY8V1Sy4NLLw3PoWAQEBAPwNurgQWtZc2vxW-QrkThOAqgABAQD8YtqGkM_-iLv8S6honyhxfxsAAQEA_IQkpD5ff3dL_PWFAPe70YGdAAEBAPzAeWI2klky4_z2Hj0NEK8_SQABAQD8_umSEQ7qr8r8X6cp3-PRWkMAAQEA_HO_cPO-gTRc_HPIvTchhYGrAAEBAPxz9lmvhT4sWfx3bwZB8cgWpgABAQD8yJPyZFS50W78kFRDF4eiBEgAAQEA_FHSjzhX4lob_C2rebP-lwJEAAEBAPy1tQY4jkPGofzBq6R0hFWIXQABAQD86ioBtX9NOof8t8xEfolwlcgAAQEA_N1NKa2P8BD9_EwCHCeide-_AAEBAPzeVDeFLTLis_ysON0ilP8lcgABAQD8XimnEw9u3ID8HgU5ShQTbmQAAQEA_A_8Jbh3PeSS_OSx-9UTwrNJAAEBAPwXDEuk3MzD0PztPHvYaMHmgAABAQD8Cpx8UO8bZ-T8yJzxJpyytn0AAAEBAQD8JowTSM4d0M38VdgVlhakOA4AAQEA_Kj9CNgQYE9c_DvRUZ2bbXAYAAEBAPzP-bOJEIEGCPxvv4HLraFRxwABAQD8HVt-uSoKajD8gAiMGqVxst0AAQEA_JWZ0oUvufXY_AHbTRjb-T9-AAEBAPxiKxWA8OU6tPwPlVnn1CyYBQABAQD8MLEQJRMViQv8Wgudt73phXAAAQEA_A_OlF2x6O-j_K9vyGWbSsPGAAEBAPwcPKmO-hfW5PwibIQwPsY6WQABAQD86slv6MmYrif8UPaqoKaN91QAAQEA_KWdLTd_RTBs_CGhZbTsmnayAAEBAPzpDMrPMDvTUPyjKbBO5HW0zQABAQD8BFwS5CGn6Q389XBXuXbYz_wAAQEA_HU_iSQBC34V_ERm_BDdgU5GAAEBAPxnOgIWM9a58Px_QUZ0H3o3uAABAQD87Dx3l4ZyKp38camX6ou6lf0AAQEA_LLQd-lmOeio_J71b-7najTLAAAAAQACbRK1HtPQKCxREBBZccdDjHU0-QLp8FLurP98jkEryTDpOYoPwJJWZCsESiHlH6sEfsqov-UqAVAdUUuYpd-sLJ-UT6Yx-QTKuh6m-jRdEsUPeMfiUdZTuXZUmAYsXrAKtsYF3TPplpwwZ1qCIWOlPI6hNfHYm1zuKUyl99Kd7CcCAQEA_OmBgXhHjTm1_Lh55iHcSG10AAEBAPzfCsWG11IvN_wZa02iarsw0wABAQD86zyvzP9F2qD8-Pfv5yFROD0AAQEA_MjJIFo-9eEa_Ji2qEcKH8wmAAEBAPxsoIuVu1_YM_zALmf-HomoWwABAQD8KBiyqkF7zM78_97PuV4TRW8AAQEA_NYmEh3D1pNQ_K20I78zH4AqAAEBAPwmQqpPc6zfs_w373QXE_-WPQABAQD8kUr6FmQPqNz8-6OSL0cwPbcAAQEA_NYFzkdcYrcH_IkPpsiL5llXAAEBAPzJm7iNqrVwafxQNyU1925A8AABAQD8ETBI0EO7HrT8qeAZ-jGZrTcAAQEA_IHqVKRJDilz_GxbwBTjfzI9AAEBAPwoz7-tg2F2XPw1cXkIYzrc3QABAQD84_e3EQCIMvr8iSdQA13SdvAAAQEA_Ig17d1DV51d_AIqQLC3A6SSAAEBAPxYBmPQnydQDfwXfKhj7gUwOAABAQD8advHbJSKpbb8F3ZiIGA9xJkAAAEBAPwMxWnKbTOhCPyLhhJ9-g_wwwABAQD8iQCz_prWi3v8ESi5ao3S87MAAQEA_MEHNYZwuM9z_Jzn68Ml7JtyAAEBAPwlT6tXKLZbCvzygOs6g5ivsQABAQD8uSqnVrRwc638_J7x1SP5TzYAAQEA_AB8L45iHIdZ_IfMJqJz9secAAEBAPyv8raeHYJUI_x-9X320Wu51QABAQD89oaQoND3exT8aCokQM5iXmIAAQEA_A6tVjJjG8av_PvhH6EQcoAJAAEBAPyRQazKvh5Y-fymybc-mdUeVwABAQD8vcNkzaNQTqr8aMX-wQrnFNgAAQEA_G3eXoLfrB2y_KUH28UXogj-AAEBAPx_qubp1g9Ogvwsf7lOmDr2_AABAQD8ygQbcSuIMcP8KSautsesOZEAAQEA_O9Rgf1Hjw_c_IeVO8RDeqkAAAEBAPy_MobRHtg4YPyrBaqicLyz-QABAQD8Wkev5eDSdZT89tLDrgKny9EAAQEA_AR8Lfn2D3i-_FTi-zKRWD3hAAABAQEBu9vieqa7yWTyBogbEOHhfNVk632asr95SVubw6gq7SQBAdCnZUZt6h572RmRI_dXcEnlS7vIiSgM4bfcc-qV2gosAQHhd2UhWQTvwkK9YA4TytkCdcAXkBXSF2_dkqCrgMRvAgEBoADmjNTspznDWrTT4JyGLsq3UMxq3wAzWZr4VFVUnjgBBU9HCaXPXLx5CIKdYkqsM9LNjxQ_Hb11UofUKJYo3EEQ8ZTV6DyAFehs5rbVz0lwqWx7SLnbp4NAtf9YJQcpDj_21Z_bWK57qtv3wMiGyeM9nNFa22FrCpJF6rPu_77PBVu5ngcjorqEQigXGbY8ZhMTjdzWNOPZ0iozNDQqAZ8UTaYteXEwXUogP7AQyOqyN32AxWyaP300aoVa_AKZ6iMBAWzva53gv6y9FkUwMUJ0BGgDq9cuB_JTnvV7fSIA-DghAQHesVUPCkFghNoMPhNWe_gO4gYiUcU__aco70TzY_2BBQEBXbDWijZLlz7niCWG4GGZ3TaoIll5bEJYsCRf58OInQ8BAQFBitc7DcQId7r7CGqAfWoHrkDTdydg8g-uwAZOk08cMgEBLLdqUsOV1gR4kRdhEWAvfY_Wp1l8RzIiFMWBe7ux5w4BAYry_Hjg6IlUY4Cg6-r6wF3jBjPtKnzcyQu7X4KXPhsYAQHQpnlxZjEJGgjpmOCrBAWXVDO4BD622lMFhDRDwhf9FwEFqUYruV6C3S0p9hJhsT2XS5L9JrydH7uQoyT7n0bJ2C0q12lv614FlN-r29nQ1pNmyn_f08wIeMSo840NjNCGKwqDQq9G2WPlhbCx-P7h9FZBax0pmoVYCcd1Yv5mNIUa8jbFV7oGL1E5SUDNghiKDvkKd_865csKWaT6pY7EIBOD2kKfC033RPXXYQonRs5_bHH1MuYJl3HHaOkneUeLGQEB8C8VcfW03D5DcRULZwMm-YdDIyk5pI1qZm0UCUrzGRoBAakbH15rA8o8tRPz13buaxk5wC8FmMnY2mK73RJS0lUBAQF0eahIbeheumvy9WmdAPyHrASjEIyQZu3MDs7ydjtOPAGVtYUn0zrftsXRFxg6Mj6fYhavBWI0KTNV2fPlOBf-OCbba1nJHwANDieDfmCpM2JUtc0zSsvjMqYIzt_4IRQAAQEBAQEBJXktCuhh87F22pDN5uMubkKiT0A22nOc8dBCcnevwSDg74lBb0ihBVJHsSSgd3LvZB_nNIIeH1ioDzD4VqreKAEBATaAgxCjaKfSFB43cKNDNavvr9oMTgKCeLiA1bVP9d8L2kw5adp9JCDkO8PkcPoxYfmOmeh4MeRGwdTGhoOOMz8BAQHcak7nzlFjRzF_2aGbNsh3n8uHI4gGVhyySHjcqnEiEOfQfq4Kab7csC0SGYRfiegkznDd2-PvFLbTywrbjS8cAQEBJV03o3nDo3ZIRLld_iqI5z8pybtI1I-YKR1F6UzCWhvaqxg9vuvnGBwszPeOpQ3qxH4o2yxYFJe2r3Akbv3FAwEBBQEBLtd5GcpAq0CXCo4MHjivoVI1HTQgRGBfrXQo2bVrWCdf2YQ8Kx0DLuPDDCz1PhRuFpT8NO7R_9Ry76hu2Yh7FgEBZLqZH2LgVsgIM9XHYjI9Mt9q5pecTOQK1-Tx0vlSpDL2uLGY9LpP0p_M3lXKWeFgYzg_iV3FyxypN0YrhxBANgEB7DK7vTKCWV_sKg1ZgtjtdQKwy4Ll86PCuvo45yspqSNbLKD-6CmidKqMwrpC8kUy55q2XqvqXYf_D0hdjteKKgEBEauh4_QXC5M3E2R_ayrao0JQCZMmpvprxfTyV84KfCKbaD_g0l8a-AgGlscrqX-kxOWmhKS2stEc8frReT2BFAEB4CoZk1iQHn6C8t2o3UOQVb8IiBChSyM2t-pnCzKpyRf32DYdred9AyNj2YoUNHIaqY7sT6ZONNzckiPtPbyKMwEBnRfB0qUGMswtDSYzcWPqrIkhuAwo8o15H69_6oXS1hlgZK5Qs0qJiR_RC--DlMMWLEjhIJ2wqZicOLZfwpoXBgEBARGph9Va-eu1Zd-tZU7quXFPYFu-NiQIHWmvL08m9OtCBlN6p84G_-LITuFCC6gQncHvB4dgdyR_qH6ZGtc-LD4WaxVg59JScTU3zczz_YqTfPYdZlUY7KSpLxTiAV4lCzzcSl1luosGeRwrqxypxmn47n674PVF2iAzVcC6ef3XNx4vr_e9TiLbk2XO9GrZjbOJVeNqZZpH8OX1NJkvj-Qp6PL6o5aY4y4yRLcvzhzAnrNR7ARearSIT3MlL7o49wIAUZiE6cxVfsRMBxlQAugipgzDaNE9n6hjs8Lzy4V0HcJ4zlxLCZco8aSp3Qwe6hhEyiqIGITj77Hqkl5ttAkVWWJ0bJc1JJqCv_PiBYStKth4KpKY-wUl7B4O1_3LHSaNl6RM7tQqW_2EP0wkWS4CzemHd19GaGwYNDL4afodD6lU7CuAljnGYpZ7N1WLf4YlNq0Dly91xZGdy4qj9NwjGV49Wi5BPfzPnPq9KItf1N0B0JPPsTHINSCrwFlY2zldQ4_DfhUga6oP4d0rKFS0C_lWx3IPBuRQ9V6_7UPFPnfYdrykYmcc4XElxQzwawxUaT3qDAveKId_a8Wr8804Pg57FKxYirPJBri15QWwfWSDlk1BS1DK98zzf0OsVA7L16atPhKCHR-IwzMXEolvgP7w1XlzCg4BAF7DB0UXKvJQt6ki3PqSnpKcYzBReM-X0el4XCugHQsHNs_nqHc4vJ5m8i39yC0GmHXCF8aOs0liCLjs0084ko6xMMxdBTetmIvcEm1ZKcUm8xOk6SAie_ceRM_FVZPJErsdu1AoAUZQYt_JoannbI8ebdNKLl0jcrFEjDG1UYhkgYUsmHUxgK3QVzc6ubmL9j9VL8eai7d0WruWjyDuQaRr1JghJxkmW3oL20n_Ub92pOxaspHUOuUSEFiyB21wVUUalGAKKPkL_YZH107yPVU21Gi58bhkO65Y5EQOnBbnBJVt5hUxRXzZYHaJsHJL90rvpB9ZXQEBS9hqPTN0LPxZv42m_St-EmM6CSPSYtdkollbUSB2aqQxpLtUbeATcd6QA7aNNXWqINUM20yLIYNPzdj0iG0W7HEBQMxqKu1HcB6b-jMKVX1zBdXVFBSfLzE430LOw7V6ZYHXkVXGUIzzqKAK5wRpW4N3IdpS9Si18_9O8HbxEvt8plWCxikrKM-H-SjPGmSwwLvqF1PmRaCpWDjpcsxwnuA4eidAvYGqxRH2590JfYHmnMSWYcgY813PBBS5WYCuBhC3r10wSY5QAhDOfhBFYP_oeIlfbC6UdAj3FRJkqoJtdc7mFxMhswD--ieEO0lIqtW8zwqWWhKiU07n3pYMJOlCksdXKlHrjNjWujAiCKUQz8tIVBbRi13wb8w9kZ7meNDZii0CgD4owK332RHnCV-3fISFWKFjuWJzbHRJ0peLgQ5AZwKwrnklIKllFqt4Yy5pd4bk8uqnXqA1YzagIQKLUfaUAMvIDWKgkd0hAo7euBKeAxdTTqG3Af8S5zySBdu34fQJZWYQcJ3VTxjOHCksBj-4HFXNZoeoxQHDY3tgodNGj5kj-JSWZuRAN44EdAybkKoj-WTLgMANfUkicPz4YYovybxqclet7pMO6Gl4Hb-FWjWlcEH31n-m01ivsqiGG-hHlSpPkygL0iuemdKAdTdZ2YFiwb43Gusr3VXf1qXjlMfun6BqOwa4DCYe_rfYjedOfGyX0mzIGBd6H3NeZ-AxiqP_8Li901Ev7wfXwx9zN9s-LBptLNjKyGmucHdyungXnlY3kqQ61BjTnpenbj7fQ6lNndz-XHPib4Rxokemo-fV-Ye14-tUF5FsA_66_g7gw5FpctPgW2sFqIX_qAjTUEyNKzVNdw4Gcfffkjf1DGKnwpuXllK1JuFdyTRQHfpnWw0xctJpcxtn-WEBQ2Gvxp5DRBTL1m13jNG8MUj2aK3qyWpxFI_iH4lDp5NzvkV3xB_LxTPuZgUG5qQFV2vvDnZBnLcEWUwOQtr08PMprd1VPnSAqWknTvOO_uYto-lXTREV8tzC1TROL3--Zn9xn2Om0KCr_s4JlFI-yDJLSgg4ysyNhloBEEz3e8NASPwuKhMHEIKWqFLl2nSslmqZY0rY1t9nwBs6pfb4uWNMtecx_REP7cmPrin5T7hVt4x8JKr1R-dzQg7vERa0ZQwFB0LuIaQF6aR1fg4DeEknIYKvQtPYfbaxPoN2WVKdeLn9bWYrEjFQe5jzYXXyAnXawAPQOuzDnaoPxEW3m6eJrxKuadHJg_CB5SQll5Rbvqp2FYt7VQWPbi-VEn4xjwADRC1YSqeibFQUwjcj64wvIRnaEEqB7x7GCDz-4xsaVrcq1qWobx5QoEBg69SCU2UvFCwbmRc5aQkNXR7rkpJo_ZL9Noh8b4WvxUDeKZ0RQoOpnqPelR-1QAQlBDk5UDeOs8nVmTXEBt9aeXz8pNCp6XZNjdPll-EYMEht6mYjkppv2HLXW1SWJUsZg_p6mppWwymVLiD8WKMw7eMEufozb7CbG7iYpjZ4ChgDx01HVRbl31gxQD3dwje_40Zv-cR5zYDdj58DZtEG6SX_DDAZrxf1huxDU4G_DPAECjXRIrqeJdpdmlbdWzx5cVSZxMoLDLYoIHOYjNkhnrxTLTRzSk6zXAd9OG5Lrvssmw5AVaVMqTx-cAIE4RO81_W1TMY46lYfnBdUkoyQrQzH2RN_GZBxwsTTVMM-JAG2vYEDvPZjdo5Q3P-M9nDSDNA7lVOe9BPoEg212qE4pDtMFuUaLxy--9mPsdshXzRiSc9vPLnt8JJ_GEp37A3lBlZV8M1esqE_sd0DbPAmKlCIlsxHe0RiqNtQqOX6MipmNEJPV6cxifqoC4deUlhxAVyN72F0EcF0NIMMXgc7RaNPMULE2HnYMiCNowEDV3z5hySrP2yHADTFfu1f6y1tTIv4EG9uJg9e3X-KU9cxQAr4JNLLdR-u54xfov3ZOue4KiUv1bllmk8AWiPNAZhm4-kTnTMGofutAy9ml7UHgpFt9gJVL5Fmy5zI5FiK3n9UoFp9VPvBfTQPUvuXXR3xe8l1wCsgUFLd9tm15pOzjljhlJWc8T5vi_V3GwvJAIiJkBXsPyt6v4pBZBfOxy7XZ6lD0bsXqOENPexPS9AvAQEBERb_v6rwwm_0JT7-AIWgVBhr_su9hU9oycG8trdJ4yEBAaSg8KwNmj5ESqkn8lAlbyFUEfAwY5RedRdKhVw8hp8iAQHYl55JVs567ybe5z2i75NeuQMij0Rk42ktlbpP6mUUKgEB_4x07ucxlU-eBoITIgq2KIqSqcM8cWb7LPxvvTlOVigBBetc9r1pEtz9AgG1t5TVGFScNWaMjU-qLkkjCWNW3IwAmKupubaJ3lD3ZTMhitYoEE9I1nyfqNa_mBcGh4ELpCmP-UBz752wVZR5aKb12gwDr3VDYc1rxzOugUNiyL35IRc-aQPsChipGuMpM-DDB2odjK0Jp81wEYxA1923Td0mipfSe6n7364OHNZJvszvpzy-qwiavHdqF5The4NDghEBAfRqGYspINKZKP3BjJe3zAAvL0SrOmGVvOVxxcOJ-qUzAQFOjbvc-jxzVPXzfZflJ48MmzK0ofHixiqJq758ES3DPQEB-Gw-tUeXX-YzRuaxMlX53bVd3XTK-0TjBTIofs1o3yoBAQHZDHPhbHCXrsa0Mwv6xcDFNqCgvjIO2hexUS5tg8lSKgEBkJCfN7qHOMUkhi0BwYCPpOa3SNamDjKA6wmDEts_Ry0BAfJKeGG_c6ydJCBp2PjwBXYLjMI-5QlFQ19C1hAuKtogAQEWQRq_0_Mriv-KhcYsWbKLkBg2L8NpqIBDYE54gj87MAEFxjK6LxGPgWJ3mkFKLKsD-H6PkQdFUy1tqwMUDuz8Qgu54U6OVf6iwviFM_o9f5jGraFERg1A2e3cOmDG6q3CFZCzP2IyyldSegDXSjVNhUqolrSYol5NTx46pVoMF-ASKJPZecnD5CuWeVJt1lDNl1OEzV_IovzkyY_ruCK5OSIAwHBfBrvMOhQ5EXBvIkjsfGRKNZDquSHtNz3vAvRqHQEB4JRhQLKXrntoJf7GDLJpCfxSAzl5m4dS6qNM95gqxBMBAdZ0RCVdhl_FpvPcx2Y_fLKETUgZCxl3vxqEjf9fqj07AQH2A7gnFW84pVpFqtayjdxa9U2-_PGB6r4a4IjXyHgaKQ==","staged_ledger_diff":{"diff":[{"completed_works":[],"commands":[],"coinbase":["Zero"],"internal_command_balances":[]},null]},"delta_transition_chain_proof":["jy2cuJtmoeouyXfHcXXMgKLEBUXAJsemw6KfJTrtbd3zQ9dhJVR",[]]}}}
{"timestamp":"2021-01-05 05:57:02.181711Z","level":"Info","source":{"module":"Block_producer","location":"File \"src/lib/block_producer/block_producer.ml\", line 379, characters 2-13"},"message":"Build breadcrumb on produced block","metadata":{"host":"165.227.15.94","peer_id":"12D3KooWHosL4GxDy4UPbTbKLJbYsMj5trjRuEDQdA3tmD9tVJuL","pid":8373,"port":8302,"time":103805}}
{"timestamp":"2021-01-05 05:57:02.182548Z","level":"Info","source":{"module":"Block_producer","location":"File \"src/lib/block_producer/block_producer.ml\", line 565, characters 20-35"},"message":"Successfully produced a new block","metadata":{"breadcrumb":{"validated_transition":{"data":{"protocol_state":{"previous_state_hash":"3NK8ei2Tks8aFRyUPC5bobEojq6NqYfEk1Wag7MmjQQEGg8XL4id","body":{"genesis_state_hash":"3NLxYrjb7zmHdoFgBrubCN8ijM8v7eT8kvLiPLc9DHt3M8XrDDEG","blockchain_state":{"staged_ledger_hash":{"non_snark":{"ledger_hash":"jwf54CDsSicYARUa9JMCnN43a4cfvsHESTwvNH3HVQxivVaZCWC","aux_hash":"UvwToEGRwWmAH2qE1ZSHZhZ4VzuwmQc6pWYrzhtLdCg5hK2R12","pending_coinbase_aux":"XnsksovkTnZdKFmoTZkoetqdVE4wnqoNsRk8PTnMNWr6o8nEnD"},"pending_coinbase_hash":"2n1HdFfKv9hJFXuDhh4LfCEyWrGqm6ZHpdWKf51DcbhaviBpyVzJ"},"snarked_ledger_hash":"jxcCCAV1UUggSH2RcVLtMVy54dvxZwZ1ndpdMDjNc5sP5yownYh","genesis_ledger_hash":"jxX6VJ84HaafrKozFRA4qjnni4aPXqXC2H5vQLKSryNpKTXuz1R","snarked_next_available_token":"2","timestamp":"1609826010000"},"consensus_state":{"blockchain_length":"489","epoch_count":"0","min_window_density":"1","sub_window_densities":["0","1","1","1","1","2","0","2","3","2","2"],"last_vrf_output":"kxJaM3ECnJRC3ce4fmsFVdK-RDnNKxdw1woZFDq_AAA=","total_currency":"166055800000001000","curr_global_slot":{"slot_number":"2613","slots_per_epoch":"7140"},"global_slot_since_genesis":"2613","staking_epoch_data":{"ledger":{"hash":"jxX6VJ84HaafrKozFRA4qjnni4aPXqXC2H5vQLKSryNpKTXuz1R","total_currency":"165950000000001000"},"seed":"2va9BGv9JrLTtrzZttiEMDYw1Zj6a6EHzXjmP9evHDTG3oEquURA","start_checkpoint":"3NK2tkzqqK5spR2sZ7tujjqPksL45M3UUrcA4WhCkeiPtnugyE2x","lock_checkpoint":"3NK2tkzqqK5spR2sZ7tujjqPksL45M3UUrcA4WhCkeiPtnugyE2x","epoch_length":"1"},"next_epoch_data":{"ledger":{"hash":"jxX6VJ84HaafrKozFRA4qjnni4aPXqXC2H5vQLKSryNpKTXuz1R","total_currency":"165950000000001000"},"seed":"2vb6wvE996pgoJDiwbwYNCgy3UJWK5L258iTFbgt34APg1Bf8LnD","start_checkpoint":"3NK2tkzqqK5spR2sZ7tujjqPksL45M3UUrcA4WhCkeiPtnugyE2x","lock_checkpoint":"3NK8ei2Tks8aFRyUPC5bobEojq6NqYfEk1Wag7MmjQQEGg8XL4id","epoch_length":"490"},"has_ancestor_in_same_checkpoint_window":true,"block_stake_winner":"B62qk1SBjPx1Qs6Fvi2eurXtFW6Vi9KKDbD61GuC16bDPRiNk3dBxrD","block_creator":"B62qk1SBjPx1Qs6Fvi2eurXtFW6Vi9KKDbD61GuC16bDPRiNk3dBxrD","coinbase_receiver":"B62qk1SBjPx1Qs6Fvi2eurXtFW6Vi9KKDbD61GuC16bDPRiNk3dBxrD","supercharge_coinbase":true},"constants":{"k":"290","slots_per_epoch":"7140","slots_per_sub_window":"7","delta":"0","genesis_state_timestamp":"1609355670000"}}},"protocol_state_proof":"<opaque>","staged_ledger_diff":"<opaque>","delta_transition_chain_proof":"<opaque>","current_protocol_version":"1.1.0","proposed_protocol_version":"<None>"},"hash":"3NLRuVQNhsGMNCmyM2pLCcVNUxVvLDaGcqoqWwzfpodx7sc3nZLT"},"staged_ledger":"<opaque>","just_emitted_a_proof":true,"transition_receipt_time":"2021-01-05T05:55:18.376070Z"},"host":"165.227.15.94","peer_id":"12D3KooWHosL4GxDy4UPbTbKLJbYsMj5trjRuEDQdA3tmD9tVJuL","pid":8373,"port":8302},"event_id":"64e2d3e86c37c09b15efdaf7470ce879"}
{"timestamp":"2021-01-05 05:57:04.670592Z","level":"Info","source":{"module":"Block_producer","location":"File \"src/lib/block_producer/block_producer.ml\", line 602, characters 24-35"},"message":"Generated transition $state_hash was accepted into transition frontier","metadata":{"host":"165.227.15.94","peer_id":"12D3KooWHosL4GxDy4UPbTbKLJbYsMj5trjRuEDQdA3tmD9tVJuL","pid":8373,"port":8302,"state_hash":"3NLRuVQNhsGMNCmyM2pLCcVNUxVvLDaGcqoqWwzfpodx7sc3nZLT"}}
{"timestamp":"2021-01-05 05:57:04.679978Z","level":"Info","source":{"module":"Work_selector__Work_lib","location":"File \"src/lib/work_selector/work_lib.ml\", line 78, characters 22-33"},"message":"Updating new available work took $time ms","metadata":{"host":"165.227.15.94","peer_id":"12D3KooWHosL4GxDy4UPbTbKLJbYsMj5trjRuEDQdA3tmD9tVJuL","pid":8373,"port":8302,"time":9.255886077880859}}
{"timestamp":"2021-01-05 05:57:06.522698Z","level":"Info","source":{"module":"Consensus__Proof_of_stake","location":"File \"src/lib/consensus/proof_of_stake.ml\", line 3128, characters 4-15"},"message":"Determining next slot to produce block","metadata":{"host":"165.227.15.94","peer_id":"12D3KooWHosL4GxDy4UPbTbKLJbYsMj5trjRuEDQdA3tmD9tVJuL","pid":8373,"port":8302}}
{"timestamp":"2021-01-05 05:57:06.522958Z","level":"Info","source":{"module":"Consensus__Proof_of_stake","location":"File \"src/lib/consensus/proof_of_stake.ml\", line 3242, characters 16-27"},"message":"Checking VRF evaluations at epoch: $epoch, slot: $slot","metadata":{"epoch":0,"host":"165.227.15.94","peer_id":"12D3KooWHosL4GxDy4UPbTbKLJbYsMj5trjRuEDQdA3tmD9tVJuL","pid":8373,"port":8302,"slot":2614}}
...Checking VRF evaluations ...
{"timestamp":"2021-01-05 05:57:06.822120Z","level":"Info","source":{"module":"Consensus__Proof_of_stake","location":"File \"src/lib/consensus/proof_of_stake.ml\", line 3242, characters 16-27"},"message":"Checking VRF evaluations at epoch: $epoch, slot: $slot","metadata":{"epoch":0,"host":"165.227.15.94","peer_id":"12D3KooWHosL4GxDy4UPbTbKLJbYsMj5trjRuEDQdA3tmD9tVJuL","pid":8373,"port":8302,"slot":3002}}
{"timestamp":"2021-01-05 05:57:06.826181Z","level":"Info","source":{"module":"Consensus__Proof_of_stake","location":"File \"src/lib/consensus/proof_of_stake.ml\", line 3278, characters 10-21"},"message":"Producing block in 388 slots","metadata":{"host":"165.227.15.94","peer_id":"12D3KooWHosL4GxDy4UPbTbKLJbYsMj5trjRuEDQdA3tmD9tVJuL","pid":8373,"port":8302}}
{"timestamp":"2021-01-05 05:57:07.444673Z","level":"Error","source":{"module":"Mina_lib","location":"File \"src/lib/mina_lib/mina_lib.ml\", line 1278, characters 26-53"},"message":"Internally generated block $state_hash cannot be rebroadcast because it's not a valid time to do so ($timing)","metadata":{"external_transition":{"data":{"protocol_state":{"previous_state_hash":"3NK8ei2Tks8aFRyUPC5bobEojq6NqYfEk1Wag7MmjQQEGg8XL4id","body":{"genesis_state_hash":"3NLxYrjb7zmHdoFgBrubCN8ijM8v7eT8kvLiPLc9DHt3M8XrDDEG","blockchain_state":{"staged_ledger_hash":{"non_snark":{"ledger_hash":"jwf54CDsSicYARUa9JMCnN43a4cfvsHESTwvNH3HVQxivVaZCWC","aux_hash":"UvwToEGRwWmAH2qE1ZSHZhZ4VzuwmQc6pWYrzhtLdCg5hK2R12","pending_coinbase_aux":"XnsksovkTnZdKFmoTZkoetqdVE4wnqoNsRk8PTnMNWr6o8nEnD"},"pending_coinbase_hash":"2n1HdFfKv9hJFXuDhh4LfCEyWrGqm6ZHpdWKf51DcbhaviBpyVzJ"},"snarked_ledger_hash":"jxcCCAV1UUggSH2RcVLtMVy54dvxZwZ1ndpdMDjNc5sP5yownYh","genesis_ledger_hash":"jxX6VJ84HaafrKozFRA4qjnni4aPXqXC2H5vQLKSryNpKTXuz1R","snarked_next_available_token":"2","timestamp":"1609826010000"},"consensus_state":{"blockchain_length":"489","epoch_count":"0","min_window_density":"1","sub_window_densities":["0","1","1","1","1","2","0","2","3","2","2"],"last_vrf_output":"kxJaM3ECnJRC3ce4fmsFVdK-RDnNKxdw1woZFDq_AAA=","total_currency":"166055800000001000","curr_global_slot":{"slot_number":"2613","slots_per_epoch":"7140"},"global_slot_since_genesis":"2613","staking_epoch_data":{"ledger":{"hash":"jxX6VJ84HaafrKozFRA4qjnni4aPXqXC2H5vQLKSryNpKTXuz1R","total_currency":"165950000000001000"},"seed":"2va9BGv9JrLTtrzZttiEMDYw1Zj6a6EHzXjmP9evHDTG3oEquURA","start_checkpoint":"3NK2tkzqqK5spR2sZ7tujjqPksL45M3UUrcA4WhCkeiPtnugyE2x","lock_checkpoint":"3NK2tkzqqK5spR2sZ7tujjqPksL45M3UUrcA4WhCkeiPtnugyE2x","epoch_length":"1"},"next_epoch_data":{"ledger":{"hash":"jxX6VJ84HaafrKozFRA4qjnni4aPXqXC2H5vQLKSryNpKTXuz1R","total_currency":"165950000000001000"},"seed":"2vb6wvE996pgoJDiwbwYNCgy3UJWK5L258iTFbgt34APg1Bf8LnD","start_checkpoint":"3NK2tkzqqK5spR2sZ7tujjqPksL45M3UUrcA4WhCkeiPtnugyE2x","lock_checkpoint":"3NK8ei2Tks8aFRyUPC5bobEojq6NqYfEk1Wag7MmjQQEGg8XL4id","epoch_length":"490"},"has_ancestor_in_same_checkpoint_window":true,"block_stake_winner":"B62qk1SBjPx1Qs6Fvi2eurXtFW6Vi9KKDbD61GuC16bDPRiNk3dBxrD","block_creator":"B62qk1SBjPx1Qs6Fvi2eurXtFW6Vi9KKDbD61GuC16bDPRiNk3dBxrD","coinbase_receiver":"B62qk1SBjPx1Qs6Fvi2eurXtFW6Vi9KKDbD61GuC16bDPRiNk3dBxrD","supercharge_coinbase":true},"constants":{"k":"290","slots_per_epoch":"7140","slots_per_sub_window":"7","delta":"0","genesis_state_timestamp":"1609355670000"}}},"protocol_state_proof":"<opaque>","staged_ledger_diff":"<opaque>","delta_transition_chain_proof":"<opaque>","current_protocol_version":"1.1.0","proposed_protocol_version":"<None>"},"hash":"3NLRuVQNhsGMNCmyM2pLCcVNUxVvLDaGcqoqWwzfpodx7sc3nZLT"},"host":"165.227.15.94","peer_id":"12D3KooWHosL4GxDy4UPbTbKLJbYsMj5trjRuEDQdA3tmD9tVJuL","pid":8373,"port":8302,"state_hash":"3NLRuVQNhsGMNCmyM2pLCcVNUxVvLDaGcqoqWwzfpodx7sc3nZLT","timing":"1 slots too late"}}

Summary of the log:

05:53:30 started producing
05:55:18 finished proving
05:56:04 received a block
05:56:47 decide not to rebroadcast
05:57:02 successfully produced/created breadcrumb
05:57:04 added to the frontier
05:57:07 tried to broadcast but failed due to invalid time

Pausing validation would cause gossiped messaged to be dropped/timed out for the duration of the block production (at most slot duration) but should not impact the overall network traffic much since only a few nodes are would be producing blocks per slot

deepthiskumar commented 3 years ago

Also, the seeing the following operations before the produced block is broadcasted:

  1. updated available snark work
  2. evaluate VRFs for every slot until a slot is won or until the end of the epoch
  3. respond to rpc requests for transition chain, transition chain proof, sync ledger etc)

These operation could delay broadcast further

deepthiskumar commented 2 years ago

This is still applicable and should be fixed. @nholland94's validation performance fixes might have helped but we need to confirm. The issue still exists and needs to be prioritized if we see more reports of this with the updated daemon (2) from the list above can be removed as vrf eval is now async.