Open EmrePiconbello opened 3 years ago
I'm not entirely sure that I'm saying the same thing Emre is, but they may be related. Here's what I observed.
I also ran the VRF tool to compare my epoch 9. I didn't miss any blocks being generated; however when checking against Gareth's performance script which checks to see if the block was produced at the canonical height, it showed that I missed that block.
on 7/28, I had seen a chain kMRx->Bwyz->8Ngb->LsBz before I started producing a block on top of LsBz.
After producing that block (ending in YntX), it was broadcast, but then the node presumably went to catchup to fast forward to Bwyz as the tip, and then produced a second block BLYk which was rejected as too late and not sent.
(I'm assuming it went to catchup because of the double block production. I log status every 5 mins and it always showed in sync, but this could have been a quick catchup. I do see this in the catchup log message around that time ... at 21:40:48.237628Z shows 1 to build breadcrumb... "states":[[["To_build_breadcrumb"],1],[["To_initial_validate"],0],[["Finished"],622],[["To_verify"],0],[["To_download"],0],[["Wait_for_parent"],0]] )
I'm not sure if YntX was really broadcast, but nobody I asked reported seeing it. Presumably it would be dropped as part of a short fork, so that makes sense.
What doesn't make sense is that I saw Bwyz initially at 21:36, but at 21:39 was producing off of LsBz.
This all makes me suspect blocks are getting held up in the network buffer between things -- it's hard to imagine it really taking 3 mins to propagate -- I suspect it's way more likely that it was sitting around waiting to be processed.
21:39:00.001448Z - "Producing new block with parent $breadcrumb" ... **"previous_state_hash":"3NKe3F7f4ALxSvknH4DrVMSiPxvgjg7krEbmw4CtC2vTWguiLsBz"
21:40:27.013942Z - "Successfully produced a new block" ... previous_state_hash":"3NLFoaiYzY5B1XkFVfCmfSgmkkVh9n7b8bzr2FmqD1ecC8xcZofh" ..** "hash":"3NLqz46LSErA3HTJA3K9EHw2UNE6Nqc2qE2BBg1Mp792rgodYntX"
21:40:27.014515Z - "Waiting for block $state_hash to be inserted into frontier" ... "state_hash":"3NLqz46LSErA3HTJA3K9EHw2UNE6Nqc2qE2BBg1Mp792rgodYntX"
21:40:30.864098Z - "Generated transition $state_hash was accepted into transition frontier" ... "state_hash":"3NLqz46LSErA3HTJA3K9EHw2UNE6Nqc2qE2BBg1Mp792rgodYntX"
21:40:30.885492Z - "Saw block with state hash $state_hash" ... "previous_state_hash":"3NL7ukD8U4DuS8HEKFZVJmrzEBWZuZqtakaFTDqDF1sBYqZB8Ngb" ... "state_hash":"3NKydDuNc6frgKxnGRrxRgSrcBNXKKLYv4VAeJM5zVEmUS3eBwyz"
21:40:47.945952Z - "Producing block in 0 slots"
21:40:47.946374Z - "Producing new block with parent $breadcrumb" ... "previous_state_hash":"3NL7ukD8U4DuS8HEKFZVJmrzEBWZuZqtakaFTDqDF1sBYqZB8Ngb"
**21:41:00.140980Z - "Broadcasting new state over gossip net","metadata":{"host":"145.40.90.199","message":["New_state",{"protocol_state":{"previous_state_hash":"3NLFoaiYzY5B1XkFVfCmfSgmkkVh9n7b8bzr2FmqD1ecC8xcZofh**" ... "state_hash":"3NLqz46LSErA3HTJA3K9EHw2UNE6Nqc2qE2BBg1Mp792rgodYntX"...
21:41:51.521787Z - "Successfully produced a new block" ... "previous_state_hash":"3NKydDuNc6frgKxnGRrxRgSrcBNXKKLYv4VAeJM5zVEmUS3eBwyz"
21:42:03.479635Z - "Generated transition $state_hash was accepted into transition frontier" ... "state_hash":"3NKVvM9FAUFwVxEQnmhca5uxfGaTAboK7BLtJ87E1LZTWAZFBLYk"
21:42:49.437706Z - Error "Internally generated block $state_hash cannot be rebroadcast because it's not a valid time to do so ($timing)" ... "previous_state_hash":"3NKydDuNc6frgKxnGRrxRgSrcBNXKKLYv4VAeJM5zVEmUS3eBwyz" ... "state_hash":"3NKVvM9FAUFwVxEQnmhca5uxfGaTAboK7BLtJ87E1LZTWAZFBLYk","timing":"1 slots too late"
I had already seen other blocks that would be the longer (and eventually canonical chain) but produced off of the older because they were rejected initially:
{
"timestamp": "2021-07-28 21:36:01.933503Z",
"level": "Error",
"source": {
"module": "Transition_router__Initial_validator",
"location": "File \"src/lib/transition_router/initial_validator.ml\", line 69, characters 2-14"
},
"message": "Validation error: external transition with state hash $state_hash was rejected for reason $reason",
"metadata": {
"host": "145.40.90.199",
"peer_id": "12D3KooWB9YemkwqWVkRPMYr2T7htNC3A5sVBWTg32uncNTd4TaR",
"pid": 23365,
"port": 8302,
"reason": "invalid time",
"slot_diff": "1",
"state_hash": "3NKydDuNc6frgKxnGRrxRgSrcBNXKKLYv4VAeJM5zVEmUS3eBwyz",
"time_error": "too late",
"time_received": "2021-07-28 21:36:01.932000Z"
}
}
Here are the receipt times of each of the chain candidates:
21:30:00.289989Z - 3NKe3F7f4ALxSvknH4DrVMSiPxvgjg7krEbmw4CtC2vTWguiLsBz
21:31:55.297343Z - 3NLFoaiYzY5B1XkFVfCmfSgmkkVh9n7b8bzr2FmqD1ecC8xcZofh
21:32:06.611819Z - 3NL7ukD8U4DuS8HEKFZVJmrzEBWZuZqtakaFTDqDF1sBYqZB8Ngb
21:33:43.585521Z - 3NKGvFcVqNKq7pYGd2VSLoQYj3xL1t6ZEXJTuZcjE9oE9iqxWfrf
21:36:01.932111Z - 3NKydDuNc6frgKxnGRrxRgSrcBNXKKLYv4VAeJM5zVEmUS3eBwyz
21:38:43.115240Z - 3NLchfxkvkB6ei5NSW274cqRQS2DF24opGsV4W586Y2rY4BmkMRx
-- All were seen before I started producing at 21:39 on top of LsBz.
mina.log.21-22 have the details. Included .20 and .23 for additional context if needed...
https://drive.google.com/drive/folders/1z6OojQemOs_wpU8cjiCCV9bpgaN4uD-X?usp=sharing
For some more clear data, these are the block creations time stamps
{"timestamp":"2021-08-12 23:22:07.879719Z","level":"Info","source":{"module":"Block_producer","location":"File \"src/lib/block_producer/block_producer.ml\", line 682, characters 20-35"},"message":"Successfully produced a new block","metadata":{"breadcrumb":{"validated_transition":{"data":{"protocol_state":{"previous_state_hash":"3NKYxYikusTVNwN8ghFe47Yx3mekTUSEKjA3WWkjzFyUADfzNBZA","body":{"genesis_state_hash":"3NKeMoncuHab5ScarV5ViyF16cJPT4taWNSaTLS64Dp67wuXigPZ","blockchain_state":{"staged_ledger_hash":{"non_snark":{"ledger_hash":"jxZdJj3KntfuXAzKPTDL4iYrmDyco9d1372MXpAvgMYWBajspHj","aux_hash":"VETxDeabGFAAZRLQHmiWqtNZk8SzsRxickonUCmeB1Xe5MrsWj","pending_coinbase_aux":"Wb66BTQUERqbNyqudPDrKUuxeUPAUDCFDnRFcp8psdDp9J6aWj"},"pending_coinbase_hash":"2n25VYrtfdYzwFo72iGvwwpD2cXYgtdXz67UmLsAprEuLeHK8krP"},"snarked_ledger_hash":"jxNWtLPNLR6tk2KJXghPeM8uLziFY84XUFC7sGNEA5ayf7Dmhb2","genesis_ledger_hash":"jx7buQVWFLsXTtzRgSxbYcT8EYLS8KCZbLrfDcJxMtyy4thw2Ee","snarked_next_available_token":"2","timestamp":"1628810460000"},"consensus_state":{"blockchain_length":"50672","epoch_count":"10","min_window_density":"37","sub_window_densities":["5","1","6","7","3","6","3","2","5","4","4"],"last_vrf_output":"mTgIjbhwhMXuLkJcLMk5yBneHJbhIN3IyKPw4eYQAAA=","total_currency":"845888572840039233","curr_global_slot":{"slot_number":"71507","slots_per_epoch":"7140"},"global_slot_since_genesis":"71507","staking_epoch_data":{"ledger":{"hash":"jxhjiLBeMR7pgtV8ogcJvqXdr6asoNrC3g6hoUzEDLBSnZoxUDJ","total_currency":"841909132840039233"},"seed":"2vaVfCZQhLLbwojx47Xm9SozCzmyYNcSkAWYPwshqNcmJeAfrN94","start_checkpoint":"3NLGNyQrCRyasxVuv7BtmLdi8hnb3LiPaRAYiXom2L7Eb43Tyz3Y","lock_checkpoint":"3NLwFcEKfbHKEToLfTQ5h9aHgbXZ5YtV3UN1Dqf1YhgJsr9Kc1xt","epoch_length":"4867"},"next_epoch_data":{"ledger":{"hash":"jx2XUFjvsvtTKB4HPAzih5boAtuoR34kxjEoU1RUhfXTATyx8tw","total_currency":"845838172840039233"},"seed":"2vboWtfdoizrHWU62emBFSft2EJGZXPnQ6Cpk2k6cQvsGvPchzR7","start_checkpoint":"3NKf6tcVQo74pcoUwxXuWRjeSHzkdKd3UeYTpAUrKuEfQ9qu8NbD","lock_checkpoint":"3NKYxYikusTVNwN8ghFe47Yx3mekTUSEKjA3WWkjzFyUADfzNBZA","epoch_length":"66"},"has_ancestor_in_same_checkpoint_window":true,"block_stake_winner":"B62qmxzH7BDSVHjCPJEaYGriE9JsZSDRVYeZaBzAF5P3rFKQD4TNuE7","block_creator":"B62qrQiw9JhUumq457sMxicgQ94Z1WD9JChzJu19kBE8Szb5T8tcUAC","coinbase_receiver":"B62qrQiw9JhUumq457sMxicgQ94Z1WD9JChzJu19kBE8Szb5T8tcUAC","supercharge_coinbase":true},"constants":{"k":"290","slots_per_epoch":"7140","slots_per_sub_window":"7","delta":"0","genesis_state_timestamp":"1615939200000"}}},"protocol_state_proof":"<opaque>","staged_ledger_diff":"<opaque>","delta_transition_chain_proof":"<opaque>","current_protocol_version":"2.0.0","proposed_protocol_version":"<None>"},"hash":"3NKBdvCsMQtNnpXYrmCQaRgvcqMiwwQVbRQseZrN7SyVPbQNn4A3"},"staged_ledger":"<opaque>","just_emitted_a_proof":true,"transition_receipt_time":"2021-08-12T23:22:01.753215Z"},"host":"77.102.115.129","peer_id":"12D3KooWEfzjaT1YCG5igFcoWbPpePHu2ZcT3dy6XvAjjyMHKbcT","pid":16,"port":8302},"event_id":"64e2d3e86c37c09b15efdaf7470ce879"}
{"timestamp":"2021-08-12 23:22:05.815288Z","level":"Info","source":{"module":"Block_producer","location":"File \"src/lib/block_producer/block_producer.ml\", line 608, characters 20-35"},"message":"Successfully produced a new block","metadata":{"$duplicated_keys":["host","peer_id","port"],"breadcrumb":{"validated_transition":{"data":{"protocol_state":{"previous_state_hash":"3NKYxYikusTVNwN8ghFe47Yx3mekTUSEKjA3WWkjzFyUADfzNBZA","body":{"genesis_state_hash":"3NKeMoncuHab5ScarV5ViyF16cJPT4taWNSaTLS64Dp67wuXigPZ","blockchain_state":{"staged_ledger_hash":{"non_snark":{"ledger_hash":"jx7MJs9Cfpi9xG2SeDY7scNHrTwg5pZi2ihfPU51M8UVdmUX9HH","aux_hash":"Vr1hsKbSo8Nu8Gw6r8apXH9nuzyh3YUUznjkjqGLDFPUqEjgYW","pending_coinbase_aux":"Wb66BTQUERqbNyqudPDrKUuxeUPAUDCFDnRFcp8psdDp9J6aWj"},"pending_coinbase_hash":"2n25VYrtfdYzwFo72iGvwwpD2cXYgtdXz67UmLsAprEuLeHK8krP"},"snarked_ledger_hash":"jxNWtLPNLR6tk2KJXghPeM8uLziFY84XUFC7sGNEA5ayf7Dmhb2","genesis_ledger_hash":"jx7buQVWFLsXTtzRgSxbYcT8EYLS8KCZbLrfDcJxMtyy4thw2Ee","snarked_next_available_token":"2","timestamp":"1628810460000"},"consensus_state":{"blockchain_length":"50672","epoch_count":"10","min_window_density":"37","sub_window_densities":["5","1","6","7","3","6","3","2","5","4","4"],"last_vrf_output":"mTgIjbhwhMXuLkJcLMk5yBneHJbhIN3IyKPw4eYQAAA=","total_currency":"845888572840039233","curr_global_slot":{"slot_number":"71507","slots_per_epoch":"7140"},"global_slot_since_genesis":"71507","staking_epoch_data":{"ledger":{"hash":"jxhjiLBeMR7pgtV8ogcJvqXdr6asoNrC3g6hoUzEDLBSnZoxUDJ","total_currency":"841909132840039233"},"seed":"2vaVfCZQhLLbwojx47Xm9SozCzmyYNcSkAWYPwshqNcmJeAfrN94","start_checkpoint":"3NLGNyQrCRyasxVuv7BtmLdi8hnb3LiPaRAYiXom2L7Eb43Tyz3Y","lock_checkpoint":"3NLwFcEKfbHKEToLfTQ5h9aHgbXZ5YtV3UN1Dqf1YhgJsr9Kc1xt","epoch_length":"4867"},"next_epoch_data":{"ledger":{"hash":"jx2XUFjvsvtTKB4HPAzih5boAtuoR34kxjEoU1RUhfXTATyx8tw","total_currency":"845838172840039233"},"seed":"2vboWtfdoizrHWU62emBFSft2EJGZXPnQ6Cpk2k6cQvsGvPchzR7","start_checkpoint":"3NKf6tcVQo74pcoUwxXuWRjeSHzkdKd3UeYTpAUrKuEfQ9qu8NbD","lock_checkpoint":"3NKYxYikusTVNwN8ghFe47Yx3mekTUSEKjA3WWkjzFyUADfzNBZA","epoch_length":"66"},"has_ancestor_in_same_checkpoint_window":true,"block_stake_winner":"B62qmxzH7BDSVHjCPJEaYGriE9JsZSDRVYeZaBzAF5P3rFKQD4TNuE7","block_creator":"B62qrQiw9JhUumq457sMxicgQ94Z1WD9JChzJu19kBE8Szb5T8tcUAC","coinbase_receiver":"B62qrQiw9JhUumq457sMxicgQ94Z1WD9JChzJu19kBE8Szb5T8tcUAC","supercharge_coinbase":true},"constants":{"k":"290","slots_per_epoch":"7140","slots_per_sub_window":"7","delta":"0","genesis_state_timestamp":"1615939200000"}}},"protocol_state_proof":"<opaque>","staged_ledger_diff":"<opaque>","delta_transition_chain_proof":"<opaque>","current_protocol_version":"2.0.0","proposed_protocol_version":"<None>"},"hash":"3NKDrAFMbMx3s2yYK6NCaKZwTK8kxQ1VxCkxZMJs8zzYw5AvB6dr"},"staged_ledger":"<opaque>","just_emitted_a_proof":true,"transition_receipt_time":"2021-08-12T23:21:59.796456Z"},"host":"77.102.115.129","peer_id":"12D3KooWECfmci3MAajoToLUfZC5mYt5xAQgs1xErvgqntDgpKrt","pid":13,"port":9000},"event_id":"64e2d3e86c37c09b15efdaf7470ce879"}
{"timestamp":"2021-08-12 23:22:19.992915Z","level":"Info","source":{"module":"Block_producer","location":"File \"src/lib/block_producer/block_producer.ml\", line 608, characters 20-35"},"message":"Successfully produced a new block","metadata":{"breadcrumb":{"validated_transition":{"data":{"protocol_state":{"previous_state_hash":"3NKYxYikusTVNwN8ghFe47Yx3mekTUSEKjA3WWkjzFyUADfzNBZA","body":{"genesis_state_hash":"3NKeMoncuHab5ScarV5ViyF16cJPT4taWNSaTLS64Dp67wuXigPZ","blockchain_state":{"staged_ledger_hash":{"non_snark":{"ledger_hash":"jx1w3nkMnShGsL8D6T8jJsbzh3F23drpbtB3FUhw8yUnrFscBuT","aux_hash":"VjmXGrZZBU79m6ebdQxixB9LM2EDioAqf1mz3oT3Ztx9ZFVeqF","pending_coinbase_aux":"Wb66BTQUERqbNyqudPDrKUuxeUPAUDCFDnRFcp8psdDp9J6aWj"},"pending_coinbase_hash":"2n25VYrtfdYzwFo72iGvwwpD2cXYgtdXz67UmLsAprEuLeHK8krP"},"snarked_ledger_hash":"jxNWtLPNLR6tk2KJXghPeM8uLziFY84XUFC7sGNEA5ayf7Dmhb2","genesis_ledger_hash":"jx7buQVWFLsXTtzRgSxbYcT8EYLS8KCZbLrfDcJxMtyy4thw2Ee","snarked_next_available_token":"2","timestamp":"1628810460000"},"consensus_state":{"blockchain_length":"50672","epoch_count":"10","min_window_density":"37","sub_window_densities":["5","1","6","7","3","6","3","2","5","4","4"],"last_vrf_output":"mTgIjbhwhMXuLkJcLMk5yBneHJbhIN3IyKPw4eYQAAA=","total_currency":"845888572840039233","curr_global_slot":{"slot_number":"71507","slots_per_epoch":"7140"},"global_slot_since_genesis":"71507","staking_epoch_data":{"ledger":{"hash":"jxhjiLBeMR7pgtV8ogcJvqXdr6asoNrC3g6hoUzEDLBSnZoxUDJ","total_currency":"841909132840039233"},"seed":"2vaVfCZQhLLbwojx47Xm9SozCzmyYNcSkAWYPwshqNcmJeAfrN94","start_checkpoint":"3NLGNyQrCRyasxVuv7BtmLdi8hnb3LiPaRAYiXom2L7Eb43Tyz3Y","lock_checkpoint":"3NLwFcEKfbHKEToLfTQ5h9aHgbXZ5YtV3UN1Dqf1YhgJsr9Kc1xt","epoch_length":"4867"},"next_epoch_data":{"ledger":{"hash":"jx2XUFjvsvtTKB4HPAzih5boAtuoR34kxjEoU1RUhfXTATyx8tw","total_currency":"845838172840039233"},"seed":"2vboWtfdoizrHWU62emBFSft2EJGZXPnQ6Cpk2k6cQvsGvPchzR7","start_checkpoint":"3NKf6tcVQo74pcoUwxXuWRjeSHzkdKd3UeYTpAUrKuEfQ9qu8NbD","lock_checkpoint":"3NKYxYikusTVNwN8ghFe47Yx3mekTUSEKjA3WWkjzFyUADfzNBZA","epoch_length":"66"},"has_ancestor_in_same_checkpoint_window":true,"block_stake_winner":"B62qmxzH7BDSVHjCPJEaYGriE9JsZSDRVYeZaBzAF5P3rFKQD4TNuE7","block_creator":"B62qrQiw9JhUumq457sMxicgQ94Z1WD9JChzJu19kBE8Szb5T8tcUAC","coinbase_receiver":"B62qrQiw9JhUumq457sMxicgQ94Z1WD9JChzJu19kBE8Szb5T8tcUAC","supercharge_coinbase":true},"constants":{"k":"290","slots_per_epoch":"7140","slots_per_sub_window":"7","delta":"0","genesis_state_timestamp":"1615939200000"}}},"protocol_state_proof":"<opaque>","staged_ledger_diff":"<opaque>","delta_transition_chain_proof":"<opaque>","current_protocol_version":"2.0.0","proposed_protocol_version":"<None>"},"hash":"3NLWboZscRkukQsHWVudknKyLqYJ5jx6R7RdRhQGGEMsV5AoXUvE"},"staged_ledger":"<opaque>","just_emitted_a_proof":true,"transition_receipt_time":"2021-08-12T23:22:12.021175Z"},"host":"34.82.7.217","peer_id":"12D3KooWJw88DqM6r5ApTQ3Kc5CHKXLk8wkSoF1UnFKQBRSCcWMQ","pid":13,"port":8302},"event_id":"64e2d3e86c37c09b15efdaf7470ce879"}
{"timestamp":"2021-08-13 10:16:31.789925Z","level":"Info","source":{"module":"Block_producer","location":"File \"src/lib/block_producer/block_producer.ml\", line 608, characters 20-35"},"message":"Successfully produced a new block","metadata":{"breadcrumb":{"validated_transition":{"data":{"protocol_state":{"previous_state_hash":"3NLNsnorDq1iCUGwdZVk5AkWXdvBxMxw6c6QJJWz6dpzsTZquMFU","body":{"genesis_state_hash":"3NKeMoncuHab5ScarV5ViyF16cJPT4taWNSaTLS64Dp67wuXigPZ","blockchain_state":{"staged_ledger_hash":{"non_snark":{"ledger_hash":"jxJqzwM7JEdfVntTYJspmJTQi6ug4LSHcVFhkLf6HBNuwBYUVXU","aux_hash":"Vab4qE22yRfV9k2s99dHwsmmvaiNzmXpJ922qXF4r3oBoYBQ8p","pending_coinbase_aux":"WZK9b6zvFtMbjDrAxaTTfqw8wJmLHa3b3Hpy8gs37Ei52zHk3f"},"pending_coinbase_hash":"2n1z4TC3ELq9N5o1WwFYLtCGT27vnT9Pdps4vNMxk4VQDmLkVucR"},"snarked_ledger_hash":"jwXGGFVbKk19HA5ECeKUXLTik2x3WLihAjJm1vUWCxTiviFJHxV","genesis_ledger_hash":"jx7buQVWFLsXTtzRgSxbYcT8EYLS8KCZbLrfDcJxMtyy4thw2Ee","snarked_next_available_token":"2","timestamp":"1628849700000"},"consensus_state":{"blockchain_length":"50804","epoch_count":"10","min_window_density":"37","sub_window_densities":["3","5","5","3","6","3","3","3","3","5","5"],"last_vrf_output":"6yHb-YGT8tEMky3XSNMuyrLJ0gIMrtIiVPHBAH5BAAA=","total_currency":"845987932840039233","curr_global_slot":{"slot_number":"71725","slots_per_epoch":"7140"},"global_slot_since_genesis":"71725","staking_epoch_data":{"ledger":{"hash":"jxhjiLBeMR7pgtV8ogcJvqXdr6asoNrC3g6hoUzEDLBSnZoxUDJ","total_currency":"841909132840039233"},"seed":"2vaVfCZQhLLbwojx47Xm9SozCzmyYNcSkAWYPwshqNcmJeAfrN94","start_checkpoint":"3NLGNyQrCRyasxVuv7BtmLdi8hnb3LiPaRAYiXom2L7Eb43Tyz3Y","lock_checkpoint":"3NLwFcEKfbHKEToLfTQ5h9aHgbXZ5YtV3UN1Dqf1YhgJsr9Kc1xt","epoch_length":"4867"},"next_epoch_data":{"ledger":{"hash":"jx2XUFjvsvtTKB4HPAzih5boAtuoR34kxjEoU1RUhfXTATyx8tw","total_currency":"845838172840039233"},"seed":"2vau87iive38GYZv1hw3CyKEsQYbhFqu1T2hrmzpQDPWoxzwMPCG","start_checkpoint":"3NKf6tcVQo74pcoUwxXuWRjeSHzkdKd3UeYTpAUrKuEfQ9qu8NbD","lock_checkpoint":"3NLNsnorDq1iCUGwdZVk5AkWXdvBxMxw6c6QJJWz6dpzsTZquMFU","epoch_length":"198"},"has_ancestor_in_same_checkpoint_window":true,"block_stake_winner":"B62qrYRkV2BKxoH6Nwuz6hMdZUk8U5Pefo7jdRWD8STdi6UL2MMaTaj","block_creator":"B62qrQiw9JhUumq457sMxicgQ94Z1WD9JChzJu19kBE8Szb5T8tcUAC","coinbase_receiver":"B62qrQiw9JhUumq457sMxicgQ94Z1WD9JChzJu19kBE8Szb5T8tcUAC","supercharge_coinbase":false},"constants":{"k":"290","slots_per_epoch":"7140","slots_per_sub_window":"7","delta":"0","genesis_state_timestamp":"1615939200000"}}},"protocol_state_proof":"<opaque>","staged_ledger_diff":"<opaque>","delta_transition_chain_proof":"<opaque>","current_protocol_version":"2.0.0","proposed_protocol_version":"<None>"},"hash":"3NK8g1c4Z2WybxFtwTLk2PtNQeXWyqhxaBMZPgT2Fv1hhKEdFtGM"},"staged_ledger":"<opaque>","just_emitted_a_proof":false,"transition_receipt_time":"2021-08-13T10:16:26.272069Z"},"host":"34.82.7.217","peer_id":"12D3KooWJw88DqM6r5ApTQ3Kc5CHKXLk8wkSoF1UnFKQBRSCcWMQ","pid":13,"port":8302},"event_id":"64e2d3e86c37c09b15efdaf7470ce879"}
{"timestamp":"2021-08-13 10:16:01.986533Z","level":"Info","source":{"module":"Block_producer","location":"File \"src/lib/block_producer/block_producer.ml\", line 608, characters 20-35"},"message":"Successfully produced a new block","metadata":{"breadcrumb":{"validated_transition":{"data":{"protocol_state":{"previous_state_hash":"3NLNsnorDq1iCUGwdZVk5AkWXdvBxMxw6c6QJJWz6dpzsTZquMFU","body":{"genesis_state_hash":"3NKeMoncuHab5ScarV5ViyF16cJPT4taWNSaTLS64Dp67wuXigPZ","blockchain_state":{"staged_ledger_hash":{"non_snark":{"ledger_hash":"jxWhYSZAcfP9uvb1WBd1aT2zZh5YqZTUPD2cRrQHKSE4ETJfae1","aux_hash":"W78dCWyLQdF89FxZBtBEf1smC2hbRmkpW7rzyL2kigSqa9u2UU","pending_coinbase_aux":"WDgeLJqoPj59Ljyf7q6moJwNDXCV9SrBW75zJo7NnAFbTEd3k1"},"pending_coinbase_hash":"2mzosL5tv5Rm6oyU5pYFQBGsBVHTGEoD8b7FJEx2s1wqzb5yjQxS"},"snarked_ledger_hash":"jwXGGFVbKk19HA5ECeKUXLTik2x3WLihAjJm1vUWCxTiviFJHxV","genesis_ledger_hash":"jx7buQVWFLsXTtzRgSxbYcT8EYLS8KCZbLrfDcJxMtyy4thw2Ee","snarked_next_available_token":"2","timestamp":"1628849700000"},"consensus_state":{"blockchain_length":"50804","epoch_count":"10","min_window_density":"37","sub_window_densities":["3","5","5","3","6","3","3","3","3","5","5"],"last_vrf_output":"6yHb-YGT8tEMky3XSNMuyrLJ0gIMrtIiVPHBAH5BAAA=","total_currency":"845987932840039233","curr_global_slot":{"slot_number":"71725","slots_per_epoch":"7140"},"global_slot_since_genesis":"71725","staking_epoch_data":{"ledger":{"hash":"jxhjiLBeMR7pgtV8ogcJvqXdr6asoNrC3g6hoUzEDLBSnZoxUDJ","total_currency":"841909132840039233"},"seed":"2vaVfCZQhLLbwojx47Xm9SozCzmyYNcSkAWYPwshqNcmJeAfrN94","start_checkpoint":"3NLGNyQrCRyasxVuv7BtmLdi8hnb3LiPaRAYiXom2L7Eb43Tyz3Y","lock_checkpoint":"3NLwFcEKfbHKEToLfTQ5h9aHgbXZ5YtV3UN1Dqf1YhgJsr9Kc1xt","epoch_length":"4867"},"next_epoch_data":{"ledger":{"hash":"jx2XUFjvsvtTKB4HPAzih5boAtuoR34kxjEoU1RUhfXTATyx8tw","total_currency":"845838172840039233"},"seed":"2vau87iive38GYZv1hw3CyKEsQYbhFqu1T2hrmzpQDPWoxzwMPCG","start_checkpoint":"3NKf6tcVQo74pcoUwxXuWRjeSHzkdKd3UeYTpAUrKuEfQ9qu8NbD","lock_checkpoint":"3NLNsnorDq1iCUGwdZVk5AkWXdvBxMxw6c6QJJWz6dpzsTZquMFU","epoch_length":"198"},"has_ancestor_in_same_checkpoint_window":true,"block_stake_winner":"B62qrYRkV2BKxoH6Nwuz6hMdZUk8U5Pefo7jdRWD8STdi6UL2MMaTaj","block_creator":"B62qrQiw9JhUumq457sMxicgQ94Z1WD9JChzJu19kBE8Szb5T8tcUAC","coinbase_receiver":"B62qrQiw9JhUumq457sMxicgQ94Z1WD9JChzJu19kBE8Szb5T8tcUAC","supercharge_coinbase":false},"constants":{"k":"290","slots_per_epoch":"7140","slots_per_sub_window":"7","delta":"0","genesis_state_timestamp":"1615939200000"}}},"protocol_state_proof":"<opaque>","staged_ledger_diff":"<opaque>","delta_transition_chain_proof":"<opaque>","current_protocol_version":"2.0.0","proposed_protocol_version":"<None>"},"hash":"3NKwdKVomLvezWcWcbND6shJc8oX1EyUW4LsmwaQxS5PcmfNxVdZ"},"staged_ledger":"<opaque>","just_emitted_a_proof":false,"transition_receipt_time":"2021-08-13T10:15:55.853357Z"},"host":"77.102.115.129","peer_id":"12D3KooWCh6Qw2XiqdvCUTiWWnQZfFm57Wddb8evm7aRTQ9KJ9L9","pid":13,"port":9000},"event_id":"64e2d3e86c37c09b15efdaf7470ce879"}
{"timestamp":"2021-08-19 01:28:45.730076Z","level":"Info","source":{"module":"Block_producer","location":"File \"src/lib/block_producer/block_producer.ml\", line 608, characters 20-35"},"message":"Successfully produced a new block","metadata":{"breadcrumb":{"validated_transition":{"data":{"protocol_state":{"previous_state_hash":"3NL3jZxLQ98YrrV26A31F3XYZfD3G5tMujEWYMyBHKm9E2V9pMgh","body":{"genesis_state_hash":"3NKeMoncuHab5ScarV5ViyF16cJPT4taWNSaTLS64Dp67wuXigPZ","blockchain_state":{"staged_ledger_hash":{"non_snark":{"ledger_hash":"jxoYrKJ5FQtrHVdTFJHsX6ZQ89yzwUEhJmEtRciC652F2kPPpxx","aux_hash":"VFTvpVdxPFzdCPYkzCz3APiwB57GLpe4b3x2H1djqHn7JnMavc","pending_coinbase_aux":"XUiL44dnuq6YcZo8JdUkV9GueeFfCsrhJ6B28Cjv5xwf68Q2vN"},"pending_coinbase_hash":"2mzXXi1NFLvfeH2QhybX7ovx9HpJ9wyqRsVBkjQ41WQbeHsHq9sL"},"snarked_ledger_hash":"jwEofwVBBjxzwXsUoJCQ8w6F9N4NcruwtP1rHidmx6DsQxWctT3","genesis_ledger_hash":"jx7buQVWFLsXTtzRgSxbYcT8EYLS8KCZbLrfDcJxMtyy4thw2Ee","snarked_next_available_token":"2","timestamp":"1629336420000"},"consensus_state":{"blockchain_length":"52651","epoch_count":"10","min_window_density":"35","sub_window_densities":["4","4","6","3","4","7","5","5","6","5","5"],"last_vrf_output":"BR3ALLy66p7ZKa3U0JmOQPmCd9xIM4_EaGnJKIcCAAA=","total_currency":"847412092840039233","curr_global_slot":{"slot_number":"74429","slots_per_epoch":"7140"},"global_slot_since_genesis":"74429","staking_epoch_data":{"ledger":{"hash":"jxhjiLBeMR7pgtV8ogcJvqXdr6asoNrC3g6hoUzEDLBSnZoxUDJ","total_currency":"841909132840039233"},"seed":"2vaVfCZQhLLbwojx47Xm9SozCzmyYNcSkAWYPwshqNcmJeAfrN94","start_checkpoint":"3NLGNyQrCRyasxVuv7BtmLdi8hnb3LiPaRAYiXom2L7Eb43Tyz3Y","lock_checkpoint":"3NLwFcEKfbHKEToLfTQ5h9aHgbXZ5YtV3UN1Dqf1YhgJsr9Kc1xt","epoch_length":"4867"},"next_epoch_data":{"ledger":{"hash":"jx2XUFjvsvtTKB4HPAzih5boAtuoR34kxjEoU1RUhfXTATyx8tw","total_currency":"845838172840039233"},"seed":"2vaKnB4FxFNDn4ksRKfhKJu1w1NVwx9dApKzDBc9D6he5vhL6PWd","start_checkpoint":"3NKf6tcVQo74pcoUwxXuWRjeSHzkdKd3UeYTpAUrKuEfQ9qu8NbD","lock_checkpoint":"3NL3jZxLQ98YrrV26A31F3XYZfD3G5tMujEWYMyBHKm9E2V9pMgh","epoch_length":"2045"},"has_ancestor_in_same_checkpoint_window":true,"block_stake_winner":"B62qm4XxcqM7AZddwaLNLLHqgxM2uAUwA3gmH4t4wQ7o7gsCutoWtbp","block_creator":"B62qrQiw9JhUumq457sMxicgQ94Z1WD9JChzJu19kBE8Szb5T8tcUAC","coinbase_receiver":"B62qrQiw9JhUumq457sMxicgQ94Z1WD9JChzJu19kBE8Szb5T8tcUAC","supercharge_coinbase":true},"constants":{"k":"290","slots_per_epoch":"7140","slots_per_sub_window":"7","delta":"0","genesis_state_timestamp":"1615939200000"}}},"protocol_state_proof":"<opaque>","staged_ledger_diff":"<opaque>","delta_transition_chain_proof":"<opaque>","current_protocol_version":"2.0.0","proposed_protocol_version":"<None>"},"hash":"3NKExeN39FqcsVAbGRq5LoGMweFvbNLTEMr4xH1jqqwCD575RCBR"},"staged_ledger":"<opaque>","just_emitted_a_proof":true,"transition_receipt_time":"2021-08-19T01:28:37.498915Z"},"host":"34.82.7.217","peer_id":"12D3KooWJTYfKgnHMarvyMhvqjXfmCnPJSfiqKi7sp3yk4FBagBg","pid":13,"port":8302},"event_id":"64e2d3e86c37c09b15efdaf7470ce879"}
{"timestamp":"2021-08-19 01:28:31.009434Z","level":"Info","source":{"module":"Block_producer","location":"File \"src/lib/block_producer/block_producer.ml\", line 682, characters 20-35"},"message":"Successfully produced a new block","metadata":{"breadcrumb":{"validated_transition":{"data":{"protocol_state":{"previous_state_hash":"3NL3jZxLQ98YrrV26A31F3XYZfD3G5tMujEWYMyBHKm9E2V9pMgh","body":{"genesis_state_hash":"3NKeMoncuHab5ScarV5ViyF16cJPT4taWNSaTLS64Dp67wuXigPZ","blockchain_state":{"staged_ledger_hash":{"non_snark":{"ledger_hash":"jxoYrKJ5FQtrHVdTFJHsX6ZQ89yzwUEhJmEtRciC652F2kPPpxx","aux_hash":"VFTvpVdxPFzdCPYkzCz3APiwB57GLpe4b3x2H1djqHn7JnMavc","pending_coinbase_aux":"XUiL44dnuq6YcZo8JdUkV9GueeFfCsrhJ6B28Cjv5xwf68Q2vN"},"pending_coinbase_hash":"2mzXXi1NFLvfeH2QhybX7ovx9HpJ9wyqRsVBkjQ41WQbeHsHq9sL"},"snarked_ledger_hash":"jwEofwVBBjxzwXsUoJCQ8w6F9N4NcruwtP1rHidmx6DsQxWctT3","genesis_ledger_hash":"jx7buQVWFLsXTtzRgSxbYcT8EYLS8KCZbLrfDcJxMtyy4thw2Ee","snarked_next_available_token":"2","timestamp":"1629336420000"},"consensus_state":{"blockchain_length":"52651","epoch_count":"10","min_window_density":"35","sub_window_densities":["4","4","6","3","4","7","5","5","6","5","5"],"last_vrf_output":"BR3ALLy66p7ZKa3U0JmOQPmCd9xIM4_EaGnJKIcCAAA=","total_currency":"847412092840039233","curr_global_slot":{"slot_number":"74429","slots_per_epoch":"7140"},"global_slot_since_genesis":"74429","staking_epoch_data":{"ledger":{"hash":"jxhjiLBeMR7pgtV8ogcJvqXdr6asoNrC3g6hoUzEDLBSnZoxUDJ","total_currency":"841909132840039233"},"seed":"2vaVfCZQhLLbwojx47Xm9SozCzmyYNcSkAWYPwshqNcmJeAfrN94","start_checkpoint":"3NLGNyQrCRyasxVuv7BtmLdi8hnb3LiPaRAYiXom2L7Eb43Tyz3Y","lock_checkpoint":"3NLwFcEKfbHKEToLfTQ5h9aHgbXZ5YtV3UN1Dqf1YhgJsr9Kc1xt","epoch_length":"4867"},"next_epoch_data":{"ledger":{"hash":"jx2XUFjvsvtTKB4HPAzih5boAtuoR34kxjEoU1RUhfXTATyx8tw","total_currency":"845838172840039233"},"seed":"2vaKnB4FxFNDn4ksRKfhKJu1w1NVwx9dApKzDBc9D6he5vhL6PWd","start_checkpoint":"3NKf6tcVQo74pcoUwxXuWRjeSHzkdKd3UeYTpAUrKuEfQ9qu8NbD","lock_checkpoint":"3NL3jZxLQ98YrrV26A31F3XYZfD3G5tMujEWYMyBHKm9E2V9pMgh","epoch_length":"2045"},"has_ancestor_in_same_checkpoint_window":true,"block_stake_winner":"B62qm4XxcqM7AZddwaLNLLHqgxM2uAUwA3gmH4t4wQ7o7gsCutoWtbp","block_creator":"B62qrQiw9JhUumq457sMxicgQ94Z1WD9JChzJu19kBE8Szb5T8tcUAC","coinbase_receiver":"B62qrQiw9JhUumq457sMxicgQ94Z1WD9JChzJu19kBE8Szb5T8tcUAC","supercharge_coinbase":true},"constants":{"k":"290","slots_per_epoch":"7140","slots_per_sub_window":"7","delta":"0","genesis_state_timestamp":"1615939200000"}}},"protocol_state_proof":"<opaque>","staged_ledger_diff":"<opaque>","delta_transition_chain_proof":"<opaque>","current_protocol_version":"2.0.0","proposed_protocol_version":"<None>"},"hash":"3NKExeN39FqcsVAbGRq5LoGMweFvbNLTEMr4xH1jqqwCD575RCBR"},"staged_ledger":"<opaque>","just_emitted_a_proof":true,"transition_receipt_time":"2021-08-19T01:28:21.534988Z"},"host":"77.102.115.129","peer_id":"12D3KooWMcbAgBArsf447AqPEhnvtzLzTAbt6z2xD7vAjAcHMUwe","pid":16,"port":8302},"event_id":"64e2d3e86c37c09b15efdaf7470ce879"}
{"timestamp":"2021-08-19 01:28:23.252844Z","level":"Info","source":{"module":"Block_producer","location":"File \"src/lib/block_producer/block_producer.ml\", line 608, characters 20-35"},"message":"Successfully produced a new block","metadata":{"breadcrumb":{"validated_transition":{"data":{"protocol_state":{"previous_state_hash":"3NL3jZxLQ98YrrV26A31F3XYZfD3G5tMujEWYMyBHKm9E2V9pMgh","body":{"genesis_state_hash":"3NKeMoncuHab5ScarV5ViyF16cJPT4taWNSaTLS64Dp67wuXigPZ","blockchain_state":{"staged_ledger_hash":{"non_snark":{"ledger_hash":"jxoYrKJ5FQtrHVdTFJHsX6ZQ89yzwUEhJmEtRciC652F2kPPpxx","aux_hash":"VFTvpVdxPFzdCPYkzCz3APiwB57GLpe4b3x2H1djqHn7JnMavc","pending_coinbase_aux":"XUiL44dnuq6YcZo8JdUkV9GueeFfCsrhJ6B28Cjv5xwf68Q2vN"},"pending_coinbase_hash":"2mzXXi1NFLvfeH2QhybX7ovx9HpJ9wyqRsVBkjQ41WQbeHsHq9sL"},"snarked_ledger_hash":"jwEofwVBBjxzwXsUoJCQ8w6F9N4NcruwtP1rHidmx6DsQxWctT3","genesis_ledger_hash":"jx7buQVWFLsXTtzRgSxbYcT8EYLS8KCZbLrfDcJxMtyy4thw2Ee","snarked_next_available_token":"2","timestamp":"1629336420000"},"consensus_state":{"blockchain_length":"52651","epoch_count":"10","min_window_density":"35","sub_window_densities":["4","4","6","3","4","7","5","5","6","5","5"],"last_vrf_output":"BR3ALLy66p7ZKa3U0JmOQPmCd9xIM4_EaGnJKIcCAAA=","total_currency":"847412092840039233","curr_global_slot":{"slot_number":"74429","slots_per_epoch":"7140"},"global_slot_since_genesis":"74429","staking_epoch_data":{"ledger":{"hash":"jxhjiLBeMR7pgtV8ogcJvqXdr6asoNrC3g6hoUzEDLBSnZoxUDJ","total_currency":"841909132840039233"},"seed":"2vaVfCZQhLLbwojx47Xm9SozCzmyYNcSkAWYPwshqNcmJeAfrN94","start_checkpoint":"3NLGNyQrCRyasxVuv7BtmLdi8hnb3LiPaRAYiXom2L7Eb43Tyz3Y","lock_checkpoint":"3NLwFcEKfbHKEToLfTQ5h9aHgbXZ5YtV3UN1Dqf1YhgJsr9Kc1xt","epoch_length":"4867"},"next_epoch_data":{"ledger":{"hash":"jx2XUFjvsvtTKB4HPAzih5boAtuoR34kxjEoU1RUhfXTATyx8tw","total_currency":"845838172840039233"},"seed":"2vaKnB4FxFNDn4ksRKfhKJu1w1NVwx9dApKzDBc9D6he5vhL6PWd","start_checkpoint":"3NKf6tcVQo74pcoUwxXuWRjeSHzkdKd3UeYTpAUrKuEfQ9qu8NbD","lock_checkpoint":"3NL3jZxLQ98YrrV26A31F3XYZfD3G5tMujEWYMyBHKm9E2V9pMgh","epoch_length":"2045"},"has_ancestor_in_same_checkpoint_window":true,"block_stake_winner":"B62qm4XxcqM7AZddwaLNLLHqgxM2uAUwA3gmH4t4wQ7o7gsCutoWtbp","block_creator":"B62qrQiw9JhUumq457sMxicgQ94Z1WD9JChzJu19kBE8Szb5T8tcUAC","coinbase_receiver":"B62qrQiw9JhUumq457sMxicgQ94Z1WD9JChzJu19kBE8Szb5T8tcUAC","supercharge_coinbase":true},"constants":{"k":"290","slots_per_epoch":"7140","slots_per_sub_window":"7","delta":"0","genesis_state_timestamp":"1615939200000"}}},"protocol_state_proof":"<opaque>","staged_ledger_diff":"<opaque>","delta_transition_chain_proof":"<opaque>","current_protocol_version":"2.0.0","proposed_protocol_version":"<None>"},"hash":"3NKExeN39FqcsVAbGRq5LoGMweFvbNLTEMr4xH1jqqwCD575RCBR"},"staged_ledger":"<opaque>","just_emitted_a_proof":true,"transition_receipt_time":"2021-08-19T01:28:14.368913Z"},"host":"77.102.115.129","peer_id":"12D3KooWDdzTydUVzkKHAMM1qamWV3b8gQi6ypppHM34ipy8F5Rh","pid":13,"port":9000},"event_id":"64e2d3e86c37c09b15efdaf7470ce879"}
{"timestamp":"2021-08-20 04:54:55.568387Z","level":"Info","source":{"module":"Block_producer","location":"File \"src/lib/block_producer/block_producer.ml\", line 608, characters 20-35"},"message":"Successfully produced a new block","metadata":{"breadcrumb":{"validated_transition":{"data":{"protocol_state":{"previous_state_hash":"3NKwjninngXvhU5YCoDGv29XDVkh4EbLcSKq2QwM1nfYz9a1VymY","body":{"genesis_state_hash":"3NKeMoncuHab5ScarV5ViyF16cJPT4taWNSaTLS64Dp67wuXigPZ","blockchain_state":{"staged_ledger_hash":{"non_snark":{"ledger_hash":"jxQT96tS67EQUsnXeyC6jM6tMgZu8AbUwfk2x7w1tshX2D78WaM","aux_hash":"VpVp745apSsuDYtiVFnpY4ZsX8oRi5LeV6MiBRQmaiYav8jZVt","pending_coinbase_aux":"XA9MrmeWgZhry38fi2Vnui4HxagN39p1RzgeFydq1hmhcuLydd"},"pending_coinbase_hash":"2n1xdtK5bWURvBQPRH46gUkUXpCJDVn5XhQ4snZ1Y393ojZaxH1B"},"snarked_ledger_hash":"jwHiKrNpgymfKx3SMZKMTXARDzerw6t4Nr3mn7qSetMaWxYQB7j","genesis_ledger_hash":"jx7buQVWFLsXTtzRgSxbYcT8EYLS8KCZbLrfDcJxMtyy4thw2Ee","snarked_next_available_token":"2","timestamp":"1629435240000"},"consensus_state":{"blockchain_length":"53026","epoch_count":"10","min_window_density":"35","sub_window_densities":["4","6","6","6","5","7","4","4","1","6","6"],"last_vrf_output":"2XogmBSsLXIeIo4MZTptZn63Bgyl6TbMuThGPdrwAAA=","total_currency":"847687852840039233","curr_global_slot":{"slot_number":"74978","slots_per_epoch":"7140"},"global_slot_since_genesis":"74978","staking_epoch_data":{"ledger":{"hash":"jxhjiLBeMR7pgtV8ogcJvqXdr6asoNrC3g6hoUzEDLBSnZoxUDJ","total_currency":"841909132840039233"},"seed":"2vaVfCZQhLLbwojx47Xm9SozCzmyYNcSkAWYPwshqNcmJeAfrN94","start_checkpoint":"3NLGNyQrCRyasxVuv7BtmLdi8hnb3LiPaRAYiXom2L7Eb43Tyz3Y","lock_checkpoint":"3NLwFcEKfbHKEToLfTQ5h9aHgbXZ5YtV3UN1Dqf1YhgJsr9Kc1xt","epoch_length":"4867"},"next_epoch_data":{"ledger":{"hash":"jx2XUFjvsvtTKB4HPAzih5boAtuoR34kxjEoU1RUhfXTATyx8tw","total_currency":"845838172840039233"},"seed":"2vawg9VyP1iHTt496SqjjJvuwhAs2Tr39NEB6QkLrHu6qvnyzBKy","start_checkpoint":"3NKf6tcVQo74pcoUwxXuWRjeSHzkdKd3UeYTpAUrKuEfQ9qu8NbD","lock_checkpoint":"3NKwjninngXvhU5YCoDGv29XDVkh4EbLcSKq2QwM1nfYz9a1VymY","epoch_length":"2420"},"has_ancestor_in_same_checkpoint_window":true,"block_stake_winner":"B62qpMo7Vgs1pHzsVfEBhBDyQEXG1nTDptoQFXEP37L7cvpZuMgjtxS","block_creator":"B62qrQiw9JhUumq457sMxicgQ94Z1WD9JChzJu19kBE8Szb5T8tcUAC","coinbase_receiver":"B62qrQiw9JhUumq457sMxicgQ94Z1WD9JChzJu19kBE8Szb5T8tcUAC","supercharge_coinbase":true},"constants":{"k":"290","slots_per_epoch":"7140","slots_per_sub_window":"7","delta":"0","genesis_state_timestamp":"1615939200000"}}},"protocol_state_proof":"<opaque>","staged_ledger_diff":"<opaque>","delta_transition_chain_proof":"<opaque>","current_protocol_version":"2.0.0","proposed_protocol_version":"<None>"},"hash":"3NKwYQDhGEuRXrLHC6dZ96xPSF5RmxoiLyoT5n47JmnDJwJGgHt2"},"staged_ledger":"<opaque>","just_emitted_a_proof":false,"transition_receipt_time":"2021-08-20T04:54:52.775624Z"},"host":"34.82.7.217","peer_id":"12D3KooWMZcpYho2VtWT5D57AGxsnxitkSstGAfEzoY1TW6eaCbV","pid":13,"port":8302},"event_id":"64e2d3e86c37c09b15efdaf7470ce879"}
This is the only block that takes over 1 min 45-second reaching 2 min 47 second
{"timestamp":"2021-08-20 04:56:47.142673Z","level":"Info","source":{"module":"Block_producer","location":"File \"src/lib/block_producer/block_producer.ml\", line 682, characters 20-35"},"message":"Successfully produced a new block","metadata":{"breadcrumb":{"validated_transition":{"data":{"protocol_state":{"previous_state_hash":"3NKNiJLXRSaJEXG6G2fZEx64Mie7TBzbvg1A5dwpuvjUAjAa8HP1","body":{"genesis_state_hash":"3NKeMoncuHab5ScarV5ViyF16cJPT4taWNSaTLS64Dp67wuXigPZ","blockchain_state":{"staged_ledger_hash":{"non_snark":{"ledger_hash":"jwptTGoNmx4ACtbbqA921EuxQL683jFw1tsWQm3ukM4ye2zHAhf","aux_hash":"Ue7HnbhfURx7aBdHbNBcJboGXfMqs55mJPsrERRs229hvcXcve","pending_coinbase_aux":"XA9MrmeWgZhry38fi2Vnui4HxagN39p1RzgeFydq1hmhcuLydd"},"pending_coinbase_hash":"2n1vZ2TBVPk2PL8RBssFguH2ecPunRXq4YvexeVyPqvUgF6rmcTc"},"snarked_ledger_hash":"jwHiKrNpgymfKx3SMZKMTXARDzerw6t4Nr3mn7qSetMaWxYQB7j","genesis_ledger_hash":"jx7buQVWFLsXTtzRgSxbYcT8EYLS8KCZbLrfDcJxMtyy4thw2Ee","snarked_next_available_token":"2","timestamp":"1629435240000"},"consensus_state":{"blockchain_length":"53027","epoch_count":"10","min_window_density":"35","sub_window_densities":["4","6","6","6","5","7","4","4","2","6","6"],"last_vrf_output":"2XogmBSsLXIeIo4MZTptZn63Bgyl6TbMuThGPdrwAAA=","total_currency":"847687852840039233","curr_global_slot":{"slot_number":"74978","slots_per_epoch":"7140"},"global_slot_since_genesis":"74978","staking_epoch_data":{"ledger":{"hash":"jxhjiLBeMR7pgtV8ogcJvqXdr6asoNrC3g6hoUzEDLBSnZoxUDJ","total_currency":"841909132840039233"},"seed":"2vaVfCZQhLLbwojx47Xm9SozCzmyYNcSkAWYPwshqNcmJeAfrN94","start_checkpoint":"3NLGNyQrCRyasxVuv7BtmLdi8hnb3LiPaRAYiXom2L7Eb43Tyz3Y","lock_checkpoint":"3NLwFcEKfbHKEToLfTQ5h9aHgbXZ5YtV3UN1Dqf1YhgJsr9Kc1xt","epoch_length":"4867"},"next_epoch_data":{"ledger":{"hash":"jx2XUFjvsvtTKB4HPAzih5boAtuoR34kxjEoU1RUhfXTATyx8tw","total_currency":"845838172840039233"},"seed":"2vbBtTkkDX8zthsxarAiMRocnVQPZhdjbZ24za4AV5E6VfL5TA59","start_checkpoint":"3NKf6tcVQo74pcoUwxXuWRjeSHzkdKd3UeYTpAUrKuEfQ9qu8NbD","lock_checkpoint":"3NKNiJLXRSaJEXG6G2fZEx64Mie7TBzbvg1A5dwpuvjUAjAa8HP1","epoch_length":"2421"},"has_ancestor_in_same_checkpoint_window":true,"block_stake_winner":"B62qpMo7Vgs1pHzsVfEBhBDyQEXG1nTDptoQFXEP37L7cvpZuMgjtxS","block_creator":"B62qrQiw9JhUumq457sMxicgQ94Z1WD9JChzJu19kBE8Szb5T8tcUAC","coinbase_receiver":"B62qrQiw9JhUumq457sMxicgQ94Z1WD9JChzJu19kBE8Szb5T8tcUAC","supercharge_coinbase":true},"constants":{"k":"290","slots_per_epoch":"7140","slots_per_sub_window":"7","delta":"0","genesis_state_timestamp":"1615939200000"}}},"protocol_state_proof":"<opaque>","staged_ledger_diff":"<opaque>","delta_transition_chain_proof":"<opaque>","current_protocol_version":"2.0.0","proposed_protocol_version":"<None>"},"hash":"3NLx7sLgp8cw4QopmzbdQWxf3fqAwLSQjD9w8aKaRt97j81fmXij"},"staged_ledger":"<opaque>","just_emitted_a_proof":false,"transition_receipt_time":"2021-08-20T04:56:39.986536Z"},"host":"77.102.115.129","peer_id":"12D3KooWMcbAgBArsf447AqPEhnvtzLzTAbt6z2xD7vAjAcHMUwe","pid":16,"port":8302},"event_id":"64e2d3e86c37c09b15efdaf7470ce879"}
There is also this one that built on the locally generated blocks since we had 71723 and 71725 and nothing appear on block 71724 all blocks build on blocks our nodes created. The difference here is this one was a beta version and created a block with a different hash and build on the local block which should be canonical. This is the block https://minaexplorer.com/block/3NLNsnorDq1iCUGwdZVk5AkWXdvBxMxw6c6QJJWz6dpzsTZquMFU the weird part is we have all our 3 nodes creating valid canonical blocks so all blocks build on blocks created by these nodes and 6 minutes later again blocks created but this time other nodes don't see blocks from our nodes in time.
{"timestamp":"2021-08-13 10:15:58.005847Z","level":"Info","source":{"module":"Block_producer","location":"File \"src/lib/block_producer/block_producer.ml\", line 682, characters 20-35"},"message":"Successfully produced a new block","metadata":{"breadcrumb":{"validated_transition":{"data":{"protocol_state":{"previous_state_hash":"3NLa9UXVXUVnGqaUQs3vkNFiK1FVLpaEYjiDGA9TasVVp1g6ypZ3","body":{"genesis_state_hash":"3NKeMoncuHab5ScarV5ViyF16cJPT4taWNSaTLS64Dp67wuXigPZ","blockchain_state":{"staged_ledger_hash":{"non_snark":{"ledger_hash":"jxuEszo75WsgJqNtZ1K7WgSPRC4cvV7cvJpSDVqNWHsr1FbK7DS","aux_hash":"VjtbfJobifKPxd8k1T6auUB3q7EAtq5Rt1ZquuNjj9sJxMpApP","pending_coinbase_aux":"XZBcNAKB9BGynKkBSezHjKPXZoQqDbBQo6wCYKDDMGuhYVjuSo"},"pending_coinbase_hash":"2mzx7UTmnocbudQw9mYqkCqKznLTdcdNX5ntEAohMnWVk496gJre"},"snarked_ledger_hash":"jx3KZjjvCGfbo4dofy3b9MHzH46Y4SjSrtorzDbnJcZasibFZvb","genesis_ledger_hash":"jx7buQVWFLsXTtzRgSxbYcT8EYLS8KCZbLrfDcJxMtyy4thw2Ee","snarked_next_available_token":"2","timestamp":"1628849700000"},"consensus_state":{"blockchain_length":"50803","epoch_count":"10","min_window_density":"37","sub_window_densities":["3","5","5","3","6","2","3","3","3","5","5"],"last_vrf_output":"6yHb-YGT8tEMky3XSNMuyrLJ0gIMrtIiVPHBAH5BAAA=","total_currency":"845985772840039233","curr_global_slot":{"slot_number":"71725","slots_per_epoch":"7140"},"global_slot_since_genesis":"71725","staking_epoch_data":{"ledger":{"hash":"jxhjiLBeMR7pgtV8ogcJvqXdr6asoNrC3g6hoUzEDLBSnZoxUDJ","total_currency":"841909132840039233"},"seed":"2vaVfCZQhLLbwojx47Xm9SozCzmyYNcSkAWYPwshqNcmJeAfrN94","start_checkpoint":"3NLGNyQrCRyasxVuv7BtmLdi8hnb3LiPaRAYiXom2L7Eb43Tyz3Y","lock_checkpoint":"3NLwFcEKfbHKEToLfTQ5h9aHgbXZ5YtV3UN1Dqf1YhgJsr9Kc1xt","epoch_length":"4867"},"next_epoch_data":{"ledger":{"hash":"jx2XUFjvsvtTKB4HPAzih5boAtuoR34kxjEoU1RUhfXTATyx8tw","total_currency":"845838172840039233"},"seed":"2vbt2d6umLPM37WwEVfc6mD7Urb83KfdCWKdBt5Qs6x4gWFfBsvS","start_checkpoint":"3NKf6tcVQo74pcoUwxXuWRjeSHzkdKd3UeYTpAUrKuEfQ9qu8NbD","lock_checkpoint":"3NLa9UXVXUVnGqaUQs3vkNFiK1FVLpaEYjiDGA9TasVVp1g6ypZ3","epoch_length":"197"},"has_ancestor_in_same_checkpoint_window":true,"block_stake_winner":"B62qrYRkV2BKxoH6Nwuz6hMdZUk8U5Pefo7jdRWD8STdi6UL2MMaTaj","block_creator":"B62qrQiw9JhUumq457sMxicgQ94Z1WD9JChzJu19kBE8Szb5T8tcUAC","coinbase_receiver":"B62qrQiw9JhUumq457sMxicgQ94Z1WD9JChzJu19kBE8Szb5T8tcUAC","supercharge_coinbase":false},"constants":{"k":"290","slots_per_epoch":"7140","slots_per_sub_window":"7","delta":"0","genesis_state_timestamp":"1615939200000"}}},"protocol_state_proof":"<opaque>","staged_ledger_diff":"<opaque>","delta_transition_chain_proof":"<opaque>","current_protocol_version":"2.0.0","proposed_protocol_version":"<None>"},"hash":"3NKvoNbd5YzG5cJjCYo3u66fJ77hkK2rw57i2NNJniBzicF6YH1Y"},"staged_ledger":"<opaque>","just_emitted_a_proof":false,"transition_receipt_time":"2021-08-13T10:15:54.349530Z"},"host":"77.102.115.129","peer_id":"12D3KooWEfzjaT1YCG5igFcoWbPpePHu2ZcT3dy6XvAjjyMHKbcT","pid":16,"port":8302},"event_id":"64e2d3e86c37c09b15efdaf7470ce879"}```
Thanks for those logs. We would take a look and see what happened there.
I have similar issue. My two nodes have been created block in 3 mins but block missing and not even orphan.
Node1 / v1.1.8 2021-09-27 23:24:00 UTC [Info] Producing new block with parent $breadcrumb breadcrumb: { "validated_transition": { "data": { "protocol_state": { "previous_state_hash": "3NLAQvszaaVLqBAYU7Pe8scqbo5NLXUqcCG1zggvrUMjdq2uNfCw", "body": { ...... 2021-09-27 23:24:18 UTC [Info] Received a block from $sender sender: { "Remote": { "host": "159.65.200.94", "peer_id": "12D3KooWLLqypAX3QyYLjPt8S3Ajg5qDZzLWbcKBRcWm4amfPqrv", "libp2p_port": 8302 } } 2021-09-27 23:24:18 UTC [Error] Validation error: external transition with state hash $state_hash was rejected for reason "invalid time" state_hash: "3NKujCLhkNdB6fu9n3s9bww5ZqPSAAiS2i82vemR8v1C3t5wQpz6" 2021-09-27 23:24:49 UTC [Info] verifier successfully got sigkill 2021-09-27 23:24:49 UTC [Info] verifier successfully stopped 2021-09-27 23:24:49 UTC [Info] Daemon started process of kind "Verifier" with pid 6644 2021-09-27 23:25:00 UTC [Info] Protocol_state_proof proving time(ms): 51165 2021-09-27 23:25:08 UTC [Info] Build breadcrumb on produced block: 8537 2021-09-27 23:25:08 UTC [Info] Successfully produced a new block 2021-09-27 23:25:10 UTC [Info] Generated transition $state_hash was accepted into transition frontier state_hash: "3NLydjuRhqFNNfF6HX6r1rGoZQTB7PgQR1UEJ5SKVVfV76QoH3yh" 2021-09-27 23:25:10 UTC [Info] Updating new available work took 14.775514602661133 ms 2021-09-27 23:25:39 UTC [Info] Determining next slot to produce block 2021-09-27 23:25:39 UTC [Info] Checking VRF evaluations at epoch: 13, slot: 769 2021-09-27 23:25:39 UTC [Info] Checking VRF evaluations at epoch: 13, slot: 770 2021-09-27 23:25:39 UTC [Info] Checking VRF evaluations at epoch: 13, slot: 771 2021-09-27 23:25:39 UTC [Info] Checking VRF evaluations at epoch: 13, slot: 772
Node2 Beta 2021-09-27 23:24:00 UTC [Info] Producing new block with parent $breadcrumb breadcrumb: { "validated_transition": { "data": { "protocol_state": { "previous_state_hash": "3NLAQvszaaVLqBAYU7Pe8scqbo5NLXUqcCG1zggvrUMjdq2uNfCw", "body": { .... 2021-09-27 23:25:12 UTC [Info] Processing gossip message with IPC delay of 430827 nanoseconds 2021-09-27 23:25:13 UTC [Info] Protocol_state_proof proving time(ms): 62719 2021-09-27 23:25:14 UTC [Info] Processing gossip message with IPC delay of 793743461 nanoseconds 2021-09-27 23:25:15 UTC [Info] Processing gossip message with IPC delay of 1163633456 nanoseconds 2021-09-27 23:25:17 UTC [Info] Processing gossip message with IPC delay of 1166943271 nanoseconds 2021-09-27 23:25:19 UTC [Info] Processing gossip message with IPC delay of 1446506578 nanoseconds 2021-09-27 23:25:19 UTC [Info] Processing gossip message with IPC delay of 145958143 nanoseconds 2021-09-27 23:25:23 UTC [Info] Build breadcrumb on produced block: 9922 2021-09-27 23:25:23 UTC [Info] Successfully produced a new block 2021-09-27 23:25:26 UTC [Info] Generated transition $state_hash was accepted into transition frontier state_hash: "3NK3L5PrbbfaHRGbHBsoEuk9q7z6n7t1s7TYqCD3uGV7j2zUy4f9" 2021-09-27 23:25:26 UTC [Info] Updating new available work took 19.806861877441406 ms 2021-09-27 23:25:31 UTC [Info] Determining next slot to produce block 2021-09-27 23:25:31 UTC [Info] Checking VRF evaluations at epoch: 13, slot: 769 2021-09-27 23:25:31 UTC [Info] Checking VRF evaluations at epoch: 13, slot: 770 2021-09-27 23:25:31 UTC [Info] Checking VRF evaluations at epoch: 13, slot: 771
Same here with B62qrcQDeAfg2qSnZhdF8wQ78G9QmKNDqpESVt2SL43F97kF3Cyv32a I was supposed to produce blocks at 15/944 and 15/964 ... but nothing... they are even not in the orphaned blocks.
Oct 28 01:26:39 ip-172-31-80-92 mina[5038]: 2021-10-28 01:26:39 UTC [Info] Checking VRF evaluations at epoch: 15, slot: 944
Oct 28 01:27:48 ip-172-31-80-92 mina[5038]: 2021-10-28 01:27:48 UTC [Info] Producing block in 16 slots
Oct 28 02:13:28 ip-172-31-80-92 mina[5038]: 2021-10-28 02:13:28 UTC [Info] Checking VRF evaluations at epoch: 15, slot: 964
Oct 28 02:14:40 ip-172-31-80-92 mina[5038]: 2021-10-28 02:14:40 UTC [Info] Producing block in 19 slots
Version is 1.2 stable Log export in https://s3.amazonaws.com/mina.crowbar.systems/mainnet/export/2021-10-28_07-18-20.tar.gz
@hsq125 I had a scan of these logs in the message above. There is just one block you produced in there (the latter of the two) at 03:12 UTC as likely the logs rotated.
The block you produced was 3NKnZubgwNfqGpT7g8nTC97yD2nSTuMFCPfrPgC1trhNgfxgk8cZ
which seems to have been broadcast (ref Broadcasting new state over gossip net
). I searched for this state hash in my logs and it only turned up on one of my nodes which didn't rebroadcast as received too late. Seems it did take a very long time to propagate?
@garethtdavies thanks for answering!
Regarding log rotation > Good point. I was actually surprised that I could win two slots in a row.
I'm digging a bit in the log and writing what I found here.
journalctl --user -u mina -n 50000 | grep -i --regexp="Successfully produced a new block" -A 3
gives me this
Oct 27 01:46:29 ip-172-31-80-92 mina[1486]: 2021-10-27 01:46:29 UTC [Info] Successfully produced a new block
Oct 27 01:46:31 ip-172-31-80-92 mina[1486]: 2021-10-27 01:46:31 UTC [Info] Generated transition $state_hash was accepted into transition frontier
Oct 27 01:46:31 ip-172-31-80-92 mina[1486]: state_hash: "3NLuTSpXNH8kpRnEbbFmxzqHxbAazCY553QMeK2XkPnjLxqeXB1J"
Oct 27 01:46:31 ip-172-31-80-92 mina[1486]: 2021-10-27 01:46:31 UTC [Info] Updating new available work took 7.303476333618164 ms
--
Oct 27 23:28:10 ip-172-31-80-92 mina[704]: 2021-10-27 23:28:10 UTC [Info] Successfully produced a new block
Oct 27 23:28:13 ip-172-31-80-92 mina[704]: 2021-10-27 23:28:13 UTC [Info] Generated transition $state_hash was accepted into transition frontier
Oct 27 23:28:13 ip-172-31-80-92 mina[704]: state_hash: "3NKf989hV9QVk8dVwrQtkwSAQxrX5xNjnBvgxRqfaZse4t7qhcm5"
Oct 27 23:28:13 ip-172-31-80-92 mina[704]: 2021-10-27 23:28:13 UTC [Info] Updating new available work took 12.995243072509766 ms
--
Oct 28 02:13:24 ip-172-31-80-92 mina[5038]: 2021-10-28 02:13:24 UTC [Info] Successfully produced a new block
Oct 28 02:13:26 ip-172-31-80-92 mina[5038]: 2021-10-28 02:13:26 UTC [Info] Generated transition $state_hash was accepted into transition frontier
Oct 28 02:13:26 ip-172-31-80-92 mina[5038]: state_hash: "3NKrwRPsAjYxpEUpuJ4g6d22mQkUebUm8DNdP2DNAPBdhR1YGMLz"
Oct 28 02:13:26 ip-172-31-80-92 mina[5038]: 2021-10-28 02:13:26 UTC [Info] Updating new available work took 8.159160614013672 ms
--
Oct 28 03:13:23 ip-172-31-80-92 mina[5038]: 2021-10-28 03:13:23 UTC [Info] Successfully produced a new block
Oct 28 03:13:25 ip-172-31-80-92 mina[5038]: 2021-10-28 03:13:25 UTC [Info] Generated transition $state_hash was accepted into transition frontier
Oct 28 03:13:25 ip-172-31-80-92 mina[5038]: state_hash: "3NKnZubgwNfqGpT7g8nTC97yD2nSTuMFCPfrPgC1trhNgfxgk8cZ"
Oct 28 03:13:25 ip-172-31-80-92 mina[5038]: 2021-10-28 03:13:25 UTC [Info] Updating new available work took 9.974956512451172 ms
--
Oct 28 21:01:10 ip-172-31-80-92 mina[701]: 2021-10-28 21:01:10 UTC [Info] Successfully produced a new block
Oct 28 21:01:13 ip-172-31-80-92 mina[701]: 2021-10-28 21:01:13 UTC [Info] Generated transition $state_hash was accepted into transition frontier
Oct 28 21:01:13 ip-172-31-80-92 mina[701]: state_hash: "3NKxwMwAoTtcwnioWP2Ct22yRV3xC7qnDC1KfP5eUeC4BXQDBCQL"
Oct 28 21:01:13 ip-172-31-80-92 mina[701]: 2021-10-28 21:01:13 UTC [Info] Updating new available work took 18.558263778686523 ms
In total, I created 5 blocks.
3NLuTSpXNH8kpRnEbbFmxzqHxbAazCY553QMeK2XkPnjLxqeXB1J
3NKf989hV9QVk8dVwrQtkwSAQxrX5xNjnBvgxRqfaZse4t7qhcm5
3NKrwRPsAjYxpEUpuJ4g6d22mQkUebUm8DNdP2DNAPBdhR1YGMLz
produced at 2021-10-28 02:13:24 UTC
3NKnZubgwNfqGpT7g8nTC97yD2nSTuMFCPfrPgC1trhNgfxgk8cZ
produced at 2021-10-28 03:13:23 UTC
.3NKxwMwAoTtcwnioWP2Ct22yRV3xC7qnDC1KfP5eUeC4BXQDBCQL
I did the same as you, looking if my other nodes received the produced blocks.
3NKnZubgwNfqGpT7g8nTC97yD2nSTuMFCPfrPgC1trhNgfxgk8cZ
has NOT been received by my other node, but by yours at 2021-10-28 02:15:07
. 3NKrwRPsAjYxpEUpuJ4g6d22mQkUebUm8DNdP2DNAPBdhR1YGMLz
produced at 2021-10-28 02:13:24 UTC
was received by ONE node only at 2021-10-28 02:15:32 UTC
2021-10-28 02:15:32 UTC [Error] Validation error: external transition with state hash $state_hash was rejected for reason "invalid time"
state_hash: "3NKrwRPsAjYxpEUpuJ4g6d22mQkUebUm8DNdP2DNAPBdhR1YGMLz"
but was rejected for invalid time
which is strange because in this case it is just 2'8'' after.
I also have a message "not rebroadcasting"... but looks like it is for another block (...8Ni) ...
2021-10-28 02:15:49 UTC [Warn] Not rebroadcasting block $state_hash because it was received "1 slots too late"
state_hash: "3NKieJZkuNpTH4yquu6NAav5okDEaE9TVAYKMmuNZqGC4ap2o8Ni"
Maybe it is just a glitch in the logging. I should switch to JSON structured logs.
Coming back to your remark about propagation time : the blocks were visible for you after 1'40" and 2'8'' for me.
I had a look at the block production time, here there are 1'29", 1'10", 1'24", 1'23", 1'10"
If we take the worst case
it sums up to 3'39". I noticed in the mina client status
that there is a acceptable network delay
of 3 minutes. I guess I cannot do anything for block propagation. Does it means that my computer is not fast enough in creating the block?
Could you (and also anyone else reading this) post me the output of the following command, so I have an idea of your block creation time? I would really appreciate it.
journalctl --user -u mina -n 100000 | grep -i --regexp="Successfully produced a new block\|Producing new block with parent"
Information about CPU
lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 46 bits physical, 48 bits virtual
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 2
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz
Stepping: 7
CPU MHz: 2999.998
BogoMIPS: 5999.99
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 128 KiB
L1i cache: 128 KiB
L2 cache: 4 MiB
L3 cache: 35.8 MiB
NUMA node0 CPU(s): 0-7
Vulnerability Itlb multihit: KVM: Mitigation: VMX unsupported
Vulnerability L1tf: Mitigation; PTE Inversion
Vulnerability Mds: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Meltdown: Mitigation; PTI
Vulnerability Spec store bypass: Vulnerable
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Full generic retpoline, STIBP disabled, RSB filling
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology
nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor la
hf_lm abm 3dnowprefetch invpcid_single pti fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx
512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke
It is suspected that these issues are primarily caused by long-running single-threaded operations in ocaml; 1.2.1 alpha has significant improvements in queue handling which may resolve this (and other) issues caused by underlying resource starvation (due long-running, single thread.) Looking for feedback from devnet.
It did not produce blocks. log is attached.
Version : Ubuntu 18.04 Bionic Beaver: minaprotocol/mina-daemon:1.3.0-9b0369c-bionic-mainnet
A possible instance of this issue occurred in 1.3.1 where a block with this state_hash 3NLpEJKNmeWWNRoMvaCnLett212svUxU4GS1fczhxPh4bVqYp7Rt
seemingly failed to propagate. However there aren't logs to corroborate this.
This may be different than the older parts of the thread, but I have confirmed once more failure to propagate and have logs. At 2022-06-16 02:34:38.661015Z state hash 3NLs4VZmdwsyBA3KGHCb9T7WqMyRxwnP5AcD6CjUyXwpBCDDpmGc "was accepted into transition frontier", but it doesn't seem to be visible to the network. Logs here: https://drive.google.com/drive/u/3/folders/1z6OojQemOs_wpU8cjiCCV9bpgaN4uD-X in file producedblocknotseen-20220616-mina.log.15
I can confirm we are gathering all data including old epochs. The current state of the issue is way much better than how it is, not to mention at acceptable levels. The network is significantly healthy. From epoch 22 to 28 we had this issue only 3 times. Since releases are slower and also stable. We are planning to observe the state for future epochs to have more concrete data.
After running with vrf check on epoch 10 we spot 4 blocks missing we have on crash on slot time and one block generated 1 block behind while rest is valid. Asking several members with archive node their node saw the block late while we don't spot any issue. We weigh on the network gossip layer issue. Logs are below log for slot 74978 Node 1 (US Region).txt logs for slot 71507 Node 1 (US Region).txt logs for slot 71725 Node 1 (US Region).txt log for slot 74429 Node 1 (US Region).txt log for slot 74978 node 2 (EU Region).txt logs for slot 71507 node 2 (EU Region).txt logs for slot 71725 node 2 (EU Region).txt log for slot 74429 node 2 (EU Region).txt log for slot 74978 node 3 (EU Region).txt logs for slot 71507 node 3 (EU Region).txt logs for slot 71725 node 3 (EU Region).txt log for slot 74429 node 3 (EU Region).txt