AleoNet / snarkOS

A Decentralized Operating System for ZK Applications
http://snarkos.org
Apache License 2.0
4.24k stars 2.59k forks source link

[Bug] Consensus halt on isonet #3356

Open joske opened 2 weeks ago

joske commented 2 weeks ago

🐛 Bug Report

A new dev network was spun up on AWS, with only execution transactions. Load started at about 18:45 UTC on July 3rd 2024. The goal was to check RSS growth when only execution transactions (no deployments) are present. A load of about 11 TPS was observed.

At about 12:15 on 6th of July 2024, consensus came to a halt due to a rounds mismatch. Block 288032 was the last block produced.

Log excerpt from node 0:

Jul 06 12:07:49 ip-172-31-43-154 snarkos[47149]: Our batch with 34 transmissions for round 576092 was certified!
Jul 06 12:07:49 ip-172-31-43-154 snarkos[47149]: 2024-07-06T12:07:49.689911Z  INFO snarkos_node_bft::bft:
Jul 06 12:07:49 ip-172-31-43-154 snarkos[47149]: Round 576092 elected a leader - aleo1rhgdu77hgyqd3xjj8ucu3jj9r2krwz6mnzyd80gncr5fxcwlh5rsvzp9px
Jul 06 12:07:49 ip-172-31-43-154 snarkos[47149]: 2024-07-06T12:07:49.689943Z DEBUG snarkos_node_bft::primary: Primary is not ready to propose the next round
Jul 06 12:07:49 ip-172-31-43-154 snarkos[47149]: 2024-07-06T12:07:49.765758Z DEBUG snarkos_node_bft::primary: Stored a batch certificate for round 576092 from '18.207.151.1:5000'
Jul 06 12:07:49 ip-172-31-43-154 snarkos[47149]: 2024-07-06T12:07:49.765902Z  INFO snarkos_node_bft::bft:
Jul 06 12:07:49 ip-172-31-43-154 snarkos[47149]: Round 576092 elected a leader - aleo1rhgdu77hgyqd3xjj8ucu3jj9r2krwz6mnzyd80gncr5fxcwlh5rsvzp9px
Jul 06 12:07:49 ip-172-31-43-154 snarkos[47149]: 2024-07-06T12:07:49.765941Z  INFO snarkos_node_bft::helpers::storage: Starting round 576093...
Jul 06 12:07:49 ip-172-31-43-154 snarkos[47149]: 2024-07-06T12:07:49.765954Z DEBUG snarkos_node_bft::primary: Primary is ready to propose the next round
Jul 06 12:07:49 ip-172-31-43-154 snarkos[47149]: 2024-07-06T12:07:49.765971Z DEBUG snarkos_node_bft::primary: Primary is safely skipping a batch proposal - Timestamp is too soon after the previo>
Jul 06 12:07:49 ip-172-31-43-154 snarkos[47149]: 2024-07-06T12:07:49.997261Z DEBUG snarkos_node_bft::primary: Signed a batch for round 576093 from '184.73.103.186:5000'
Jul 06 12:07:50 ip-172-31-43-154 snarkos[47149]: 2024-07-06T12:07:50.008066Z DEBUG snarkos_node_bft::primary: Stored a batch certificate for round 576093 from '184.73.103.186:5000'
Jul 06 12:07:50 ip-172-31-43-154 snarkos[47149]: 2024-07-06T12:07:50.077498Z DEBUG snarkos_node_bft::primary: Signed a batch for round 576093 from '54.175.16.47:5000'
Jul 06 12:07:52 ip-172-31-43-154 snarkos[47149]: 2024-07-06T12:07:52.449277Z DEBUG snarkos_node_bft::primary: Signed a batch for round 576093 from '18.207.151.1:5000'
Jul 06 12:07:53 ip-172-31-43-154 snarkos[47149]: 2024-07-06T12:07:53.111824Z  INFO snarkos_node_bft::primary: Proposing a batch with 24 transmissions for round 576093...
Jul 06 12:07:53 ip-172-31-43-154 snarkos[47149]: 2024-07-06T12:07:53.194971Z  INFO snarkos_node_bft::primary: Received a batch signature for round 576093 from '184.73.103.186:5000'
Jul 06 12:07:53 ip-172-31-43-154 snarkos[47149]: 2024-07-06T12:07:53.195831Z  INFO snarkos_node_bft::primary: Received a batch signature for round 576093 from '18.207.151.1:5000'
Jul 06 12:07:53 ip-172-31-43-154 snarkos[47149]: 2024-07-06T12:07:53.200389Z  INFO snarkos_node_bft::primary: Received a batch signature for round 576093 from '54.175.16.47:5000'
Jul 06 12:07:53 ip-172-31-43-154 snarkos[47149]: 2024-07-06T12:07:53.200681Z  INFO snarkos_node_bft::primary: Quorum threshold reached - Preparing to certify our batch for round 576093...
Jul 06 12:07:53 ip-172-31-43-154 snarkos[47149]: 2024-07-06T12:07:53.203852Z DEBUG snarkos_node_bft::primary: Stored a batch certificate for round 576093
Jul 06 12:07:53 ip-172-31-43-154 snarkos[47149]: 2024-07-06T12:07:53.203948Z  INFO snarkos_node_bft::bft: Proceeding to commit round 576092 with leader 'aleo1rhgdu77hgyq..'
Jul 06 12:07:53 ip-172-31-43-154 snarkos[47149]: 2024-07-06T12:07:53.500634Z DEBUG snarkos_node_bft::primary: Stored a batch certificate for round 576093 from '18.207.151.1:5000'
Jul 06 12:07:53 ip-172-31-43-154 snarkos[47149]: 2024-07-06T12:07:53.670968Z DEBUG snarkos_node_bft::primary: Stored a batch certificate for round 576093 from '54.175.16.47:5000'
Jul 06 12:07:54 ip-172-31-43-154 snarkos[47149]: 2024-07-06T12:07:54.273684Z DEBUG snarkos_node_bft::primary: Signed a batch for round 576092 from '35.173.249.135:5000'
Jul 06 12:07:54 ip-172-31-43-154 snarkos[47149]: 2024-07-06T12:07:54.431138Z  INFO snarkos_node_bft_ledger_service::ledger:
Jul 06 12:07:54 ip-172-31-43-154 snarkos[47149]: Advanced to block 288031 at round 576090 - ab1qw8rhvepqcs0pqfxrvwstcy0naymrcqrfzl8xd8apqv5x0p6tsxsasf04d
Jul 06 12:07:54 ip-172-31-43-154 snarkos[47149]: 2024-07-06T12:07:54.431892Z  INFO snarkos_node_bft::bft:
Jul 06 12:07:54 ip-172-31-43-154 snarkos[47149]: Committing a subdag from round 576090 with 64 transmissions: [(576088, 3), (576089, 4), (576090, 1)]
Jul 06 12:07:54 ip-172-31-43-154 snarkos[47149]: 2024-07-06T12:07:54.688209Z  INFO snarkos_node_bft::gateway: Connected to 4 validators (of 4 bonded validators)
Jul 06 12:07:54 ip-172-31-43-154 snarkos[47149]: 2024-07-06T12:07:54.688244Z DEBUG snarkos_node_bft::gateway:   18.207.151.1:5000 - aleo12ux3gdauck0v60westgcpqj7v8rrcr3v346e4jtq04q7kkt22czsh808v2
Jul 06 12:07:54 ip-172-31-43-154 snarkos[47149]: 2024-07-06T12:07:54.688253Z DEBUG snarkos_node_bft::gateway:   54.175.16.47:5000 - aleo1p9sg8gapg22p3j42tang7c8kqzp4lhe6mg77gx32yys2a5y7pq9sxh6wrd
Jul 06 12:07:54 ip-172-31-43-154 snarkos[47149]: 2024-07-06T12:07:54.688260Z DEBUG snarkos_node_bft::gateway:   184.73.103.186:5000 - aleo1s3ws5tra87fjycnjrwsjcrnw2qxr8jfqqdugnf0xzqqw29q9m5pqem2>
Jul 06 12:07:54 ip-172-31-43-154 snarkos[47149]: 2024-07-06T12:07:54.688266Z DEBUG snarkos_node_bft::gateway:   35.173.249.135:5000 - aleo1ashyu96tjwe63u0gtnnv8z5lhapdu4l5pjsl2kha7fv7hvz2eqxs5dz>
Jul 06 12:07:54 ip-172-31-43-154 snarkos[47149]: 2024-07-06T12:07:54.688301Z DEBUG snarkos_node_bft::gateway: Connecting to validator 18.209.175.104:5000...
Jul 06 12:07:54 ip-172-31-43-154 snarkos[47149]: 2024-07-06T12:07:54.688703Z DEBUG snarkos_node_bft::gateway: [MemoryPool] Gateway is connecting to 18.209.175.104:5000...
Jul 06 12:07:54 ip-172-31-43-154 snarkos[47149]: 2024-07-06T12:07:54.688875Z DEBUG snarkos_node_bft::gateway: [MemoryPool] Gateway received a connection request from '18.209.175.104:54192'
Jul 06 12:07:54 ip-172-31-43-154 snarkos[47149]: 2024-07-06T12:07:54.689201Z  WARN snarkos_node_bft::gateway: Unable to connect to '18.209.175.104:5000' - [MemoryPool] '18.209.175.104:5000' disc>
Jul 06 12:07:55 ip-172-31-43-154 snarkos[47149]: 2024-07-06T12:07:55.616411Z DEBUG snarkos_node_bft::primary: Skipping batch proposal (node is syncing)
Jul 06 12:07:56 ip-172-31-43-154 snarkos[47149]: 2024-07-06T12:07:56.784454Z  INFO snarkos_node_bft_ledger_service::ledger:
Jul 06 12:07:56 ip-172-31-43-154 snarkos[47149]: Advanced to block 288032 at round 576092 - ab1w07j9k67kqdpwwfcjt7mcf5dceq48gnxze7pqqdsp28w98mfycrqahj3n2
Jul 06 12:07:56 ip-172-31-43-154 snarkos[47149]: 2024-07-06T12:07:56.785067Z  INFO snarkos_node_bft::bft:
Jul 06 12:07:56 ip-172-31-43-154 snarkos[47149]: Committing a subdag from round 576092 with 126 transmissions: [(576090, 4), (576091, 4), (576092, 1)]
Jul 06 12:07:56 ip-172-31-43-154 snarkos[47149]: 2024-07-06T12:07:56.790316Z  INFO snarkos_node_bft::primary:
Jul 06 12:07:56 ip-172-31-43-154 snarkos[47149]: Our batch with 24 transmissions for round 576093 was certified!
Jul 06 12:07:56 ip-172-31-43-154 snarkos[47149]: 2024-07-06T12:07:56.790384Z  INFO snarkos_node_bft::helpers::storage: Starting round 576094...
Jul 06 12:07:56 ip-172-31-43-154 snarkos[47149]: 2024-07-06T12:07:56.790401Z DEBUG snarkos_node_bft::bft: BFT is safely skipping an update for round 576093, as storage is at round 576094
Jul 06 12:07:56 ip-172-31-43-154 snarkos[47149]: 2024-07-06T12:07:56.790408Z DEBUG snarkos_node_bft::bft: BFT is safely skipping an update for round 576093, as storage is at round 576094
Jul 06 12:07:56 ip-172-31-43-154 snarkos[47149]: 2024-07-06T12:07:56.790417Z DEBUG snarkos_node_bft::primary: Primary is not ready to propose the next round
Jul 06 12:07:56 ip-172-31-43-154 snarkos[47149]: 2024-07-06T12:07:56.790416Z DEBUG snarkos_node_bft::primary: Primary is ready to propose the next round
Jul 06 12:07:56 ip-172-31-43-154 snarkos[47149]: 2024-07-06T12:07:56.790452Z DEBUG snarkos_node_bft::primary: Primary is not ready to propose the next round
Jul 06 12:07:58 ip-172-31-43-154 snarkos[47149]: 2024-07-06T12:07:58.092768Z  INFO snarkos_node_bft::primary: Proposing a batch with 37 transmissions for round 576094...
Jul 06 12:07:58 ip-172-31-43-154 snarkos[47149]: 2024-07-06T12:07:58.118475Z DEBUG snarkos_node_bft::primary: Skipping batch proposal (node is syncing)

Log excerpt from node 4:

Jul 06 12:07:49 ip-172-31-41-79 snarkos[45347]: Round 576092 elected a leader - aleo1rhgdu77hgyqd3xjj8ucu3jj9r2krwz6mnzyd80gncr5fxcwlh5rsvzp9px
Jul 06 12:07:49 ip-172-31-41-79 snarkos[45347]: 2024-07-06T12:07:49.695706Z  INFO snarkos_node_bft::helpers::storage: Starting round 576093...
Jul 06 12:07:49 ip-172-31-41-79 snarkos[45347]: 2024-07-06T12:07:49.695719Z DEBUG snarkos_node_bft::primary: Primary is ready to propose the next round
Jul 06 12:07:49 ip-172-31-41-79 snarkos[45347]: 2024-07-06T12:07:49.918644Z  INFO snarkos_node_bft::primary: Proposing a batch with 10 transmissions for round 576093...
Jul 06 12:07:49 ip-172-31-41-79 snarkos[45347]: 2024-07-06T12:07:49.996669Z DEBUG snarkos_node_bft::primary: Signed a batch for round 576093 from '184.73.103.186:5000'
Jul 06 12:07:50 ip-172-31-41-79 snarkos[45347]: 2024-07-06T12:07:50.007402Z DEBUG snarkos_node_bft::primary: Stored a batch certificate for round 576093 from '184.73.103.186:5000'
Jul 06 12:07:50 ip-172-31-41-79 snarkos[45347]: 2024-07-06T12:07:50.044941Z  INFO snarkos_node_bft::primary: Received a batch signature for round 576093 from '184.73.103.186:5000'
Jul 06 12:07:50 ip-172-31-41-79 snarkos[45347]: 2024-07-06T12:07:50.079148Z  INFO snarkos_node_bft::primary: Received a batch signature for round 576093 from '18.209.175.104:5000'
Jul 06 12:07:50 ip-172-31-41-79 snarkos[45347]: 2024-07-06T12:07:50.080415Z  INFO snarkos_node_bft::primary: Received a batch signature for round 576093 from '18.207.151.1:5000'
Jul 06 12:07:50 ip-172-31-41-79 snarkos[45347]: 2024-07-06T12:07:50.080788Z  INFO snarkos_node_bft::primary: Quorum threshold reached - Preparing to certify our batch for round 576093...
Jul 06 12:07:50 ip-172-31-41-79 snarkos[45347]: 2024-07-06T12:07:50.082982Z DEBUG snarkos_node_bft::primary: Stored a batch certificate for round 576093
Jul 06 12:07:50 ip-172-31-41-79 snarkos[45347]: 2024-07-06T12:07:50.083053Z  INFO snarkos_node_bft::bft: Proceeding to commit round 576092 with leader 'aleo1rhgdu77hgyq..'
Jul 06 12:07:51 ip-172-31-41-79 snarkos[45347]: 2024-07-06T12:07:51.318055Z  INFO snarkos_node_bft_ledger_service::ledger:
Jul 06 12:07:51 ip-172-31-41-79 snarkos[45347]: Advanced to block 288031 at round 576090 - ab1qw8rhvepqcs0pqfxrvwstcy0naymrcqrfzl8xd8apqv5x0p6tsxsasf04d
Jul 06 12:07:51 ip-172-31-41-79 snarkos[45347]: 2024-07-06T12:07:51.318917Z  INFO snarkos_node_bft::bft:
Jul 06 12:07:51 ip-172-31-41-79 snarkos[45347]: Committing a subdag from round 576090 with 64 transmissions: [(576088, 3), (576089, 4), (576090, 1)]
Jul 06 12:07:52 ip-172-31-41-79 snarkos[45347]: 2024-07-06T12:07:52.361937Z DEBUG snarkos_node_bft::primary: Signed a batch for round 576093 from '18.207.151.1:5000'
Jul 06 12:07:52 ip-172-31-41-79 snarkos[45347]: 2024-07-06T12:07:52.567190Z  INFO snarkos_node_bft::gateway: Connected to 4 validators (of 4 bonded validators)
Jul 06 12:07:52 ip-172-31-41-79 snarkos[45347]: 2024-07-06T12:07:52.567264Z DEBUG snarkos_node_bft::gateway:   184.73.103.186:5000 - aleo1s3ws5tra87fjycnjrwsjcrnw2qxr8jfqqdugnf0xzqqw29q9m5pqem2u>
Jul 06 12:07:52 ip-172-31-41-79 snarkos[45347]: 2024-07-06T12:07:52.567277Z DEBUG snarkos_node_bft::gateway:   35.173.249.135:5000 - aleo1ashyu96tjwe63u0gtnnv8z5lhapdu4l5pjsl2kha7fv7hvz2eqxs5dz0>
Jul 06 12:07:52 ip-172-31-41-79 snarkos[45347]: 2024-07-06T12:07:52.567287Z DEBUG snarkos_node_bft::gateway:   18.207.151.1:5000 - aleo12ux3gdauck0v60westgcpqj7v8rrcr3v346e4jtq04q7kkt22czsh808v2
Jul 06 12:07:52 ip-172-31-41-79 snarkos[45347]: 2024-07-06T12:07:52.567297Z DEBUG snarkos_node_bft::gateway:   18.209.175.104:5000 - aleo1rhgdu77hgyqd3xjj8ucu3jj9r2krwz6mnzyd80gncr5fxcwlh5rsvzp9>
Jul 06 12:07:52 ip-172-31-41-79 snarkos[45347]: 2024-07-06T12:07:52.567388Z DEBUG snarkos_node_bft::gateway: Connecting to validator 54.175.16.47:5000...
Jul 06 12:07:52 ip-172-31-41-79 snarkos[45347]: 2024-07-06T12:07:52.567725Z DEBUG snarkos_node_bft::gateway: [MemoryPool] Gateway is connecting to 54.175.16.47:5000...
Jul 06 12:07:52 ip-172-31-41-79 snarkos[45347]: 2024-07-06T12:07:52.567868Z DEBUG snarkos_node_bft::gateway: [MemoryPool] Gateway received a connection request from '54.175.16.47:60938'
Jul 06 12:07:52 ip-172-31-41-79 snarkos[45347]: 2024-07-06T12:07:52.568304Z  WARN snarkos_node_bft::gateway: Unable to connect to '54.175.16.47:5000' - [MemoryPool] '54.175.16.47:5000' disconnec>
Jul 06 12:07:53 ip-172-31-41-79 snarkos[45347]: 2024-07-06T12:07:53.199076Z DEBUG snarkos_node_bft::primary: Signed a batch for round 576093 from '18.209.175.104:5000'
Jul 06 12:07:53 ip-172-31-41-79 snarkos[45347]: 2024-07-06T12:07:53.500945Z DEBUG snarkos_node_bft::primary: Stored a batch certificate for round 576093 from '18.207.151.1:5000'
Jul 06 12:07:53 ip-172-31-41-79 snarkos[45347]: 2024-07-06T12:07:53.665673Z  INFO snarkos_node_bft_ledger_service::ledger:
Jul 06 12:07:53 ip-172-31-41-79 snarkos[45347]: Advanced to block 288032 at round 576092 - ab1w07j9k67kqdpwwfcjt7mcf5dceq48gnxze7pqqdsp28w98mfycrqahj3n2
Jul 06 12:07:53 ip-172-31-41-79 snarkos[45347]: 2024-07-06T12:07:53.666516Z  INFO snarkos_node_bft::bft:
Jul 06 12:07:53 ip-172-31-41-79 snarkos[45347]: Committing a subdag from round 576092 with 126 transmissions: [(576090, 4), (576091, 4), (576092, 1)]
Jul 06 12:07:53 ip-172-31-41-79 snarkos[45347]: 2024-07-06T12:07:53.670803Z  INFO snarkos_node_bft::primary:
Jul 06 12:07:53 ip-172-31-41-79 snarkos[45347]: Our batch with 10 transmissions for round 576093 was certified!
Jul 06 12:07:53 ip-172-31-41-79 snarkos[45347]: 2024-07-06T12:07:53.670866Z DEBUG snarkos_node_bft::primary: Primary is not ready to propose the next round
Jul 06 12:07:55 ip-172-31-41-79 snarkos[45347]: 2024-07-06T12:07:55.571979Z DEBUG snarkos_node_bft::primary: Stored a batch certificate for round 576093 from '18.209.175.104:5000'
Jul 06 12:07:55 ip-172-31-41-79 snarkos[45347]: 2024-07-06T12:07:55.572111Z  INFO snarkos_node_bft::helpers::storage: Starting round 576094...
Jul 06 12:07:55 ip-172-31-41-79 snarkos[45347]: 2024-07-06T12:07:55.572128Z DEBUG snarkos_node_bft::primary: Primary is ready to propose the next round
Jul 06 12:07:55 ip-172-31-41-79 snarkos[45347]: 2024-07-06T12:07:55.988541Z  INFO snarkos_node_bft::primary: Proposing a batch with 18 transmissions for round 576094...
Jul 06 12:07:56 ip-172-31-41-79 snarkos[45347]: 2024-07-06T12:07:56.159933Z  INFO snarkos_node_bft::primary: Received a batch signature for round 576094 from '18.207.151.1:5000'
Jul 06 12:07:56 ip-172-31-41-79 snarkos[45347]: 2024-07-06T12:07:56.530602Z DEBUG snarkos_node_bft::primary: Proposed batch for round 576094 is still valid
Jul 06 12:07:56 ip-172-31-41-79 snarkos[45347]: 2024-07-06T12:07:56.530628Z DEBUG snarkos_node_bft::primary: Resending batch proposal for round 576094 to peer '184.73.103.186:5000'
Jul 06 12:07:56 ip-172-31-41-79 snarkos[45347]: 2024-07-06T12:07:56.530613Z DEBUG snarkos_node_bft::primary: Resending batch proposal for round 576094 to peer '18.209.175.104:5000'
Jul 06 12:07:56 ip-172-31-41-79 snarkos[45347]: 2024-07-06T12:07:56.530647Z DEBUG snarkos_node_bft::primary: Resending batch proposal for round 576094 to peer '35.173.249.135:5000'
Jul 06 12:07:57 ip-172-31-41-79 snarkos[45347]: 2024-07-06T12:07:57.524607Z DEBUG snarkos_node_bft::primary: Primary is on round 576094, and no longer signing for round 576092 from '35.173.249.1>
Jul 06 12:07:57 ip-172-31-41-79 snarkos[45347]: 2024-07-06T12:07:57.994841Z DEBUG snarkos_node_bft::primary: Signed a batch for round 576094 from '18.207.151.1:5000'
Jul 06 12:07:58 ip-172-31-41-79 snarkos[45347]: 2024-07-06T12:07:58.184622Z DEBUG snarkos_node_bft::primary: Signed a batch for round 576094 from '18.209.175.104:5000'
Jul 06 12:07:59 ip-172-31-41-79 snarkos[45347]: 2024-07-06T12:07:59.032134Z DEBUG snarkos_node_bft::primary: Proposed batch for round 576094 is still valid

We downloaded the ledger and tried to analyze it, but it fails with the following error:

2024-07-08T13:09:34.287940Z  INFO test_ledger_load: ledger loading
2024-07-08T13:09:34.329680Z  INFO snarkvm_ledger: Loading the ledger from storage...
thread 'main' panicked at ledger/src/bin.rs:39:102:
called `Result::unwrap()` on an `Err` value: Failed to load block 288032 from the ledger

We tracked this to

Invalid block metadata: {"network":0,"round":576092,"height":288032,"cumulative_weight":0,"cumulative_proof_target":0,"coinbase_target":1023,"proof_target":256,"last_coinbase_target":1023,"last_coinbase_timestamp":1696118400,"timestamp":1720267664}

Restarting the nodes didn't resume consensus.

Steps to Reproduce

not know how to reproduce

Expected Behavior

Consensus doesn't stop

Your Environment

raychu86 commented 2 weeks ago

Good thing is that it doesn't seem to be a fork, so I'm less concerned about a consensus issue.

@joske @ljedrz snarkOS https://github.com/AleoNet/snarkOS/commit/c4d725f923f92bad8d6aab0fa15824867efbcc0c is an older version that uses a lower minimum coinbase/proof target. Is there any chance that you are trying to reload the ledger with a newer version of snarkOS that has the updated minimum values?

joske commented 2 weeks ago

@raychu86 No, we tried with the exact snarkVM version 6d64025 that snarkOS c4d725f points to. To be sure, I'll try again.

raychu86 commented 2 weeks ago

@joske The failing check is this:

    pub fn is_valid(&self) -> bool {
        match self.height == 0u32 {
            true => self.is_genesis(),
            false => {
                // Ensure the network ID is correct.
                self.network == N::ID
                    // Ensure the round is nonzero.
                    && self.round != 0u64
                    // Ensure the height is nonzero.
                    && self.height != 0u32
                    // Ensure the round is at least as large as the height.
                    && self.round >= self.height as u64
                    // Ensure the coinbase target is at or above the minimum.
                    && self.coinbase_target >= N::GENESIS_COINBASE_TARGET
                    // Ensure the proof target is at or above the minimum.
                    && self.proof_target >= N::GENESIS_PROOF_TARGET
                    // Ensure the coinbase target is larger than the proof target.
                    && self.coinbase_target > self.proof_target
                    // Ensure the last coinbase target is at or above the minimum.
                    && self.last_coinbase_target >= N::GENESIS_COINBASE_TARGET
                    // Ensure the last coinbase timestamp is after the genesis timestamp.
                    && self.last_coinbase_timestamp >= N::GENESIS_TIMESTAMP
                    // Ensure the timestamp in the block is after the genesis timestamp.
                    && self.timestamp > N::GENESIS_TIMESTAMP
            }
        }

From what I can see, all the attributes are correct. The only one I can't determine is self.network == N::ID. This will fail if you spin up the network without --network 0

joske commented 2 weeks ago

The nodes are started with

/root/.cargo/bin/snarkos start --metrics --nodisplay --bft 0.0.0.0:5000 --rest 0.0.0.0:3030 --peers 18.220.230.167:4130,18.222.48.7:4130,18.188.187.22:4130 --validators 18.227.0.57:5000,3.145.37.212:5000,18.222.184.97:5000,3.139.97.186:5000,3.138.134.228:5000 --verbosity 1 --dev 4 --dev-num-validators 5 --validator --rest-rps 10 --no-dev-txs
elderhammer commented 2 weeks ago

Did you use snarkVM's test feature when compiling the snarkOS executable?

joske commented 2 weeks ago

Don't know, it was spun up by @onetrickwolf using their ansible scripts (I think)

onetrickwolf commented 2 weeks ago

We just build with the 'build_ubuntu.sh' script found in snarkOS.

raychu86 commented 2 weeks ago

Can you try again with cargo run --release or try to re-install snarkOS rev https://github.com/AleoNet/snarkOS/commit/c4d725f923f92bad8d6aab0fa15824867efbcc0c using cargo install --path .? Just to ensure that it's not an issue with the current binary?

And an even lower level approach is to see which line/check is causing the Invalid block metadata failure.

joske commented 2 weeks ago

This isonet is gone. Restarted on commit 878624d6ccab6dfeb52f69ce54ee885464cdf7d8.