Block production in local development network stops after block #199

ThunderDeliverer commented 3 years ago

Describe the bug Every time I run the local development network, the block stop at #199. No new blocks are mined after and every transaction that I try to do when this happens raises the following error: Error: the string "1014: Priority is too low: (383246202401776 vs 383246202401776): The transaction has too low priority to replace another transaction already in the pool." was thrown, throw an Error :)

Expected Behavior New blocks should be mined after #199 and transactions included in them.

Current Behavior After block #199 is mined, block production stops and no new transactions can be sent.

Steps to Reproduce

Clone repository and run the build steps
Run the network using make run or make restart
Do enough transactions to reach block #199

Additional context

Node version: v14.15.4
OS: macOS BigSur v 11.1
Command line options: cargo run --features with-mandala-runtime -- --dev -lruntime=debug --instant-sealing

Logs. Please format it as code blocks

2021-09-27 23:11:44.839  INFO tokio-runtime-worker substrate: ✨ Imported #198 (0xd6b2…c344)    
2021-09-27 23:11:44.914  INFO tokio-runtime-worker sc_basic_authorship::basic_authorship: 🙌 Starting consensus session on top of parent 0xd6b23f184ed5681113536488cf017bc090584b4644f2f68f91c24465221bc344    
2021-09-27 23:11:44.931  INFO tokio-runtime-worker sc_basic_authorship::basic_authorship: 🎁 Prepared block for proposing at 199 [hash: 0xd8796e6f70e94410ff16184bcdfb43f208dca426b77b7c74cfec21295c5028f9; parent_hash: 0xd6b2…c344; extrinsics (3): [0x25bd…b86f, 0x4b87…1751, 0xdbe6…bcb9]]    
2021-09-27 23:11:44.932  WARN tokio-runtime-worker sc_service::client::client: Possible safety violation: attempted to re-finalize last finalized block 0xd6b23f184ed5681113536488cf017bc090584b4644f2f68f91c24465221bc344     
2021-09-27 23:11:44.933  INFO tokio-runtime-worker sc_consensus_manual_seal::rpc: Instant Seal success: CreatedBlock { hash: 0xd8796e6f70e94410ff16184bcdfb43f208dca426b77b7c74cfec21295c5028f9, aux: ImportedAux { header_only: false, clear_justification_requests: false, needs_justification: false, bad_justification: false, is_new_best: true } }    
2021-09-27 23:11:44.934  INFO tokio-runtime-worker substrate: ✨ Imported #199 (0xd879…28f9)    
2021-09-27 23:11:44.935  INFO      offchain-worker cdp-engine offchain worker: cannot run offchain worker at 199: Failed to manipulate offchain lock    
2021-09-27 23:11:45.044  INFO tokio-runtime-worker sc_basic_authorship::basic_authorship: 🙌 Starting consensus session on top of parent 0xd8796e6f70e94410ff16184bcdfb43f208dca426b77b7c74cfec21295c5028f9    
2021-09-27 23:11:45.062  INFO tokio-runtime-worker sc_basic_authorship::basic_authorship: 🎁 Prepared block for proposing at 200 [hash: 0xf7f4b21ddc9faf1233cb3ba69d2dac000f288300f73cf043ef14cdc3d779bac6; parent_hash: 0xd879…28f9; extrinsics (2): [0xfc68…657d, 0x4b87…1751]]    
2021-09-27 23:11:45.062 ERROR tokio-runtime-worker sc_consensus_manual_seal::rpc: Instant Seal encountered an error: Transaction pool is empty, set create_empty to true,if you want to create empty blocks    
2021-09-27 23:11:48.598  INFO tokio-runtime-worker substrate: 💤 Idle (0 peers), best: #199 (0xd879…28f9), finalized #199 (0xd879…28f9), ⬇ 0 ⬆ 0    
2021-09-27 23:11:53.599  INFO tokio-runtime-worker substrate: 💤 Idle (0 peers), best: #199 (0xd879…28f9), finalized #199 (0xd879…28f9), ⬇ 0 ⬆ 0    
2021-09-27 23:11:58.600  INFO tokio-runtime-worker substrate: 💤 Idle (0 peers), best: #199 (0xd879…28f9), finalized #199 (0xd879…28f9), ⬇ 0 ⬆ 0    
2021-09-27 23:12:03.601  INFO tokio-runtime-worker substrate: 💤 Idle (0 peers), best: #199 (0xd879…28f9), finalized #199 (0xd879…28f9), ⬇ 0 ⬆ 0    
2021-09-27 23:12:08.605  INFO tokio-runtime-worker substrate: 💤 Idle (0 peers), best: #199 (0xd879…28f9), finalized #199 (0xd879…28f9), ⬇ 0 ⬆ 0    
2021-09-27 23:12:13.606  INFO tokio-runtime-worker substrate: 💤 Idle (0 peers), best: #199 (0xd879…28f9), finalized #199 (0xd879…28f9), ⬇ 0 ⬆ 0    
2021-09-27 23:12:18.607  INFO tokio-runtime-worker substrate: 💤 Idle (0 peers), best: #199 (0xd879…28f9), finalized #199 (0xd879…28f9), ⬇ 0 ⬆ 0    
2021-09-27 23:12:23.608  INFO tokio-runtime-worker substrate: 💤 Idle (0 peers), best: #199 (0xd879…28f9), finalized #199 (0xd879…28f9), ⬇ 0 ⬆ 0    
2021-09-27 23:12:28.611  INFO tokio-runtime-worker substrate: 💤 Idle (0 peers), best: #199 (0xd879…28f9), finalized #199 (0xd879…28f9), ⬇ 0 ⬆ 0    
2021-09-27 23:12:33.615  INFO tokio-runtime-worker substrate: 💤 Idle (0 peers), best: #199 (0xd879…28f9), finalized #199 (0xd879…28f9), ⬇ 0 ⬆ 0    
2021-09-27 23:12:34.930  INFO tokio-runtime-worker sc_basic_authorship::basic_authorship: 🙌 Starting consensus session on top of parent 0xd8796e6f70e94410ff16184bcdfb43f208dca426b77b7c74cfec21295c5028f9    
2021-09-27 23:12:34.941  INFO tokio-runtime-worker sc_basic_authorship::basic_authorship: 🎁 Prepared block for proposing at 200 [hash: 0x0f0a858b537bfce80e57d063be5643b52331f34105de42902cd1a133be8a7d5d; parent_hash: 0xd879…28f9; extrinsics (2): [0xcb1b…1243, 0x4b87…1751]]    
2021-09-27 23:12:34.941 ERROR tokio-runtime-worker sc_consensus_manual_seal::rpc: Instant Seal encountered an error: Transaction pool is empty, set create_empty to true,if you want to create empty blocks    
2021-09-27 23:12:38.620  INFO tokio-runtime-worker substrate: 💤 Idle (0 peers), best: #199 (0xd879…28f9), finalized #199 (0xd879…28f9), ⬇ 0 ⬆ 0    
2021-09-27 23:12:43.620  INFO tokio-runtime-worker substrate: 💤 Idle (0 peers), best: #199 (0xd879…28f9), finalized #199 (0xd879…28f9), ⬇ 0 ⬆ 0    
2021-09-27 23:12:48.622  INFO tokio-runtime-worker substrate: 💤 Idle (0 peers), best: #199 (0xd879…28f9), finalized #199 (0xd879…28f9), ⬇ 0 ⬆ 0    
2021-09-27 23:12:53.626  INFO tokio-runtime-worker substrate: 💤 Idle (0 peers), best: #199 (0xd879…28f9), finalized #199 (0xd879…28f9), ⬇ 0 ⬆ 0    
2021-09-27 23:12:58.632  INFO tokio-runtime-worker substrate: 💤 Idle (0 peers), best: #199 (0xd879…28f9), finalized #199 (0xd879…28f9), ⬇ 0 ⬆ 0    
2021-09-27 23:13:03.633  INFO tokio-runtime-worker substrate: 💤 Idle (0 peers), best: #199 (0xd879…28f9), finalized #199 (0xd879…28f9), ⬇ 0 ⬆ 0    
2021-09-27 23:13:08.634  INFO tokio-runtime-worker substrate: 💤 Idle (0 peers), best: #199 (0xd879…28f9), finalized #199 (0xd879…28f9), ⬇ 0 ⬆ 0    
2021-09-27 23:13:13.638  INFO tokio-runtime-worker substrate: 💤 Idle (0 peers), best: #199 (0xd879…28f9), finalized #199 (0xd879…28f9), ⬇ 0 ⬆ 0    
2021-09-27 23:13:18.641  INFO tokio-runtime-worker substrate: 💤 Idle (0 peers), best: #199 (0xd879…28f9), finalized #199 (0xd879…28f9), ⬇ 0 ⬆ 0    
2021-09-27 23:13:23.644  INFO tokio-runtime-worker substrate: 💤 Idle (0 peers), best: #199 (0xd879…28f9), finalized #199 (0xd879…28f9), ⬇ 0 ⬆ 0    
2021-09-27 23:13:24.937  WARN tokio-runtime-worker sc_rpc::author: Failed to submit extrinsic: Transaction pool error: Too low priority (383246202401776 > 383246202401776)

zjb0807 commented 3 years ago

https://github.com/paritytech/substrate/blob/c11a0bf8537abde0cf843c770f481366b964ca6f/frame/democracy/src/lib.rs#L1737 Democracy takes full block weight.

https://github.com/AcalaNetwork/Acala/blob/8e00ac60aec37d900c9106136e4fab9edca56cf3/runtime/mandala/src/lib.rs#L634

You can increase LaunchPeriod

xlc commented 3 years ago

Why democracy is taking the full weight even when there isn't proposals? Why something taking full block weight will cause issue?

zjb0807 commented 3 years ago

https://github.com/paritytech/substrate/pull/9890

Here is an empty block check that failed. https://github.com/paritytech/substrate/blob/c11a0bf8537abde0cf843c770f481366b964ca6f/client/consensus/manual-seal/src/seal_block.rs#L139-L141

xlc commented 3 years ago

Made another issue https://github.com/paritytech/substrate/issues/9892

AcalaNetwork / Acala

Block production in local development network stops after block #199 #1436