search

stacks-network / stacks-core

The Stacks blockchain implementation
https://docs.stacks.co
GNU General Public License v3.0
3.01k stars 673 forks source link

[Network] long delay between receiving a block validation request and getting a response #5366

Open jcnelson opened 1 month ago

jcnelson commented 1 month ago

The foundation nakamoto testnet miner sees this:

INFO [1729694751.965402] [testnet/stacks-node/src/nakamoto_node/sign_coordinator.rs:346] [miner.ef7cfd2050ac8b78acb983cd879d0dac12faaf30ab4e3ef400180de87b7809a7] SignCoordinator: beginning to watch for block signatures OR posted blocks., threshold: 1020
INFO [1729694756.336104] [testnet/stacks-node/src/nakamoto_node/sign_coordinator.rs:511] [miner.ef7cfd2050ac8b78acb983cd879d0dac12faaf30ab4e3ef400180de87b7809a7] SignCoordinator: Signature Added to block, block_signer_sighash: 62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178, signer_pubkey: 03157271cca37f5ebb4147aad30538fb4b33392ec1cc7161de5f815e29d5146041, signer_slot_id: 2, signature: 01bcc792e49d83b4d7896180dc79bbf19f97eace67d43beda226b3c9a3bb21ae62548e88fb9ec46d10953fafd4ecb173bfd70881229474a31f2558ef1e2953ce95, signer_weight: 1, total_weight_signed: 1, stacks_block_hash: 62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178, stacks_block_id: 77b4a778a0072faae62164caf40576cea7fb3828c9d897d423bbe8389703133e, server_version: stacks-signer 3.0.0.0.0 (release/3.0.0.0.0:dd1ebe64603f54dae48558a5d82d9bd885e97a01, release build, linux [x86_64])
INFO [1729694756.692839] [testnet/stacks-node/src/nakamoto_node/sign_coordinator.rs:511] [miner.ef7cfd2050ac8b78acb983cd879d0dac12faaf30ab4e3ef400180de87b7809a7] SignCoordinator: Signature Added to block, block_signer_sighash: 62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178, signer_pubkey: 03fbbbc6f26cb4105faf48008f8d7ef1900943a96fdbf63a95b94d7f2d87a4e9bb, signer_slot_id: 4, signature: 01b350643ec788493b0bf1770bddf75b4fc02ac1c7ef09537b2bd42c7029928d4b3311421103ca6c2c55442dd4c1052013ee4c3ae06e6baceb08625aa0665ff393, signer_weight: 1, total_weight_signed: 2, stacks_block_hash: 62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178, stacks_block_id: 77b4a778a0072faae62164caf40576cea7fb3828c9d897d423bbe8389703133e, server_version: stacks-signer 3.0.0.0.0 (release/3.0.0.0.0:dd1ebe6, release build, linux [x86_64])
INFO [1729694785.760468] [testnet/stacks-node/src/nakamoto_node/sign_coordinator.rs:511] [miner.ef7cfd2050ac8b78acb983cd879d0dac12faaf30ab4e3ef400180de87b7809a7] SignCoordinator: Signature Added to block, block_signer_sighash: 62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178, signer_pubkey: 026ddf4e4e144eac1aad4a215641feee86386a1bff3776f981430364d0ae89cd36, signer_slot_id: 0, signature: 00bd530db16aef6ee107a3d04a471b787618d958191fcc8d8e13ff59f7c769a3aa74b5865fe630ec1dad8bbcc446c0a211606f14d95e3e3fbb82f18ca3b852cb64, signer_weight: 485, total_weight_signed: 487, stacks_block_hash: 62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178, stacks_block_id: 77b4a778a0072faae62164caf40576cea7fb3828c9d897d423bbe8389703133e, server_version: stacks-signer 3.0.0.0.0 (release/3.0.0.0.0:dd1ebe6, release build, linux [x86_64])
INFO [1729694788.199587] [testnet/stacks-node/src/nakamoto_node/sign_coordinator.rs:511] [miner.ef7cfd2050ac8b78acb983cd879d0dac12faaf30ab4e3ef400180de87b7809a7] SignCoordinator: Signature Added to block, block_signer_sighash: 62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178, signer_pubkey: 02e77164ae21938de2058eb349353f0ae2ac3b76e1a69a03ea94c6fc352164c467, signer_slot_id: 1, signature: 008b9f244f4f92588bdde7a252779b6bf08c4f0fdfa54c19743b2805ee494d40e10bb3d7c93e17ac9a52c23bd37979f55cf2274b289be47912f9e9b73d2fc8c29e, signer_weight: 485, total_weight_signed: 972, stacks_block_hash: 62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178, stacks_block_id: 77b4a778a0072faae62164caf40576cea7fb3828c9d897d423bbe8389703133e, server_version: stacks-signer 3.0.0.0.0 (release/3.0.0.0.0:dd1ebe6, release build, linux [x86_64])
INFO [1729694788.199752] [testnet/stacks-node/src/nakamoto_node/sign_coordinator.rs:511] [miner.ef7cfd2050ac8b78acb983cd879d0dac12faaf30ab4e3ef400180de87b7809a7] SignCoordinator: Signature Added to block, block_signer_sighash: 62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178, signer_pubkey: 0343b909be3379621d5e68a883e6c74c2222666f4bf6cc38fade2c9d1c57d92213, signer_slot_id: 3, signature: 010a760eb984e3388b3ba03b70a7923cba7e3ded06b4a1de71d7a1bfed723c63c51e3de0f6359b1f9803a91773221d764df160e892d81b5ecfd656f3964384516a, signer_weight: 485, total_weight_signed: 1457, stacks_block_hash: 62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178, stacks_block_id: 77b4a778a0072faae62164caf40576cea7fb3828c9d897d423bbe8389703133e, server_version: stacks-signer 3.0.0.0.0 (release/3.0.0.0.0:dd1ebe6, release build, linux [x86_64])
INFO [1729694788.199787] [testnet/stacks-node/src/nakamoto_node/sign_coordinator.rs:594] [miner.ef7cfd2050ac8b78acb983cd879d0dac12faaf30ab4e3ef400180de87b7809a7] SignCoordinator: Received enough signatures. Continuing., stacks_block_hash: 62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178, stacks_block_id: 77b4a778a0072faae62164caf40576cea7fb3828c9d897d423bbe8389703133e
INFO [1729694788.569665] [testnet/stacks-node/src/nakamoto_node/miner.rs:404] [miner.ef7cfd2050ac8b78acb983cd879d0dac12faaf30ab4e3ef400180de87b7809a7] Miner: Block signed by signer set and broadcasted, signer_sighash: 62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178, stacks_block_hash: 62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178, stacks_block_id: 77b4a778a0072faae62164caf40576cea7fb3828c9d897d423bbe8389703133e, block_height: 38268, consensus_hash: d3f81e962f1c9bbc1fe2fe3d434d2b19af0010e0

The hiro nodes and signers see this:

10/23/2024, 02:53:09.682 PM testnet-seed-nakamoto-0 - Handle incoming Nakamoto block d3f81e962f1c9bbc1fe2fe3d434d2b19af0010e0/62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178 obtained via Downloaded
10/23/2024, 02:46:56.231 PM testnet-stacks-blockchain-nakamoto-3-0 - Relayer: Submitted block-commit
10/23/2024, 02:46:35.321 PM testnet-stacks-blockchain-nakamoto-3-0 - Handle incoming Nakamoto block d3f81e962f1c9bbc1fe2fe3d434d2b19af0010e0/62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178 obtained via Uploaded
10/23/2024, 02:46:34.463 PM testnet-stacks-blockchain-nakamoto-3-0 - Handle incoming Nakamoto block d3f81e962f1c9bbc1fe2fe3d434d2b19af0010e0/62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178 obtained via Uploaded
10/23/2024, 02:46:34.428 PM testnet-stacks-blockchain-nakamoto-3-0 - Handle incoming Nakamoto block d3f81e962f1c9bbc1fe2fe3d434d2b19af0010e0/62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178 obtained via Uploaded
10/23/2024, 02:46:33.235 PM testnet-seed-nakamoto-0 - Advanced to new tip! d3f81e962f1c9bbc1fe2fe3d434d2b19af0010e0/62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178
10/23/2024, 02:46:33.128 PM testnet-pool-nakamoto-0-0 - Handle incoming Nakamoto block d3f81e962f1c9bbc1fe2fe3d434d2b19af0010e0/62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178 obtained via Pushed
10/23/2024, 02:46:31.899 PM testnet-stacks-blockchain-nakamoto-3-0 - Cycle #4 Signer #1: Got block pushed message
10/23/2024, 02:46:31.846 PM testnet-stacks-blockchain-nakamoto-3-0 - Cycle #4 Signer #0: Got block pushed message
10/23/2024, 02:46:31.814 PM testnet-seed-nakamoto-0 - Handle incoming Nakamoto block d3f81e962f1c9bbc1fe2fe3d434d2b19af0010e0/62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178 obtained via Pushed
10/23/2024, 02:46:31.784 PM testnet-stacks-blockchain-nakamoto-3-0 - Cycle #4 Signer #3: Got block pushed message
10/23/2024, 02:46:31.631 PM testnet-stacks-blockchain-nakamoto-3-0 - Handle incoming Nakamoto block d3f81e962f1c9bbc1fe2fe3d434d2b19af0010e0/62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178 obtained via Pushed
10/23/2024, 02:46:31.569 PM testnet-stacks-blockchain-nakamoto-3-0 - Handle incoming Nakamoto block d3f81e962f1c9bbc1fe2fe3d434d2b19af0010e0/62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178 obtained via Pushed
10/23/2024, 02:46:31.511 PM testnet-stacks-blockchain-nakamoto-3-0 - Handle incoming Nakamoto block d3f81e962f1c9bbc1fe2fe3d434d2b19af0010e0/62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178 obtained via Pushed
10/23/2024, 02:46:31.457 PM testnet-stacks-blockchain-nakamoto-3-0 - Handle incoming Nakamoto block d3f81e962f1c9bbc1fe2fe3d434d2b19af0010e0/62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178 obtained via Pushed
10/23/2024, 02:46:30.924 PM testnet-pool-nakamoto-0-0 - Advanced to new tip! d3f81e962f1c9bbc1fe2fe3d434d2b19af0010e0/62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178
10/23/2024, 02:46:30.825 PM testnet-pool-nakamoto-0-0 - Handle incoming Nakamoto block d3f81e962f1c9bbc1fe2fe3d434d2b19af0010e0/62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178 obtained via Pushed
10/23/2024, 02:46:30.473 PM testnet-seed-nakamoto-0 - Handle incoming Nakamoto block d3f81e962f1c9bbc1fe2fe3d434d2b19af0010e0/62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178 obtained via Pushed
10/23/2024, 02:46:29.966 PM testnet-seed-nakamoto-0 - Handle incoming Nakamoto block d3f81e962f1c9bbc1fe2fe3d434d2b19af0010e0/62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178 obtained via Pushed
10/23/2024, 02:46:29.906 PM testnet-seed-nakamoto-0 - Handle incoming Nakamoto block d3f81e962f1c9bbc1fe2fe3d434d2b19af0010e0/62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178 obtained via Pushed
10/23/2024, 02:46:29.821 PM testnet-seed-nakamoto-0 - Stored incoming Nakamoto block d3f81e962f1c9bbc1fe2fe3d434d2b19af0010e0/62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178
10/23/2024, 02:46:29.602 PM testnet-seed-nakamoto-0 - Handle incoming Nakamoto block d3f81e962f1c9bbc1fe2fe3d434d2b19af0010e0/62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178 obtained via Pushed
10/23/2024, 02:46:29.266 PM testnet-pool-nakamoto-0-0 - Handle incoming Nakamoto block d3f81e962f1c9bbc1fe2fe3d434d2b19af0010e0/62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178 obtained via Pushed
10/23/2024, 02:46:29.257 PM testnet-stacks-blockchain-nakamoto-3-0 - Advanced to new tip! d3f81e962f1c9bbc1fe2fe3d434d2b19af0010e0/62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178
10/23/2024, 02:46:29.209 PM testnet-pool-nakamoto-0-0 - Stored incoming Nakamoto block d3f81e962f1c9bbc1fe2fe3d434d2b19af0010e0/62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178
10/23/2024, 02:46:29.036 PM testnet-pool-nakamoto-0-0 - Handle incoming Nakamoto block d3f81e962f1c9bbc1fe2fe3d434d2b19af0010e0/62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178 obtained via Pushed
10/23/2024, 02:46:27.150 PM testnet-stacks-blockchain-nakamoto-3-0 - Handle incoming Nakamoto block d3f81e962f1c9bbc1fe2fe3d434d2b19af0010e0/62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178 obtained via Uploaded
10/23/2024, 02:46:27.116 PM testnet-stacks-blockchain-nakamoto-3-0 - Handle incoming Nakamoto block d3f81e962f1c9bbc1fe2fe3d434d2b19af0010e0/62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178 obtained via Uploaded
10/23/2024, 02:46:27.079 PM testnet-stacks-blockchain-nakamoto-3-0 - Stored incoming Nakamoto block d3f81e962f1c9bbc1fe2fe3d434d2b19af0010e0/62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178
10/23/2024, 02:46:26.926 PM testnet-stacks-blockchain-nakamoto-3-0 - Handle incoming Nakamoto block d3f81e962f1c9bbc1fe2fe3d434d2b19af0010e0/62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178 obtained via Uploaded
10/23/2024, 02:46:23.627 PM testnet-stacks-blockchain-nakamoto-3-0 - Cycle #4 Signer #3: Broadcasting a block response to stacks node: Accepted(BlockAccepted { signer_signature_hash: 62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178, signature: 010a760eb984e3388b3ba03b70a7923cba7e3ded06b4a1de71d7a1bfed723c63c51e3de0f6359b1f9803a91773221d764df160e892d81b5ecfd656f3964384516a, metadata: SignerMessageMetadata { server_version: "stacks-signer 3.0.0.0.0 (release/3.0.0.0.0:dd1ebe6, release build, linux [x86_64])" } })
10/23/2024, 02:46:23.587 PM testnet-stacks-blockchain-nakamoto-3-0 - Cycle #4 Signer #1: Broadcasting a block response to stacks node: Accepted(BlockAccepted { signer_signature_hash: 62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178, signature: 008b9f244f4f92588bdde7a252779b6bf08c4f0fdfa54c19743b2805ee494d40e10bb3d7c93e17ac9a52c23bd37979f55cf2274b289be47912f9e9b73d2fc8c29e, metadata: SignerMessageMetadata { server_version: "stacks-signer 3.0.0.0.0 (release/3.0.0.0.0:dd1ebe6, release build, linux [x86_64])" } })
10/23/2024, 02:46:23.579 PM testnet-stacks-blockchain-nakamoto-3-0 - Cycle #4 Signer #0: Broadcasting a block response to stacks node: Accepted(BlockAccepted { signer_signature_hash: 62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178, signature: 00bd530db16aef6ee107a3d04a471b787618d958191fcc8d8e13ff59f7c769a3aa74b5865fe630ec1dad8bbcc446c0a211606f14d95e3e3fbb82f18ca3b852cb64, metadata: SignerMessageMetadata { server_version: "stacks-signer 3.0.0.0.0 (release/3.0.0.0.0:dd1ebe6, release build, linux [x86_64])" } })
10/23/2024, 02:46:23.529 PM testnet-stacks-blockchain-nakamoto-3-0 - Cycle #4 Signer #3: Received a block validate response: Ok(BlockValidateOk { signer_signature_hash: 62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178, cost: ExecutionCost { write_length: 2676198, write_count: 1227, read_length: 1099372, read_count: 1207, runtime: 76922159 }, size: 498559 })
10/23/2024, 02:46:23.516 PM testnet-stacks-blockchain-nakamoto-3-0 - Cycle #4 Signer #1: Received a block validate response: Ok(BlockValidateOk { signer_signature_hash: 62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178, cost: ExecutionCost { write_length: 2676198, write_count: 1227, read_length: 1099372, read_count: 1207, runtime: 76922159 }, size: 498559 })
10/23/2024, 02:46:23.502 PM testnet-stacks-blockchain-nakamoto-3-0 - Cycle #4 Signer #0: Received a block validate response: Ok(BlockValidateOk { signer_signature_hash: 62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178, cost: ExecutionCost { write_length: 2676198, write_count: 1227, read_length: 1099372, read_count: 1207, runtime: 76922159 }, size: 498559 })
10/23/2024, 02:46:23.494 PM testnet-stacks-blockchain-nakamoto-3-0 - Participant: validated anchored block
10/23/2024, 02:46:22.486 PM testnet-stacks-blockchain-nakamoto-3-0 - Received block proposal request
10/23/2024, 02:46:21.392 PM testnet-stacks-blockchain-nakamoto-3-0 - Received block proposal request
10/23/2024, 02:46:21.370 PM testnet-stacks-blockchain-nakamoto-3-0 - Received block proposal request
10/23/2024, 02:45:53.205 PM testnet-stacks-blockchain-nakamoto-3-0 - Cycle #4 Signer #1: received a block proposal for a new block. Submit block for validation. 
10/23/2024, 02:45:53.141 PM testnet-stacks-blockchain-nakamoto-3-0 - Cycle #4 Signer #0: received a block proposal for a new block. Submit block for validation. 
10/23/2024, 02:45:53.072 PM testnet-stacks-blockchain-nakamoto-3-0 - Cycle #4 Signer #3: received a block proposal for a new block. Submit block for validation.

Why is there such a long delay between the signer submitting a block proposal and the node receiving it?

jcnelson commented 1 month ago

Seems the root cause is that these nodes are just super-backlogged with HTTP requests:

10/23/2024, 02:46:23.627 PM testnet-stacks-blockchain-nakamoto-3-0 - Cycle #4 Signer #3: Broadcasting a block response to stacks node: Accepted(BlockAccepted { signer_signature_hash: 62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178, signature: 010a760eb984e3388b3ba03b70a7923cba7e3ded06b4a1de71d7a1bfed723c63c51e3de0f6359b1f9803a91773221d764df160e892d81b5ecfd656f3964384516a, metadata: SignerMessageMetadata { server_version: "stacks-signer 3.0.0.0.0 (release/3.0.0.0.0:dd1ebe6, release build, linux [x86_64])" } }) -  - 
10/23/2024, 02:46:23.604 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 156
10/23/2024, 02:46:23.587 PM testnet-stacks-blockchain-nakamoto-3-0 - Cycle #4 Signer #1: Broadcasting a block response to stacks node: Accepted(BlockAccepted { signer_signature_hash: 62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178, signature: 008b9f244f4f92588bdde7a252779b6bf08c4f0fdfa54c19743b2805ee494d40e10bb3d7c93e17ac9a52c23bd37979f55cf2274b289be47912f9e9b73d2fc8c29e, metadata: SignerMessageMetadata { server_version: "stacks-signer 3.0.0.0.0 (release/3.0.0.0.0:dd1ebe6, release build, linux [x86_64])" } }) -  - 
10/23/2024, 02:46:23.579 PM testnet-stacks-blockchain-nakamoto-3-0 - Cycle #4 Signer #0: Broadcasting a block response to stacks node: Accepted(BlockAccepted { signer_signature_hash: 62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178, signature: 00bd530db16aef6ee107a3d04a471b787618d958191fcc8d8e13ff59f7c769a3aa74b5865fe630ec1dad8bbcc446c0a211606f14d95e3e3fbb82f18ca3b852cb64, metadata: SignerMessageMetadata { server_version: "stacks-signer 3.0.0.0.0 (release/3.0.0.0.0:dd1ebe6, release build, linux [x86_64])" } }) -  - 
10/23/2024, 02:46:23.529 PM testnet-stacks-blockchain-nakamoto-3-0 - Cycle #4 Signer #3: Received a block validate response: Ok(BlockValidateOk { signer_signature_hash: 62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178, cost: ExecutionCost { write_length: 2676198, write_count: 1227, read_length: 1099372, read_count: 1207, runtime: 76922159 }, size: 498559 }) -  - 
10/23/2024, 02:46:23.516 PM testnet-stacks-blockchain-nakamoto-3-0 - Cycle #4 Signer #1: Received a block validate response: Ok(BlockValidateOk { signer_signature_hash: 62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178, cost: ExecutionCost { write_length: 2676198, write_count: 1227, read_length: 1099372, read_count: 1207, runtime: 76922159 }, size: 498559 }) -  - 
10/23/2024, 02:46:23.502 PM testnet-stacks-blockchain-nakamoto-3-0 - Cycle #4 Signer #0: Received a block validate response: Ok(BlockValidateOk { signer_signature_hash: 62807f1655cd545c79edd1fb65b07b147c083dbe9ed1072b6cd56d51078b4178, cost: ExecutionCost { write_length: 2676198, write_count: 1227, read_length: 1099372, read_count: 1207, runtime: 76922159 }, size: 498559 }) -  - 
10/23/2024, 02:46:23.494 PM testnet-stacks-blockchain-nakamoto-3-0 - Participant: validated anchored block -  - 
10/23/2024, 02:46:23.447 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 73
10/23/2024, 02:46:23.372 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 145
10/23/2024, 02:46:22.814 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 130
10/23/2024, 02:46:22.684 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/pox - 122
10/23/2024, 02:46:22.560 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 72
10/23/2024, 02:46:22.486 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v3/block_proposal - 0
10/23/2024, 02:46:22.486 PM testnet-stacks-blockchain-nakamoto-3-0 - Received block proposal request -  - 
10/23/2024, 02:46:22.111 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 110
10/23/2024, 02:46:22.000 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 0
10/23/2024, 02:46:22.000 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 150
10/23/2024, 02:46:21.536 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/accounts/ST2D1M978SCE52GAV07VXSRC9DQBP69X5WHX0DHN5?proof=0 - 46
10/23/2024, 02:46:21.489 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/accounts/ST484MS3VACPAZ90WHC21XQ7T6XANCV341HJYE0W?proof=0 - 47
10/23/2024, 02:46:21.441 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/accounts/ST3MNK12DGQF7JN4Q0STK6926VWE5MN21KJ4EGV0E?proof=0 - 48
10/23/2024, 02:46:21.392 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v3/block_proposal - 0
10/23/2024, 02:46:21.392 PM testnet-stacks-blockchain-nakamoto-3-0 - Received block proposal request -  - 
10/23/2024, 02:46:21.373 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/info - 0
10/23/2024, 02:46:21.373 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v3/block_proposal - 2
10/23/2024, 02:46:21.370 PM testnet-stacks-blockchain-nakamoto-3-0 - Received block proposal request -  - 
10/23/2024, 02:46:20.965 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 133
10/23/2024, 02:46:20.831 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v3/tenures/info - 0
10/23/2024, 02:46:20.830 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 129
10/23/2024, 02:46:20.700 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 136
10/23/2024, 02:46:20.255 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/pox - 117
10/23/2024, 02:46:20.137 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 108
10/23/2024, 02:46:19.750 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 138
10/23/2024, 02:46:19.224 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 134
10/23/2024, 02:46:19.090 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 0
10/23/2024, 02:46:19.089 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 68
10/23/2024, 02:46:19.020 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 69
10/23/2024, 02:46:18.593 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 106
10/23/2024, 02:46:18.486 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 65
10/23/2024, 02:46:18.420 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/accounts/ST2D1M978SCE52GAV07VXSRC9DQBP69X5WHX0DHN5?proof=0 - 47
10/23/2024, 02:46:18.372 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/accounts/ST484MS3VACPAZ90WHC21XQ7T6XANCV341HJYE0W?proof=0 - 48
10/23/2024, 02:46:18.323 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/accounts/ST3MNK12DGQF7JN4Q0STK6926VWE5MN21KJ4EGV0E?proof=0 - 48
10/23/2024, 02:46:18.271 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 75
10/23/2024, 02:46:17.901 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 65
10/23/2024, 02:46:17.834 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/pox - 116
10/23/2024, 02:46:17.717 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 66
10/23/2024, 02:46:17.650 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 103
10/23/2024, 02:46:17.546 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 128
10/23/2024, 02:46:17.016 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 115
10/23/2024, 02:46:16.900 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 142
10/23/2024, 02:46:16.758 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 128
10/23/2024, 02:46:16.629 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 107
10/23/2024, 02:46:16.520 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 131
10/23/2024, 02:46:16.045 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/info - 0
10/23/2024, 02:46:16.045 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v3/tenures/info - 0
10/23/2024, 02:46:16.044 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 130
10/23/2024, 02:46:15.913 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 130
10/23/2024, 02:46:15.782 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/accounts/ST1114TBQYGNPGFAVXKWBKZAHP0X7ZGX9K6XYYE4F?proof=0 - 51
10/23/2024, 02:46:15.729 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/accounts/ST2Q6124HQFKVKPJSS5J6156BJR74FD6EC1297HJ1?proof=0 - 47
10/23/2024, 02:46:15.681 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 114
10/23/2024, 02:46:15.566 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/accounts/ST20Q2N56E1NBWE37R4VGSF89X4HHTB3GSMD8GKYW?proof=0 - 47
10/23/2024, 02:46:15.518 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 69
10/23/2024, 02:46:15.448 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 0
10/23/2024, 02:46:15.446 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 71
10/23/2024, 02:46:15.025 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/contracts/interface/ST000000000000000000002AMW42H/pox-4 - 48
10/23/2024, 02:46:14.976 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/contracts/interface/ST000000000000000000002AMW42H/pox-4 - 51
10/23/2024, 02:46:14.924 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/pox - 115
10/23/2024, 02:46:14.808 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 102
10/23/2024, 02:46:14.705 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/contracts/interface/ST000000000000000000002AMW42H/pox-4 - 47
10/23/2024, 02:46:14.657 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 129
10/23/2024, 02:46:14.527 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/accounts/ST2D1M978SCE52GAV07VXSRC9DQBP69X5WHX0DHN5?proof=0 - 46
10/23/2024, 02:46:14.479 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/accounts/ST484MS3VACPAZ90WHC21XQ7T6XANCV341HJYE0W?proof=0 - 46
10/23/2024, 02:46:14.432 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/accounts/ST3MNK12DGQF7JN4Q0STK6926VWE5MN21KJ4EGV0E?proof=0 - 47
10/23/2024, 02:46:14.047 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/pox - 120
10/23/2024, 02:46:13.927 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/pox - 126
10/23/2024, 02:46:13.799 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/pox - 127
10/23/2024, 02:46:13.671 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 71
10/23/2024, 02:46:13.599 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 135
10/23/2024, 02:46:13.074 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 103
10/23/2024, 02:46:12.970 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 148
10/23/2024, 02:46:12.817 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/accounts/ST1114TBQYGNPGFAVXKWBKZAHP0X7ZGX9K6XYYE4F?proof=0 - 53
10/23/2024, 02:46:12.763 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/accounts/ST2Q6124HQFKVKPJSS5J6156BJR74FD6EC1297HJ1?proof=0 - 54
10/23/2024, 02:46:12.328 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 100
10/23/2024, 02:46:12.226 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 64
10/23/2024, 02:46:12.161 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 127
10/23/2024, 02:46:12.034 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 69
10/23/2024, 02:46:11.963 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 113
10/23/2024, 02:46:11.849 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/pox - 127
10/23/2024, 02:46:11.721 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 0
10/23/2024, 02:46:11.720 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/accounts/ST20Q2N56E1NBWE37R4VGSF89X4HHTB3GSMD8GKYW?proof=0 - 49
10/23/2024, 02:46:11.173 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 67
10/23/2024, 02:46:11.105 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v3/tenures/info - 0
10/23/2024, 02:46:11.104 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/accounts/ST484MS3VACPAZ90WHC21XQ7T6XANCV341HJYE0W?proof=0 - 48
10/23/2024, 02:46:11.055 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/accounts/ST2D1M978SCE52GAV07VXSRC9DQBP69X5WHX0DHN5?proof=0 - 47
10/23/2024, 02:46:11.007 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/accounts/ST3MNK12DGQF7JN4Q0STK6926VWE5MN21KJ4EGV0E?proof=0 - 48
10/23/2024, 02:46:10.958 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/pox - 119
10/23/2024, 02:46:10.480 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 134
10/23/2024, 02:46:10.344 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/info - 0
10/23/2024, 02:46:10.343 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 143
10/23/2024, 02:46:10.198 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 69
10/23/2024, 02:46:10.128 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 156
10/23/2024, 02:46:09.971 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 136
10/23/2024, 02:46:09.438 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 135
10/23/2024, 02:46:09.303 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 66
10/23/2024, 02:46:09.235 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 114
10/23/2024, 02:46:09.120 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/pox - 127
10/23/2024, 02:46:08.991 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 136
10/23/2024, 02:46:08.507 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v3/tenures/info - 0
10/23/2024, 02:46:08.507 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 106
10/23/2024, 02:46:08.400 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 137
10/23/2024, 02:46:08.261 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 110
10/23/2024, 02:46:08.150 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 0
10/23/2024, 02:46:07.723 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 67
10/23/2024, 02:46:07.655 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 129
10/23/2024, 02:46:07.523 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/accounts/ST2D1M978SCE52GAV07VXSRC9DQBP69X5WHX0DHN5?proof=0 - 47
10/23/2024, 02:46:07.476 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/accounts/ST484MS3VACPAZ90WHC21XQ7T6XANCV341HJYE0W?proof=0 - 47
10/23/2024, 02:46:07.427 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/accounts/ST3MNK12DGQF7JN4Q0STK6926VWE5MN21KJ4EGV0E?proof=0 - 47
10/23/2024, 02:46:07.075 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 118
10/23/2024, 02:46:06.955 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 112
10/23/2024, 02:46:06.842 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 79
10/23/2024, 02:46:06.762 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 67
10/23/2024, 02:46:06.247 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 73
10/23/2024, 02:46:06.173 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/pox - 133
10/23/2024, 02:46:06.039 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 133
10/23/2024, 02:46:05.622 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v3/tenures/info - 0
10/23/2024, 02:46:05.621 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 126
10/23/2024, 02:46:05.494 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 101
10/23/2024, 02:46:05.286 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v3/tenures/0ca0d2f26b6a1d4716884be8f3140cac4752385c5e60314bcb74fca6d38bd788 - 17
10/23/2024, 02:46:04.954 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 130
10/23/2024, 02:46:04.824 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 125
10/23/2024, 02:46:04.697 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 0
10/23/2024, 02:46:04.696 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 67
10/23/2024, 02:46:04.629 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/info - 0
10/23/2024, 02:46:04.226 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 114
10/23/2024, 02:46:04.111 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/accounts/ST2D1M978SCE52GAV07VXSRC9DQBP69X5WHX0DHN5?proof=0 - 47
10/23/2024, 02:46:04.063 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/accounts/ST484MS3VACPAZ90WHC21XQ7T6XANCV341HJYE0W?proof=0 - 65
10/23/2024, 02:46:03.997 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/accounts/ST3MNK12DGQF7JN4Q0STK6926VWE5MN21KJ4EGV0E?proof=0 - 50
10/23/2024, 02:46:03.621 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/pox - 118
10/23/2024, 02:46:03.502 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 89
10/23/2024, 02:46:03.147 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 68
10/23/2024, 02:46:03.078 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 105
10/23/2024, 02:46:02.973 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 114
10/23/2024, 02:46:01.856 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v3/blocks/c5807470c224c9fe41527bf7f3f90852760cfa9446fdb89e039240b7619864c4 - 0
10/23/2024, 02:46:01.470 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 0
10/23/2024, 02:46:01.168 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/pox - 116
10/23/2024, 02:46:01.051 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/accounts/ST2D1M978SCE52GAV07VXSRC9DQBP69X5WHX0DHN5?proof=0 - 64
10/23/2024, 02:46:00.986 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/accounts/ST484MS3VACPAZ90WHC21XQ7T6XANCV341HJYE0W?proof=0 - 53
10/23/2024, 02:46:00.932 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/accounts/ST3MNK12DGQF7JN4Q0STK6926VWE5MN21KJ4EGV0E?proof=0 - 52
10/23/2024, 02:46:00.496 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 69
10/23/2024, 02:46:00.426 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 68
10/23/2024, 02:45:59.926 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 109
10/23/2024, 02:45:59.534 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 74
10/23/2024, 02:45:59.458 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 74
10/23/2024, 02:45:59.383 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 74
10/23/2024, 02:45:58.948 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v3/blocks/ef7cfd2050ac8b78acb983cd879d0dac12faaf30ab4e3ef400180de87b7809a7 - 0
10/23/2024, 02:45:58.551 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 0
10/23/2024, 02:45:58.551 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/pox - 118
10/23/2024, 02:45:58.431 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/info - 0
10/23/2024, 02:45:58.016 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/accounts/ST484MS3VACPAZ90WHC21XQ7T6XANCV341HJYE0W?proof=0 - 55
10/23/2024, 02:45:57.960 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/accounts/ST2D1M978SCE52GAV07VXSRC9DQBP69X5WHX0DHN5?proof=0 - 55
10/23/2024, 02:45:57.904 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/accounts/ST3MNK12DGQF7JN4Q0STK6926VWE5MN21KJ4EGV0E?proof=0 - 50
10/23/2024, 02:45:57.773 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v3/tenures/803a02e9d50f7dbdbf71c338dd5cc2a881623e9ba39a57c12b1aa6141136d586 - 19
10/23/2024, 02:45:57.752 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 149
10/23/2024, 02:45:57.602 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 138
10/23/2024, 02:45:57.463 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 130
10/23/2024, 02:45:56.884 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 128
10/23/2024, 02:45:56.754 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 145
10/23/2024, 02:45:56.608 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 157
10/23/2024, 02:45:56.450 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 127
10/23/2024, 02:45:56.321 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 65
10/23/2024, 02:45:56.255 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/accounts/ST2Q6124HQFKVKPJSS5J6156BJR74FD6EC1297HJ1?proof=0 - 49
10/23/2024, 02:45:56.205 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 103
10/23/2024, 02:45:55.741 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 136
10/23/2024, 02:45:55.605 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/pox - 119
10/23/2024, 02:45:55.485 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/accounts/ST1114TBQYGNPGFAVXKWBKZAHP0X7ZGX9K6XYYE4F?proof=0 - 57
10/23/2024, 02:45:55.426 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 127
10/23/2024, 02:45:55.298 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/accounts/ST20Q2N56E1NBWE37R4VGSF89X4HHTB3GSMD8GKYW?proof=0 - 48
10/23/2024, 02:45:55.249 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 0
10/23/2024, 02:45:55.249 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/contracts/interface/ST000000000000000000002AMW42H/pox-4 - 49
10/23/2024, 02:45:55.199 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v3/tenures/info - 0
10/23/2024, 02:45:55.198 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 156
10/23/2024, 02:45:54.673 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 105
10/23/2024, 02:45:54.463 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v3/tenures/0ca0d2f26b6a1d4716884be8f3140cac4752385c5e60314bcb74fca6d38bd788 - 16
10/23/2024, 02:45:54.445 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 65
10/23/2024, 02:45:54.379 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/contracts/interface/ST000000000000000000002AMW42H/pox-4 - 49
10/23/2024, 02:45:54.328 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 74
10/23/2024, 02:45:54.254 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/contracts/interface/ST000000000000000000002AMW42H/pox-4 - 49
10/23/2024, 02:45:54.203 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/accounts/ST2D1M978SCE52GAV07VXSRC9DQBP69X5WHX0DHN5?proof=0 - 53
10/23/2024, 02:45:54.149 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/accounts/ST484MS3VACPAZ90WHC21XQ7T6XANCV341HJYE0W?proof=0 - 49
10/23/2024, 02:45:54.099 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/accounts/ST3MNK12DGQF7JN4Q0STK6926VWE5MN21KJ4EGV0E?proof=0 - 49
10/23/2024, 02:45:54.047 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 111
10/23/2024, 02:45:53.934 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/pox - 122
10/23/2024, 02:45:53.439 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 150
10/23/2024, 02:45:53.288 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/transactions - 73
10/23/2024, 02:45:53.214 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/pox - 141
10/23/2024, 02:45:53.205 PM testnet-stacks-blockchain-nakamoto-3-0 - Cycle #4 Signer #1: received a block proposal for a new block. Submit block for validation.  -  - 
10/23/2024, 02:45:53.141 PM testnet-stacks-blockchain-nakamoto-3-0 - Cycle #4 Signer #0: received a block proposal for a new block. Submit block for validation.  -  - 
10/23/2024, 02:45:53.072 PM testnet-stacks-blockchain-nakamoto-3-0 - Handled StacksHTTPRequest - /v2/pox - 144
10/23/2024, 02:45:53.072 PM testnet-stacks-blockchain-nakamoto-3-0 - Cycle #4 Signer #3: received a block proposal for a new block. Submit block for validation.  -  -

Looks like we have two candidates for network RPC optimization: /v2/transactions and /v2/pox.

hstove commented 1 week ago

I think it would be good to get more clear on what exactly this ticket aims to solve. It's certainly fine for this ticket to be "we have a problem, we should figure out what we want to do about it", but I think we should figure out what that work is and split that into individual tickets.

IMO there are a number of known improvements here:

[signer] Track 429 block submission failures in the database and resubmit them following a block validation response - this is a big one improve debug logging performance ignore old block validation requests

It also appears that you've identified two potential big wins:

Looks like we have two candidates for network RPC optimization: /v2/transactions and /v2/pox.

I'd note that the /v2/transactions issue came from Hiro's infra proxying /v2/transactions to all of their nodes, and I believe @CharlieC3 changed that so that the signer node doesn't get those proxied anymore.

CharlieC3 commented 1 week ago

I'd note that the /v2/transactions issue came from Hiro's infra proxying /v2/transactions to all of their nodes, and I believe @CharlieC3 changed that so that the signer node doesn't get those proxied anymore.

Maybe the signals got mixed in the lines somewhere, but it's not quite this. We only mirror /v2/transactions POSTs to 1 additional stacks node -- the one serving events to the Stacks Blockchain API. And the mirror happens asynchronously, so any performance issues on this endpoint are independent of the proxying that happens beforehand.

hstove commented 1 week ago

Ah, I also found the Slack thread where we discussed this, and there was an issue where the stacking script was trying to re-broadcast a stack-extend every ~20 seconds (we've fixed that issue)