search

AleoNet / snarkOS

A Decentralized Operating System for ZK Applications
http://snarkos.org
Apache License 2.0
4.1k stars 2.57k forks source link

[Bug] Client nodes do not work properly on testnet-beta #3284

Open elderhammer opened 1 month ago

elderhammer commented 1 month ago

🐛 Bug Report

The code on the mainnet-staging branch prohibits client nodes from syncing blocks, see here: https://github.com/AleoNet/snarkOS/blob/34abc38edcad2c7c3ec2d5389d5d7dabd06cfa1b/node/cdn/src/blocks.rs#L111 Also, the cdn address looks outdated: https://github.com/AleoNet/snarkOS/blob/34abc38edcad2c7c3ec2d5389d5d7dabd06cfa1b/cli/src/commands/start.rs#L139 If you try to synchronize blocks from this address, you will see the following error log::

WARN snarkos_node_cdn::blocks: Failed to deserialize blocks 0 to 50 - Invalid block metadata: {"network":3,"round":0,"height":0,"cumulative_weight":0,"cumulative_proof_target":0,"coinbase_target":4294967295,"proof_target":33554432,"last_coinbase_target":4294967295,"last_coinbase_timestamp":1696118400,"timestamp":1696118400} - retrying (1 attempt(s) so far)

WARN snarkos_node_cdn::blocks: Failed to deserialize blocks 50 to 100 - Invalid block metadata: {"network":3,"round":124,"height":50,"cumulative_weight":0,"cumulative_proof_target":0,"coinbase_target":4294967295,"proof_target":33554432,"last_coinbase_target":4294967295,"last_coinbase_timestamp":1696118400,"timestamp":1697744745} - retrying (1 attempt(s) so far)

Therefore, it is not possible to develop and test RPC-related services on testnet-beta.

Steps to Reproduce

Expected Behavior

Modify the code to support testnet-beta for block synchronization and change the cdn address to the one corresponding to the current test network.

Your Environment

snarkOS: 34abc38edcad2c7c3ec2d5389d5d7dabd06cfa1b

vicsn commented 1 month ago

testnet beta is currently running https://github.com/AleoNet/snarkOS/releases/tag/testnet-beta-v0.2.0

elderhammer commented 1 month ago

testnet beta is currently running https://github.com/AleoNet/snarkOS/releases/tag/testnet-beta-v0.2.0

In the source code of this tag, the CDN address is still https://s3.us-west-1.amazonaws.com/testnet3.blocks/phase3, and NETWORK_ID is still set to 0. In other words, the client node cannot be run correctly using this source code.

vicsn commented 1 month ago

I see, thank you for reporting. Does the snarkos start --nocdn flag resolve your issue whil CDN is getting spun up?

elderhammer commented 1 month ago

I see, thank you for reporting. Does the snarkos start --nocdn flag resolve your issue whil CDN is getting spun up?

No. The log is as follows:

2024-06-05T13:22:24.563200Z  WARN snarkos_node_cdn::blocks: The network (1) is not supported
2024-06-05T13:22:24.563494Z DEBUG snarkos_node_rest: REST rate limit per IP - 10 RPS
2024-06-05T13:22:24.766388Z DEBUG snarkos_node_router::heartbeat: No connected peers
2024-06-05T13:22:25.170755Z DEBUG snarkos_node_router::handshake: Connecting to 35.200.149.162:4130...
2024-06-05T13:22:25.695938Z  WARN snarkos_node_router: Unable to connect to '35.200.149.162:4130' - '35.200.149.162:4130' disconnected before sending "Message::ChallengeResponse"
2024-06-05T13:22:49.768433Z DEBUG snarkos_node_router::heartbeat: No connected peers
2024-06-05T13:22:50.770948Z  WARN snarkos_node_router: Unable to connect to '35.200.149.162:4130' - timed out
2024-06-05T13:23:14.770825Z DEBUG snarkos_node_router::heartbeat: No connected peers
2024-06-05T13:23:15.162124Z DEBUG snarkos_node_router::handshake: Connecting to 34.17.53.129:4130...
2024-06-05T13:23:15.564710Z  WARN snarkos_node_router: Unable to connect to '34.17.53.129:4130' - '34.17.53.129:4130' disconnected before sending "Message::ChallengeResponse"
2024-06-05T13:23:39.772932Z DEBUG snarkos_node_router::heartbeat: No connected peers
2024-06-05T13:23:39.937160Z DEBUG snarkos_node_router::handshake: Connecting to 34.168.118.156:4130...
2024-06-05T13:23:40.177124Z  WARN snarkos_node_router: Unable to connect to '34.168.118.156:4130' - '34.168.118.156:4130' disconnected before sending "Message::ChallengeResponse"
2024-06-05T13:24:04.776412Z DEBUG snarkos_node_router::heartbeat: No connected peers
2024-06-05T13:24:05.173990Z DEBUG snarkos_node_router::handshake: Connecting to 35.200.149.162:4130...
2024-06-05T13:24:05.685047Z  WARN snarkos_node_router: Unable to connect to '35.200.149.162:4130' - '35.200.149.162:4130' disconnected before sending "Message::ChallengeResponse"
2024-06-05T13:24:29.779043Z DEBUG snarkos_node_router::heartbeat: No connected peers
2024-06-05T13:24:30.116198Z DEBUG snarkos_node_router::handshake: Connecting to 35.231.152.213:4130...
2024-06-05T13:24:30.526530Z  WARN snarkos_node_router: Unable to connect to '35.231.152.213:4130' - Connection reset by peer (os error 54)
2024-06-05T13:24:54.781833Z DEBUG snarkos_node_router::heartbeat: No connected peers
2024-06-05T13:24:55.157425Z DEBUG snarkos_node_router::handshake: Connecting to 34.17.53.129:4130...
2024-06-05T13:24:55.534530Z  WARN snarkos_node_router: Unable to connect to '34.17.53.129:4130' - '34.17.53.129:4130' disconnected before sending "Message::ChallengeResponse"
2024-06-05T13:25:19.783895Z DEBUG snarkos_node_router::heartbeat: No connected peers
2024-06-05T13:25:20.190761Z DEBUG snarkos_node_router::handshake: Connecting to 35.231.152.213:4130...
2024-06-05T13:25:20.552473Z  WARN snarkos_node_router: Unable to connect to '35.231.152.213:4130' - Connection reset by peer (os error 54)
elderhammer commented 1 month ago

I see, thank you for reporting. Does the snarkos start --nocdn flag resolve your issue whil CDN is getting spun up?

No. The log is as follows:

2024-06-05T13:22:24.563200Z  WARN snarkos_node_cdn::blocks: The network (1) is not supported
2024-06-05T13:22:24.563494Z DEBUG snarkos_node_rest: REST rate limit per IP - 10 RPS
2024-06-05T13:22:24.766388Z DEBUG snarkos_node_router::heartbeat: No connected peers
2024-06-05T13:22:25.170755Z DEBUG snarkos_node_router::handshake: Connecting to 35.200.149.162:4130...
2024-06-05T13:22:25.695938Z  WARN snarkos_node_router: Unable to connect to '35.200.149.162:4130' - '35.200.149.162:4130' disconnected before sending "Message::ChallengeResponse"
2024-06-05T13:22:49.768433Z DEBUG snarkos_node_router::heartbeat: No connected peers
2024-06-05T13:22:50.770948Z  WARN snarkos_node_router: Unable to connect to '35.200.149.162:4130' - timed out
2024-06-05T13:23:14.770825Z DEBUG snarkos_node_router::heartbeat: No connected peers
2024-06-05T13:23:15.162124Z DEBUG snarkos_node_router::handshake: Connecting to 34.17.53.129:4130...
2024-06-05T13:23:15.564710Z  WARN snarkos_node_router: Unable to connect to '34.17.53.129:4130' - '34.17.53.129:4130' disconnected before sending "Message::ChallengeResponse"
2024-06-05T13:23:39.772932Z DEBUG snarkos_node_router::heartbeat: No connected peers
2024-06-05T13:23:39.937160Z DEBUG snarkos_node_router::handshake: Connecting to 34.168.118.156:4130...
2024-06-05T13:23:40.177124Z  WARN snarkos_node_router: Unable to connect to '34.168.118.156:4130' - '34.168.118.156:4130' disconnected before sending "Message::ChallengeResponse"
2024-06-05T13:24:04.776412Z DEBUG snarkos_node_router::heartbeat: No connected peers
2024-06-05T13:24:05.173990Z DEBUG snarkos_node_router::handshake: Connecting to 35.200.149.162:4130...
2024-06-05T13:24:05.685047Z  WARN snarkos_node_router: Unable to connect to '35.200.149.162:4130' - '35.200.149.162:4130' disconnected before sending "Message::ChallengeResponse"
2024-06-05T13:24:29.779043Z DEBUG snarkos_node_router::heartbeat: No connected peers
2024-06-05T13:24:30.116198Z DEBUG snarkos_node_router::handshake: Connecting to 35.231.152.213:4130...
2024-06-05T13:24:30.526530Z  WARN snarkos_node_router: Unable to connect to '35.231.152.213:4130' - Connection reset by peer (os error 54)
2024-06-05T13:24:54.781833Z DEBUG snarkos_node_router::heartbeat: No connected peers
2024-06-05T13:24:55.157425Z DEBUG snarkos_node_router::handshake: Connecting to 34.17.53.129:4130...
2024-06-05T13:24:55.534530Z  WARN snarkos_node_router: Unable to connect to '34.17.53.129:4130' - '34.17.53.129:4130' disconnected before sending "Message::ChallengeResponse"
2024-06-05T13:25:19.783895Z DEBUG snarkos_node_router::heartbeat: No connected peers
2024-06-05T13:25:20.190761Z DEBUG snarkos_node_router::handshake: Connecting to 35.231.152.213:4130...
2024-06-05T13:25:20.552473Z  WARN snarkos_node_router: Unable to connect to '35.231.152.213:4130' - Connection reset by peer (os error 54)

I guess the reason why these bootstrap peers can't connect is that they are all validator nodes, and my node is not in their neighbor list, so the connection is rejected. I found a solution to request the neighbor list of these nodes (for example, curl http://35.231.152.213:3030/testnet/peers/all), and then establish a connection with the nodes in the list. But this solution is not ideal and inconvenient. The bootstrap node should be some kind of client node like a beacon, providing a list of neighbors for the node running for the first time. @vicsn

HarukaMa commented 1 month ago

I believe this is partly #3011, specifically that the beacon nodes only have default MAXIMUM_NUMBER_OF_PEERS which is too low for bootstrapping other nodes.

elderhammer commented 3 weeks ago

Bootstrap nodes can be attacked by connection spam and not work properly, https://github.com/AleoNet/snarkOS/issues/3311