Roadmap towards MVP seeder in Rust

softminus commented 2 years ago

Let's think step-by-step:

General strategy

If we don't know how to implement it, go with something simpler. If we don't know what to implement, go with what the original zcash seeder or with what the ZF seeder https://github.com/ZcashFoundation/dnsseeder does.

Loops and state

[x] Start with a loop that connects, polls, then disconnects from a premade list of servers: https://github.com/superbaud/zcash-cotyledon/commit/5c227a64c53fc36b0377d4a78383af0cd10caa66
[x] properly handle and classify errors while polling peers: 7533610a8d5edae2e5c017c814b0b769ba8fc5ce
[x] get a gRPC server running in Rust: 8942c08ccf84f150ddaeb42913d0762907aa4c1d
[x] query peers in separate threads so a slow-responding network peer doesn't slow down the whole loop
[x] Write the state management code to classify and transition peers as good/bad/reachable/unreachable etc as appropriate by reachability and other good/bad behavior
[x] instead of the old Banned/Ignore logic, use heuristics to prioritize what look like valid zcash peers for polling: we can simply look at peer_derived_data -- if that's not None it means we were able to negotiate a connection which means that it's probably better than nothing
[x] do an initial crawl that's fast (in order to quickly acquire a good view of the network) and then switch to a slower-querying/crawling mode once we've done enough initial crawl to acquire a good view. i think the right way to do this is to crawl as fast as we can while we keep finding new good (correct reply to block hash request) zcash nodes, then once that stabilizes, enable the (not yet implemented) interval code
[x] for the initial acquisition we can use the traversal method without an in-flight limit that adds nodes on the fly to the work queue, then for normal polling we can use the current batched method
[ ] Write code to save (and restore) the per-peer statistics to disk: we need to figure out what sort of database to use for this? or maybe just serde

Serving

[x] add serving of IP addresses/ports over gRPC: d91b3df4ece3bc5b17d78b21ccfdf8b85e01f55c
[x] #2
[ ] #3
[x] properly differentiate A and AAAA queries, and reply properly
[ ] make sure we can fit all our A records inside the DNS packet, if not, figure out what to do

Providing perfect per-peer polling

[x] keep track of per-peer reachability statistics (start simple at first): in progress, I have multi-scale EWMA working https://github.com/superbaud/zcash-cotyledon/commit/31fd4c7b9c975844a8c3ee8b435e4e01a5d5877c
[x] write code to actually use the multi-scale EWMA data (with the same filtering/selection logic as the original seeder)
[x] obtain the per-peer version/services statistics: we need to collaborate with ZF to rework the zebra_network connection functions to return all the peer version metadata, the issue for this is https://github.com/ZcashFoundation/zebra/issues/4678 and a corresponding PR is https://github.com/ZcashFoundation/zebra/pull/4870
[x] write code to classify peers by version/services (again, start off with the same filtering/selection logic as the original seeder)
[x] retry connections that return RetryConnection
[X] we don't need retry logic for the slow walker, we just need to not mark that we attempted a connection and the (unwritten) interval logic will retry that host next time we run the slow walker
[x] try and increase ulimit -n to a big number
[x] properly handle running on hosts with low ulimit -n set -- maybe don't start the fast walker? or actually we probably can use a semaphore (with ulimit -n's value minus like 256 or something like that) to limit in-flight connections in the fast walker? or we could just ignore this and just retry connections like mad until they succeed (or fail because of the network or the remote node)

Interval/scheduling/timing stuff

[x] interval code: write code to query the right sets of peers at similar intervals as the original seeder. A way to do this might be to make a function that looks at the PeerStats structure for each peer and determines whether we should poll it or not, and filter the list of all known nodes with this function each time we iterate.
[x] more frequent revisit for nodes that have worked recently, perhaps?
[x] rewrite the Slow Walker with an eye towards reducing network burstiness (both for good hosts and also for non-good hosts)

Crawling

[x] poll peers for other peers, and add them to the list we're keeping track of as Unknown classification

Actual connection stuff

[x] Don't actually need: Do all the TCP connections on our own, to cleanly isolate TCP/IP-related errors from zebra-network errors
[x] differentiate between errors caused by us and errors caused by the peer, and don't blame the peer for errors caused by us (including EMFILE errors)
[x] Don't actually need: When appropriate (fast walker), use the same connection for both hash testing and for peer request
[x] add sensible (dependent on fast/slow walker and maybe if node is known to be good or not) timeouts for the TCP connection (and the handshake/protocol work)

Extra credit

[x] keep track of per-peer synchronization status (start simple; if they're synced to tip)
[x] Try asking peers to serve us a reasonably recent (after any notable chain forks) hardcoded block from the zcash chain. this lets us robustly exclude chain forks like ZelCash/flux even if they didn't change the network magic
[X] NOT NECESSARY: consider resetting peer_derived_data to None after a bunch of connections have failed (we could use the EWMAs for this -- if it's not been reachable for a long while it's no longer worth keeping that info around)? not sure
[x] check that the reply to the get block command actually hashes
[X] NOT NECESSARY figure out how to do a full traversal (when we find a peer, add it to the list of peers we're evaluating this time around) while also rate-limiting the number of in-flight connections and also being able to end the traverse once we haven't found any new peers and we've tried all the hosts
[X] consider using the checkpoints from Zebra instead of hardcoding our own
[ ] add a new type of test just to check if the node is up and to get a copy of the version etc (you know, like the legacy seeder did...) and use this most of the time -- the hash check is expensive network-wise for the remote nodes
[ ] #7
[ ] keep track of node connect latency and throughput, and give credit to faster nodes
[ ] In the slow walker, split the hash test and the peer request, and do the latter less often since peer requests hit a zcashd timeout

Code cleanup

[x] remove hardcoded values (timeouts, mainnet/testnet, etc) from functions and pass parameters as appropriate
[x] only compute the checkpoints once in main, write to a global variable (with std:sync:Once or something like that)

Observability

[ ] Figure out what sort of metrics to export, and how
[ ] figure out if it's useful to save historical metrics/state
[ ] https://zcash-gui.vercel.app/zebra
[ ] https://github.com/runziggurat/zcash/tree/main/src/tools/crawler

User interface

[ ] Add command-line arguments and/or a config file to set parameters like gRPC / DNS ports, DNS hostname, connection limit, etc

softminus commented 2 years ago

With d91b3df4ece3bc5b17d78b21ccfdf8b85e01f55c we are serving something over the network that is derived from the live peer set and its statistics! we don't yet do crawling nor keep detailed/granular statistics, so the peer set is hardcoded, and it's not quite machine-parseable but I have learned enough Rust to connect all these moving parts so far!

sasha_work@seasonal-dream zcash-cotyledon % grpcurl -plaintext -import-path ./proto -proto seeds.proto 127.0.0.1:50051 seeder.Seeder/Seed
{
  "IP": [
    "[PeerStats { address: 34.127.5.144:8233, attempts: 8, successes: 7 }, PeerStats { address: 157.245.172.190:8233, attempts: 8, successes: 7 }]"
  ]
}

softminus commented 2 years ago

I have the multi-scale EWMA calculations working! (though to be perfectly honest I do not quite understand the role of the count and weight tracked variables) https://github.com/superbaud/zcash-cotyledon/commit/31fd4c7b9c975844a8c3ee8b435e4e01a5d5877c

softminus commented 2 years ago

Running into something weird with the Peers replies -- I'm not getting multiple peers anymore:

Starting new connection: peer addr is 157.245.172.190:8233
peers response: Peers { peers: 1 }
peers is: [MetaAddr { addr: 157.245.172.190:8233, services: Some(NODE_NETWORK), untrusted_last_seen: Some(DateTime32 { timestamp: 1666049426, calendar: 2022-10-17T23:30:26Z }), last_response: None, last_attempt: None, last_failure: None, last_connection_state: NeverAttemptedGossiped }]

softminus commented 2 years ago

OK, since the getaddr command seems to be rate-limited (by AVG_ADDRESS_BROADCAST_INTERVAL) -- let's not use that to check if the node is live (BlocksByHash isn't rate-limited at all).

softminus commented 2 years ago

i think i want to slightly change how crawling works; by creating a pending work queue (of IPs to query) and having it be separate from the HashMap (which once i have some other stuff working, i'll figure out how to save/restore from disk, undecided yet whether to use serde or learn how to use sqlite) that keeps track of all the nodes about, and so we can add new peers into the work queue as we find them (and also add them into the HashMap) while we are doing crawling vs needing to separate and batch crawling and updating (because we can't modify the HashMap while we're iterating over it)

softminus commented 2 years ago

We have DNS serving working!

softminus / zcash-cotyledon