search

holepunchto / hyperswarm

A distributed networking stack for connecting peers.
https://docs.holepunch.to
MIT License
1.04k stars 84 forks source link

[WIP] - APPROACH CHECK-IN - v1 progress #24

Closed davidmarkclements closed 5 years ago

davidmarkclements commented 5 years ago

initial implementation of the following

tests are passing but anything newly implemented hasn't been checked at all, this PR is intended as a check-in to ensure that the approaches are valid and that peer queue has been correctly understood

key discussion points:

leave

the leave method is using discovery caches to find topics - this saves mem but will mean leave is more expensive. I'm assuming for most use cases leave usage would be light, is this assumption correct? If so, we should modify hyperswarm discovery to return a topic for a given key instead of using private apis

join

In join I'm aware that discovery can be used to both announce and lookup, and this is more efficient - however we haven't implemented that in guts and I'd rather get something down and tested first. A change to announce+lookup could be made during optimization phase.

Join is making some new assumptions regarding sane option defaults, if the instance is not ephemeral, announce defaults to true, otherwise it defaults to false. The lookup option defaults to true if announce is false (same as in v0).

hashlru

~~I've added https://github.com/dominictarr/hashlru in place of the Map in peer queue. This is a lightweight and fast LRU, the trade off is that once peer count reaches LRU_SIZE * 2 then LRU_SIZE amount of peers will be cleared. Is this an acceptable trade off?~~

peer queue

My understanding of peer queue is that it periodically reprioritises peers based on their connection status. Status is checked via polling, a back-off mechanism using retry count to determine interval duration controls polling.

It emits a readable event when the queue is populated with one or more healthy peers. Peers can then be shifted in order of priority (which essentially communicates how healthy a peer is).

Peers are added to the queue when a topic gets a peer event.

My understanding of how to consume peers from the queue is similar to the readable/while approach with node streams, see https://github.com/hyperswarm/network/compare/v1...v1-progress?expand=1#diff-17a266fdf8e53cfbbdd558c38b69cefdR31

Is this the right direction?

The peer queue also has a remove method, but I'm not sure where to use this - wouldn't the peer queue be expected to call remove on itself once a peer has been banned? Are there other scenarios I'm not seeing where we would need to call remove from Swarm?

davidmarkclements commented 5 years ago
davidmarkclements commented 5 years ago

dependent on https://github.com/hyperswarm/guts/pull/9

davidmarkclements commented 5 years ago

cc @mafintosh - ready for another check in, could probably merge after this iteration to keep PR's bite size

mafintosh commented 5 years ago

Awesome stuff @davidmarkclements - I've added a few comments for clarifications.

I'd love to get this merged (after comments are resolved) and then move on to really locking the retry/peer behaivor down with a ton of tests so we have a strong contract on how we think this should behave, wdyt?

davidmarkclements commented 5 years ago

wdyt?

agree completely

okdistribute commented 5 years ago

super stoked on this! I hear that ara (https://ara.one/) is already using hyperswarm, and other projects will start taking it up soon too :)

davidmarkclements commented 5 years ago

@mafintosh comments addressed - once approved I'll merge and get into testing