Node discovery v5

[holgerd77]

Introduction

Node discovery v5 is on it's way, which should make it a lot easier to discover suitable nodes. Though not standardized yet, ~it would be good to have some early experimentation/draft implementation to get closer to a feeling how the protocol (could) behave~ it would be good to have this library updated with a working implementation to support the practical usage in terms of a light client implementation.

Here are some draft documents on this. Also valuable in this regard: Talk from Felix at Devcon3.

Current py-evm implementation work: https://github.com/ethereum/py-evm/pull/209

Current Situation

Currently the library supports Node discovery v4 implemented in src/dpt, where kbucket.js manages the list of peers in a Kademlia-like DHT, message.js implements the different message types like ping, pong or neighbours to discover new peers and server.js sets up a server to handle the discovery communication.

There are static tests for the message types in ./test/dpt-message.js, the integrated communication process is tested in ./test/integration/dpt-simulator.js as well as the corresponding eth and les files.

With the two peer-communication example scripts in the examples folder both ETH and LES (so full- and light client wire) communication can be tested in a real-world scenario by connecting to an actual network of in-production clients.

Task Description

Node discovery v5 updated the structure of Node records, brings changes to the communication flow by adding new packet types and introduces the concept of topics for exchanging on the capability of peers.

A first step into this task would be to work out some concept/idea where these additional/changed components fit into the existing v4 implementation and where current files/classes can be extended and where additional structures have to be set up. Since network communication implementation is very error-prone it will probably ease implementation to early on think along how to extend existing tests and add new tests for added functionality.

Goals

Though being still marked as "experimental" Geth has a working v5 discovery implementation for some time. Sufficient requirement for this issue to be completed is a working peer-communication example where it is possible to set the discovery to v5 - e.g. by CL parameter or example internal constant - and get a working LES and/or (optimally both) ETH connection (working means here: e.g. LES packets are actually passed through) in a reasonable amount of time (repeatedly < 5 min until first connection occurs).

Necessary side goals are:

• An extended code base which...
  • follows the style principles and coding best practices of the existing code base
  • is readable/maintainable for other developers
  • introduces no or minimal code repetitions
• An extended test suite which keeps code coverage for the DHT on more-or-less the current level
• Some additions to the README docs which gives some usage guides and reflect the API changes

Required Skills

Everyone is invited to join and take on this task. Since this an advanced task requiring deeper knowledge of the Ethereum networking stack it is probably more likely that you succeed if you bring along skills/experience in the following fields:

• Experience with working on larger code bases in Node.js/modern ES6 Javascript
• Experience with the Node.js toolchain / ecosystem
• Knowledge/understanding of generic p2p networking concepts
• Some understanding of the Ethereum networking mechanics
• Experience with assuring code quality/testing

Guidance

For questions helping to understand the existing code base and the scope of this issue you will get active guidance by the creator of this issue (@holgerd77, EthereumJS team).

[Silur]

~I guess I'll just claim this then~

Edited (@holgerd77): Open again since @Silur seems to not have enough free resources on that.

[holgerd77]

Cool. :-)

[holgerd77]

For reference: https://github.com/ethereum/go-ethereum/issues/2117#issuecomment-370559812 ("p2p/discover: from address in ping is IPv6 unspecified in some cases")

[holgerd77]

Py-EVM implementation [in progress]: https://github.com/gsalgado/py-evm/commit/f31fbfcd654753789454863f4fb4e2c991e7af2c

[holgerd77]

@Silur Are you still open/do you have time to work on this one? Otherwise I would put this back in "help wanted", with the current comments people interested will think this is already being worked on.

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

This issue now has a funding of 1.5 ETH (1068.95 USD @ $712.63/ETH) attached to it.

• If you would like to work on this issue you can 'start work' on the Gitcoin Issue Details page.
• Questions? Checkout Gitcoin Help or the Gitcoin Slack
• $9,225.68 more funded OSS Work available on the Gitcoin Issue Explorer

[rauljordan]

Hey @holgerd77 interested in this, as we are currently exploring p2p in my team, Prysmatic Labs. Can you elaborate more on the requirements needed to get the bounty accepted? What would an acceptable PR look like? Any specific features you have in mind to close this?

[alexanderbez]

Interested in helping out with this as well.

[abitrolly]

Is there an abstract describing current situation and the problem with discovery?

[holgerd77]

I'll update the issue tomorrow with a more specific work description and requirements for a bounty to be accepted.

[holgerd77]

Hi everyone, I updated the work specification, this should now hopefully accurately enough describe what is expected here, if you still have questions let me know. This might sound a bit tighter then before (didn't realize that I had this "experimental implementation" in the introduction of the issue description, I replaced this pointing to an actual working implementation).

Please re-read, take a breath and maybe sleep over one night and think, if this is a good issue for you. This is an advanced task, requiring some intrinsic motivation as well as some pre-knowledge to not make this too demanding.

Thanks everyone to expressing interest in this so far, pretty amazed about the feedback! 😄

[vs77bb]

Hi all, Vivek from Gitcoin here. It looks like @brianherman was first to 'Start Work' here on Gitcoin and thus has precedence to give this a go + @alexanderbez has offered his help as well.

We'd encourage a collaborative effort here if it makes sense, as in line with open source ethos 😄perhaps @brianherman gives it a go and comes back around for feedback as needed?

@rauljordan good to see you here; @holgerd77 has updated with acceptance criteria above!

[holgerd77]

@brianheram Hi Brian, does this still fit your expectations after the updated spec description and do you want to start on this? If you have any questions let me know!

[vs77bb]

@brianherman see above^ 🙂

[gitcoinbot]

@brianherman Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

• [x] warning (3 days)
• [ ] escalation to mods (6 days)

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

[vs77bb]

Hi @abitrolly @alexanderbez @rauljordan are any of you interested in taking on this bounty? @holgerd77 and I have discussed and increased the contribution to 2ETH (blockchain syncing now).

As an FYI, we understand the expiry date is near on Gitcoin, but will be able to pay this out upon completion, even if it takes beyond the expiration.

[alexanderbez]

I'm certainly happy to help on this. Seems there are three major topics: packets, node records, and topics. Perhaps we could come up with a plan/design and tackle these in parallel?

[rauljordan]

I agree that we should split this up, otherwise it would be a bunch of mega PRs. I am not as experienced on devp2p, but I'm willing to learn more and help on parts of this. Perhaps someone can come up with a proposal that can be split up into several smaller PRs/Issues?

Raul E. Jordan, Harvard University | Ethereum Protocol Developer

r auljordan.com ( http://rauljordan.com ) Twitter: @raulitojordan ( https://twitter.com/raulitojordan )

gpg --recv-keys 0x89725027FC8EC0D4

On Tue, May 29, 2018 at 3:13 PM, Alexander Bezobchuk < notifications@github.com > wrote:

I'm certainly happy to help on this. Seems there are three major topics: packets, node records, and topics. Perhaps we could come up with a plan/design and tackle these in parallel?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub ( https://github.com/ethereumjs/ethereumjs-devp2p/issues/19#issuecomment-392947171 ) , or mute the thread ( https://github.com/notifications/unsubscribe-auth/AFUIPfnvvpSrtMhjutjJdQYua-aGPoaTks5t3bnogaJpZM4RDejT ).

[holgerd77]

@alexanderbez @rauljordan Cool guys, that sounds great!

[jvmaia]

Hey i'm interested in help in this issue, i'm not experienced in devp2p but i want to learn more about.

[vs77bb]

Hi @jvmaia @rauljordan @alexanderbez I wanted to follow up as the bounty funder; is there anything I can do to help coordinate? We're happy to add to the bounty as it looks like it may become a collaborative effort.

[abitrolly]

The bounty is good, but I won't have the time to research current p2p protocols and their weaknesses.

[timxor]

@holgerd77 @vs77bb @gitcoinbot can we get an extension on this? I could get it done by next week, pretty familiar with this code base -- somewhat.

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

Work has been started.

These users each claimed they can complete the work by 8 months, 3 weeks ago. Please review their action plans below:

1) tcsiwula has started work.

If an deadline extension is made available, then I could augment the current v3 protocol to the v4.

Learn more on the Gitcoin Issue Details page.

[holgerd77]

Hi @tcsiwula, so great, happy that you want to help out, let me know if you need any assistance!

As @vs77b already stated above bounty can be paid out even after expiration:

"As an FYI, we understand the expiry date is near on Gitcoin, but will be able to pay this out upon completion, even if it takes beyond the expiration."

[vs77bb]

@holgerd77 @tcsiwula Seconding Holger, this can be paid post expiration!

[timxor]

Awesome sounds good @holgerd77 @vs77bb :D ping you an update on my progress this weekend (or if I have any questions between then)!

[holgerd77]

Hi Tim @tcsiwula, don't want to pressure or something just would like to be a bit aware of the status. Have you been able to do some work on this over the weekend? Any questions?

[timxor]

Thanks for the reminder! Let me get something back here within 12 hours, if not some very detailed questions~ @holgerd77

[timxor]

I have to admit -- this is a very nice write up you gave 👍

[timxor]

Hacking into the night on it @holgerd77 , had a bunch of meetings this afternoon

[timxor]

I am falling asleep, this PR is a WIP. The rest should come tomorrow, pinky promise o_0

[holgerd77]

Hi @tcsiwula, don't want to be too pressing on this, I know we set the original time frame a bit tight, could you nevertheless give a rough estimate on your implementation plans on the issue? Any current questions or blockers?

If you think I have conceptionalized the bounty too large in scope, feel also free to express concerns in this direction. I am also not 100% sure on this, but I am confident that we'll find a solution then.

Happy Node discovery! 😄 Holger

[spm32]

Hi @tcsiwula any updates here? Just want to make sure you don't get kicked off the issue. Thanks!

[timxor]

Hey hey @holgerd77 @ceresstation sorry about falling off the map on this. i was working of it for quite some bit. i was traveling for a few weeks and then had a cold. it was kinda hard to gauge the current completion %, also making it backwards compatible with devp2p v4 was kind of challenging. While it was very fun to get this far I am thinking it might have been way more time than 1eth, at least for my skill level and lack of js. What do you guys think? Was I suppose to or not touch any of the rlpx files for this?

[spm32]

Hey @tcsiwula happy to increase the amount + add an additional tip if you're able to complete it promptly, @holgerd77 will have to answer regarding the rlpx files though.

[holgerd77]

Hi @tcsiwula, great to see that you are not completely lost without a trace!! 😄 😄

We are actually falling here completely into the ETH volatility trap, I think when I conceptionalized this bounty, ETH price was at somewhat around 1.200 $ (sigh), that's why I moved to DAI for the second bounty we set up on VM testing.

I would suggest that we also move to a DAI listing here (if still possible), and - if @ceresstation can go with this - I would suggest an updated bounty here of 1.700 DAI, since the task is really somewhat demanding and - as I wrote earlier - I was not sure from the beginning if I set this up to broadly.

For this I would now nevertheless expect some constant progress on this (at least an update every two weeks) and also a somewhat more active communication if things are stalled at points. I understand your reasons but this 4 week period of non-communication was already at the very limits of what I would regard as acceptable, please at least pre-communicate with a short note in cases like these, since we also have to do some planning/estimation when we can build on-top features on this.

I am not sure if you also have to touch rlpx files for this, my first guess would be no, you can take e.g. the Py-EVM implementation I referenced above (or others) where things need to be changed. I am also not 100% sure about the need for backwards compatibility, would be interested in some current statistics on v4/v5 distribution, maybe it's also a possibility to already drop v4 support. We can figure stuff like this out during the next 2-3 weeks.

What do you think? Are these new conditions (once/if Scott approves) acceptable for you? Would still be great, if this can be finished, especially if you already did some significant progress!

Cheers Holger

[jwasinger]

Hey @tcsiwula , if you want to pass the torch on this one, I'd be willing to take it up. However, that would also mean removing bounty from this issue (I'm not eligible given my relationship with the Ethereum Foundation).

[spm32]

Hey @tcsiwula any other updates here? Otherwise we may need to move this over to @jwasinger.

[timxor]

Hey hey ☺️☺️ @holgerd77 @jwasinger @ceresstation ! My response time is getting better, 3 day ACK!

For (my lack of) communications, I am very sorry. This is an issue of mine with emails and the such. I will make a concerted effort to be more prompt and I think your expectations are very normal =)

For the timelines, I am pretty comfortable with thinking I could get this finished within 2-3 weeks. I will be stationary and have access to a library that should allow for effective work blocks. My next trip is for ETHBerlin in about 3 weeks and if I am unable to have it finished by then would be comfortable handing it off to you @jwasinger , what do you guys think?

I just reached out to @gsalgado about his discv5 implementation to help me determine more concrete requirements and if that implementation is working and communicating with the network and the status of it. Will update you on that. I assume the geth and parity versions are working?

For the payment, yeah that works for me about the 1,700 DAI or whatever tip (goes straight to student loans =))

Let me know whatever one thinks and if it is a green light then I can sprint over the weekend and come back Monday with some questions.

[jwasinger]

Sounds good @tcsiwula .

[holgerd77]

Hi @tcsiwula, ok, we can follow on like this (though I have to admit I would be even more happy with a clear "I will do it or not" approach, but let's go on with this for now).

I think we also should be able to pay you some part of the bounty if it is not possible to finish all parts.

[vs77bb]

@holgerd77 Confirming on our end that this bounty is effectively 1,700 DAI at this point. @tcsiwula a rule of thumb we like to follow is to stay up-to-date on progress every few days, as that'll help everyone gauge current status as three weeks meanders by! Excited to see things progress. 🙂

[timxor]

Sounds great @holgerd @vs77bb . I was able to confirm the following over the weekend:

the python implementation is not working/finished the geth implementation is working but not formally specified so they can change it the specification is not backwards compatible nor will it ever be. the specification will have an upgrade path (under development/ not finished)

So this clarifies some things and helps define the bounds of expectations, at least in my mind. Removes some ambiguity about will it be able to communicate or not with other nodes (for testing/verification purposes).

Also @holgerd77 I can commit to I will do it =) -- I was just trying to be thoughtful in case the time window was super sensitive! 😀😀😀

Now that we are all on the same page and some clarity on other clients and backwards compatibility, I will have at it. Shoot you all an update later this week (in a few days 😀).

[holgerd77]

@tcsiwula Cool! 😄 👍 That's also already valuable research work! Looking forward to the update!

[timxor]

Issue update for node discovery v5

Completion status: ~46% Today's date: Saturday August 25th 2018 Target finish date: Monday September 3rd 2018 Task break down: ~46% = [6.5 / 14] = [ ((4 ✅ + (3🚧/2)) ) / (14✅🚧✖️)] Issue: #19 Pull request: #42 cc: @holgerd77 @vs77bb @ceresstation

Structural tasks

Task name	Implemented	Task Description
node records	✖️ No	add node records support for node discovery v5
packet types	🚧 Wip	add packet types support for node discovery v5
topics	🚧 Wip	add topics support for node discovery v5
cli parameter	✅ Yes	add v5 cli parameter support for node discovery v5
documentation	✅ Yes	add v5 cli parameter support for node discovery v5

Version 5 demo example tasks

Demo name	Implemented
peer-communication-v5.js	🚧 Wip
peer-communication-les-v5.js	✖️ No

New test coverage tasks

Test name	Implemented	Test Description
dpt-message-v5.js	✖️ No	static tests for v5 message types, node records, topics
dpt-simulator-v5.js	✖️ No	dpt integration communication tests for v5
eth-simulator-v5.js	✖️ No	eth integration communication tests for v5
les-simulator-v5.js	✖️ No	les integration communication tests for v5
util.js	✅ Yes

Pass existing tests, style, coverage

Name	Passing
coveralls	✖️ No
travis	✅ Yes

update next on monday 😀

[timxor]

No update for Monday. Sunday and Monday were Haskell networking. Will aim for getting half of the remaining sub tasks completed this week and update you all on Friday😃👋🏻🎉

[holgerd77]

Super-impressive summary, thanks Tim! 👍 😄

[timxor]

Thanks, will have a new update in about 12 hours or so. Taking some inspiration from geths discv5 structure.

[timxor]

Issue update for node discovery v5

latest updates

I decided it would be easier to partition the version 5 logic and created a new directory src/discv5 that mirrors src/dpt. Likewise for the examples, I made a separate directory and have a partial working communication example with the new packet types in examples/discv5/simple.js. Up next with be to complete /discv5/node.js and /discv5/topic.js and then finalize examples/discv5/simple.js and examples/discv5/peer-communication.js examples along with the tests at /tsts/discv5/*.

details

Completion status: ~?46?% Today's date: Monday September 3rd 2018 Target finish date: Monday September 10th 2018 Issue: #19 Pull request: #42 cc: @holgerd77 @vs77bb @ceresstation

Structural tasks

Task name	Implemented	Task Description
node records	🚧 Wip	add node records support for node discovery v5
packet types	✅🚧 Wip	add packet types support for node discovery v5
topics	🚧 Wip	add topics support for node discovery v5
cli parameter	✅ Yes	add v5 cli parameter support for node discovery v5
documentation	✅ Yes	add v5 cli parameter support for node discovery v5

Version 5 demo example tasks

Demo name	Implemented
/discv5/simple.js	🚧 Wip
/discv5/peer-communication.js	🚧 Wip
/discv5/peer-communication-les.js	✖️ No

New test coverage tasks

Test name	Implemented	Test Description
/tsts/discv5/dpt-message.js	✖️ No	static tests for v5 message types, node records, topics
/tsts/discv5/dpt-simulator.js	✖️ No	dpt integration communication tests for v5
/tsts/discv5/eth-simulator.js	✖️ No	eth integration communication tests for v5
/tsts/discv5/les-simulator.js	✖️ No	les integration communication tests for v5
util.js	✅ Yes

Pass existing tests, style, coverage

Name	Passing
coveralls	✖️ No
travis	✅ Yes