search

ethereum / public-attacknets

Public attacknets available for eth2
Creative Commons Zero v1.0 Universal
72 stars 18 forks source link

[lighthouse-attack-0 Issue] Network agent crashes lighthouse discovery #4

Closed jrhea closed 4 years ago

jrhea commented 4 years ago

Description

A network agent, capable of creating a large number of discv5 sessions, crashed the discovery service on several lighthouse nodes.

Attack scenario

A large number of simultaneous discv5 sessions established with a lighthouse node will fill up the session cache causing sessions to expire quickly. This can lead to a scenario where sessions and requests are no longer in sync and the handling of a delayed response can trigger an exception.

Impact

The attack left 3 of 4 nodes without the ability to discover or be discovered by new peers.

Details:

The following log is from the code that triggered the fault. You can see that at Jul 20 11:36:50.903 CDT it is connected to one of the nodes, then it loses the connection a few seconds later. This corresponds to the outage on the attacknet node.

imp: 
 Jul 20 11:36:26.684 INFO Starting imp
 imp: NetworkService
  Network Service: Crawler
   Jul 20 11:36:26.685 INFO Found 1 bootstrap enrs
   Jul 20 11:36:26.690 INFO Local Node Id: 0xad05..51af
   Jul 20 11:36:26.746 INFO calling find_node()
   Jul 20 11:36:34.907 INFO Connected Peers: 0
   Jul 20 11:36:34.907 INFO Enr Entries: 1
   Jul 20 11:36:34.907 INFO Output is enabled.  Saving data to file
   Jul 20 11:36:36.747 INFO calling find_node()
   Jul 20 11:36:44.897 INFO Connected Peers: 0
   Jul 20 11:36:44.898 INFO Enr Entries: 1
   Jul 20 11:36:46.750 INFO calling find_node()
   Jul 20 11:36:50.903 INFO Connected Peers: 1
   Jul 20 11:36:50.903 INFO Enr Entries: 1
   Jul 20 11:36:56.750 INFO Output is enabled.  Saving data to file
   Jul 20 11:36:56.750 INFO calling find_node()
   Jul 20 11:37:04.898 INFO Connected Peers: 0
   Jul 20 11:37:04.898 INFO Enr Entries: 1
   Jul 20 11:37:06.751 INFO calling find_node()
   Jul 20 11:37:14.904 INFO Connected Peers: 0
   Jul 20 11:37:14.904 INFO Enr Entries: 1
   Jul 20 11:37:16.749 INFO calling find_node()
   Jul 20 11:37:24.895 INFO Connected Peers: 0
   Jul 20 11:37:24.895 INFO Enr Entries: 1
   Jul 20 11:37:26.749 INFO calling find_node()
   Jul 20 11:37:34.898 INFO Connected Peers: 0
   Jul 20 11:37:34.898 INFO Enr Entries: 1
   Jul 20 11:37:34.898 INFO Output is enabled.  Saving data to file
   Jul 20 11:37:36.751 INFO calling find_node()
   Jul 20 11:37:44.899 INFO Connected Peers: 0
   Jul 20 11:37:44.899 INFO Enr Entries: 1
   Jul 20 11:37:46.752 INFO calling find_node()
   Jul 20 11:37:54.902 INFO Connected Peers: 0
   Jul 20 11:37:54.902 INFO Enr Entries: 1
   Jul 20 11:37:56.748 INFO Output is enabled.  Saving data to file
   Jul 20 11:37:56.749 INFO calling find_node()
   Jul 20 11:38:04.903 INFO Connected Peers: 0
   Jul 20 11:38:04.903 INFO Enr Entries: 1
   Jul 20 11:38:06.752 INFO calling find_node()
   Jul 20 11:38:14.904 INFO Connected Peers: 0
   Jul 20 11:38:14.904 INFO Enr Entries: 1
   Jul 20 11:38:16.752 INFO calling find_node()
   Jul 20 11:38:24.755 INFO Connected Peers: 0
   Jul 20 11:38:24.755 INFO Enr Entries: 1
   Jul 20 11:38:26.735 INFO calling find_node()
   Jul 20 11:38:34.740 INFO Connected Peers: 0
   Jul 20 11:38:34.740 INFO Enr Entries: 1
   Jul 20 11:38:34.740 INFO Output is enabled.  Saving data to file
   Jul 20 11:38:36.724 INFO calling find_node()
   Jul 20 11:38:44.738 INFO Connected Peers: 0
   Jul 20 11:38:44.739 INFO Enr Entries: 1
   Jul 20 11:38:46.718 INFO calling find_node()
   Jul 20 11:38:54.730 INFO Connected Peers: 0
   Jul 20 11:38:54.730 INFO Enr Entries: 1
   Jul 20 11:38:56.720 INFO calling find_node()
   Jul 20 11:39:04.737 INFO Connected Peers: 0
   Jul 20 11:39:04.737 INFO Enr Entries: 1
   Jul 20 11:39:04.737 INFO Output is enabled.  Saving data to file
   Jul 20 11:39:06.718 INFO calling find_node()
   Jul 20 11:39:14.732 INFO Connected Peers: 0
   Jul 20 11:39:14.732 INFO Enr Entries: 1
   Jul 20 11:39:16.715 INFO calling find_node()
   Jul 20 11:39:20.778 WARN Sending shutdown signal.
 imp: Agent
  Jul 20 11:39:20.778 WARN "imp_agent::agent::Agent": shutdown message received.
 imp: NetworkService
  Jul 20 11:39:20.778 WARN "imp_network::service::Service": shutdown message received.
 Jul 20 11:39:22.087 WARN Exiting imp.
Post processing starting...
Post processing complete
exit
djrtwo commented 4 years ago

Thank you @jrhea!

@agemanning fixed this here https://github.com/sigp/lighthouse/pull/1373 and the fix has been deployed to lighthouse-attack-0. Thank you :)

This issue qualifies for an honorable mention with a reward of $1k USD along with your name on the (soon to be) attacknet trophy section 🏆

jrhea commented 4 years ago

I think the "Jonny Prevention Patch" 😂 that @AgeManning made is in the discv5 repo:

https://github.com/sigp/discv5/pull/24