Investigate lack of peers reported by MainNet bootnodes

[ajsutton]

Description

The Teku MainNet bootnode reports far fewer peers than other bootnodes with most distances having no reported peers.

Our bootnode ENR: enr:-KG4QJRlj4pHagfNIm-Fsx9EVjW4rviuZYzle3tyddm2KAWMJBDGAhxfM2g-pDaaiwE8q19uvLSH4jyvWjypLMr3TIcEhGV0aDKQ9aX9QgAAAAD__________4JpZIJ2NIJpcIQDE8KdiXNlY3AyNTZrMaEDhpehBDbZjM_L9ek699Y7vhUJ-eAdMyQW_Fil522Y0fODdGNwgiMog3VkcIIjKA

For exploratory testing I've been using a new class in the discovery test source tree:

package org.ethereum.beacon.discovery;

import static org.ethereum.beacon.discovery.util.Functions.PRIVKEY_SIZE;

import java.util.ArrayList;
import java.util.List;
import java.util.Optional;
import java.util.Random;
import org.apache.tuweni.bytes.Bytes;
import org.ethereum.beacon.discovery.schema.NodeRecord;
import org.ethereum.beacon.discovery.schema.NodeRecordBuilder;
import org.ethereum.beacon.discovery.schema.NodeRecordFactory;
import org.ethereum.beacon.discovery.util.Functions;
import org.ethereum.beacon.discovery.util.Utils;
import org.web3j.crypto.ECKeyPair;

public class Playground {
  public static void main(String[] args) {
    ECKeyPair keyPair = Functions.generateECKeyPair(new Random(1));
    final Bytes privateKey =
        Bytes.wrap(Utils.extractBytesFromUnsignedBigInt(keyPair.getPrivateKey(), PRIVKEY_SIZE));

    final DiscoverySystem system =
        new DiscoverySystemBuilder()
            .listen("0.0.0.0", 9000)
            .privateKey(privateKey)
            .localNodeRecord(
                new NodeRecordBuilder()
                    .privateKey(privateKey)
                    .address("180.150.110.29", 9000)
                    .seq(0)
                    .build())
            .newAddressHandler(
                (oldRecord, proposedRecord) -> {
                  System.out.println("Propsing address: " + proposedRecord);
                  return Optional.of(proposedRecord);
                })
            .build();
    final NodeRecord efBootnode =
        NodeRecordFactory.DEFAULT.fromEnr(
            "enr:-Ku4QHqVeJ8PPICcWk1vSn_XcSkjOkNiTg6Fmii5j6vUQgvzMc9L1goFnLKgXqBJspJjIsB91LTOleFmyWWrFVATGngBh2F0dG5ldHOIAAAAAAAAAACEZXRoMpC1MD8qAAAAAP__________gmlkgnY0gmlwhAMRHkWJc2VjcDI1NmsxoQKLVXFOhp2uX6jeT0DvvDpPcU8FWMjQdR4wMuORMhpX24N1ZHCCIyg");
    final NodeRecord bootnode1 =
        NodeRecordFactory.DEFAULT.fromEnr(
            "enr:-KG4QJRlj4pHagfNIm-Fsx9EVjW4rviuZYzle3tyddm2KAWMJBDGAhxfM2g-pDaaiwE8q19uvLSH4jyvWjypLMr3TIcEhGV0aDKQ9aX9QgAAAAD__________4JpZIJ2NIJpcIQDE8KdiXNlY3AyNTZrMaEDhpehBDbZjM_L9ek699Y7vhUJ-eAdMyQW_Fil522Y0fODdGNwgiMog3VkcIIjKA");
    final NodeRecord bootnode2 =
        NodeRecordFactory.DEFAULT.fromEnr(
            "enr:-KG4QL-eqFoHy0cI31THvtZjpYUu_Jdw_MO7skQRJxY1g5HTN1A0epPCU6vi0gLGUgrzpU-ygeMSS8ewVxDpKfYmxMMGhGV0aDKQtTA_KgAAAAD__________4JpZIJ2NIJpcIQ2_DUbiXNlY3AyNTZrMaED8GJ2vzUqgL6-KD1xalo1CsmY4X1HaDnyl6Y_WayCo9GDdGNwgiMog3VkcIIjKA");
  final NodeRecord node = efBootnode;
    system.start().join();

    system.ping(node).join();
    System.out.println("Pinged node " + node.getNodeId());
    final List<Integer> all = new ArrayList<>();
    for (int i = 0; i <= 258; i++) {
      System.out.println("------------ Distance " + i);
      system.findNodes(node, List.of(0)).join();
      all.add(i);
    }
    System.out.println("------------ All");
    system.findNodes(node, all).join();
    System.exit(0);
  }
}

[ajsutton]

Before nodes can be added to the k-buckets (which is where results for FINDNODES comes from), they have to be checked for liveness. But the liveness check is setup to only run periodically and check the 5 closest nodes to the local ENR. This results in repeatedly checking the liveness of some nodes and never checking it for most so very little winds up getting added to the buckets.

We need to schedule liveness checks much more intelligently.