Issues running Teku with OpenEthereum

[mbaxter]

Description

We've got a user reporting the following stack trace when running Teku with OpenEthereum. It appears that the eth1 node is missing historical blocks, which is causing teku to throw a NPE:

16:17:20.699 INFO  - Eth1 tracker successfully caught up to chain head
16:17:20.707 FATAL - Exiting due to fatal error in Eth1DepositManager
java.util.concurrent.CompletionException: tech.pegasys.teku.service.serviceutils.FatalServiceFailureException: java.util.concurrent.CompletionException: java.lang.NullPointerException
        at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314) ~[?:?]
        at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:319) ~[?:?]
        at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:990) ~[?:?]
        at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:970) ~[?:?]
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
        at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2137) ~[?:?]
        at tech.pegasys.teku.infrastructure.async.SafeFuture.lambda$propagateResult$3(SafeFuture.java:152) ~[teku-infrastructure-async-20.11.1.jar:20.11.1]
        at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
        at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
        at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2137) ~[?:?]
        at tech.pegasys.teku.infrastructure.async.SafeFuture.lambda$propagateResult$3(SafeFuture.java:152) ~[teku-infrastructure-async-20.11.1.jar:20.11.1]
        at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
        at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:883) ~[?:?]
        at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2315) ~[?:?]
        at tech.pegasys.teku.infrastructure.async.SafeFuture.whenComplete(SafeFuture.java:503) ~[teku-infrastructure-async-20.11.1.jar:20.11.1]
        at tech.pegasys.teku.infrastructure.async.SafeFuture.whenComplete(SafeFuture.java:28) ~[teku-infrastructure-async-20.11.1.jar:20.11.1]
        at tech.pegasys.teku.infrastructure.async.SafeFuture.propagateResult(SafeFuture.java:147) ~[teku-infrastructure-async-20.11.1.jar:20.11.1]
        at tech.pegasys.teku.infrastructure.async.SafeFuture.lambda$exceptionallyCompose$25(SafeFuture.java:347) ~[teku-infrastructure-async-20.11.1.jar:20.11.1]
        at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
        at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
        at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2137) ~[?:?]
        at tech.pegasys.teku.infrastructure.async.SafeFuture.lambda$propagateResult$3(SafeFuture.java:152) ~[teku-infrastructure-async-20.11.1.jar:20.11.1]
        at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
        at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
        at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2137) ~[?:?]
        at org.web3j.utils.Async.lambda$run$1(Async.java:38) ~[core-4.6.2.jar:?]
        at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1800) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
Caused by: tech.pegasys.teku.service.serviceutils.FatalServiceFailureException: java.util.concurrent.CompletionException: java.lang.NullPointerException
        at tech.pegasys.teku.pow.Eth1DepositManager.lambda$start$3(Eth1DepositManager.java:74) ~[teku-pow-20.11.1.jar:20.11.1]
        at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:986) ~[?:?]
        ... 30 more
Caused by: java.util.concurrent.CompletionException: java.lang.NullPointerException
        at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314) ~[?:?]
        at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:319) ~[?:?]
        at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1155) ~[?:?]
        ... 29 more
Caused by: java.lang.NullPointerException
        at tech.pegasys.teku.pow.MinimumGenesisTimeBlockFinder.compareBlockTimestampToMinGenesisTime(MinimumGenesisTimeBlockFinder.java:87) ~[teku-pow-20.11.1.jar:20.11.1]
        at tech.pegasys.teku.pow.MinimumGenesisTimeBlockFinder.lambda$binarySearchLoop$0(MinimumGenesisTimeBlockFinder.java:60) ~[teku-pow-20.11.1.jar:20.11.1]
        at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1146) ~[?:?]
        ... 29 more
Teku is shutting down
16:17:20.727 INFO  - Stopped ServerConnector@52af750b{HTTP/1.1, (http/1.1)}{0.0.0.0:5051}
16:17:20.730 INFO  - Stopping Javalin ...
16:17:20.730 INFO  - Javalin has stopped
16:17:20.732 WARN  - Error shutting down connection manager

We should try to make teku's eth1 fetching logic more forgiving, if possible.

[rolfyone]

@ajsutton this is what i saw with nethermind once I fixed the initial problem of getting to head, when it goes and tries to find an earlier block.