ethereum / public-attacknets

Public attacknets available for eth2
Creative Commons Zero v1.0 Universal
72 stars 18 forks source link

[prysm-attack-0 Reward] L4 Distributed Denial of Service attack ruins the P2P connectivity and stops finality #9

Closed AlexSSD7 closed 4 years ago

AlexSSD7 commented 4 years ago

Description

Prysm nodes are vulnerable to L4 DDoS attacks. The attacker can stop finality for any amount of time.

Attack scenario

There are 4 nodes on this network, and I've rented 4x10 DigitalOcean droplets and started the L4 DoS utility on each of them. It was enough to break the whole network P2P connectivity.

Impact

The nodes were unable to normally communicate with each other. As a result, new nodes were unable to sync, and validators submit their attestations.

Details

Attack period: 2109-2125 epochs

Attack requirements

Attack log

[2020-07-29 12:39:02] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:39:02] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:39:02] ERROR sync: Could not send recent block request: i/o deadline reached
[2020-07-29 12:39:02]  INFO sync: Requesting parent block currentSlot=67660 parentRoot=17fa5b2bb063
[2020-07-29 12:39:12] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:39:13]  INFO sync: Requesting parent block currentSlot=67681 parentRoot=87e4fe87c0c6
[2020-07-29 12:39:17] ERROR sync: Could not process block from slot 67489: could not process block: could not execute state transition: could not process slot: expected state.slot 67496 < slot 67489
[2020-07-29 12:39:17]  INFO sync: Requesting parent block currentSlot=67490 parentRoot=70f8a3e2f405
[2020-07-29 12:39:22] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:39:27] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:39:27] ERROR sync: Could not send recent block request: i/o deadline reached
[2020-07-29 12:39:27]  INFO sync: Requesting parent block currentSlot=67590 parentRoot=86fd2d8d917f
[2020-07-29 12:39:32] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:39:35] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:39:37] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:39:37] ERROR sync: Could not send recent block request: i/o deadline reached
[2020-07-29 12:39:37]  INFO sync: Requesting parent block currentSlot=67647 parentRoot=991909b26e61
[2020-07-29 12:39:40]  INFO sync: Requesting parent block currentSlot=67681 parentRoot=87e4fe87c0c6
[2020-07-29 12:39:46]  INFO sync: Requesting parent block currentSlot=67490 parentRoot=70f8a3e2f405
[2020-07-29 12:39:51] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:39:51] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:39:56] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:39:56] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:39:56] ERROR sync: Could not send recent block request: i/o deadline reached
[2020-07-29 12:39:56]  INFO sync: Requesting parent block currentSlot=67590 parentRoot=86fd2d8d917f
[2020-07-29 12:40:01] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:40:01] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:40:06] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:40:06] ERROR sync: Could not send recent block request: i/o deadline reached
[2020-07-29 12:40:06] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:40:06]  INFO sync: Requesting parent block currentSlot=67646 parentRoot=2914e3534f4f
[2020-07-29 12:40:11] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:40:16] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:40:16] ERROR sync: Could not send recent block request: i/o deadline reached
[2020-07-29 12:40:16]  INFO sync: Requesting parent block currentSlot=67681 parentRoot=87e4fe87c0c6
[2020-07-29 12:40:19] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:40:21] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:40:25] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:40:26] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:40:26] ERROR sync: Could not send recent block request: i/o deadline reached
[2020-07-29 12:40:26]  INFO sync: Requesting parent block currentSlot=67490 parentRoot=70f8a3e2f405
[2020-07-29 12:40:30] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:40:31] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:40:35] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:40:36] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:40:36] ERROR sync: Could not send recent block request: i/o deadline reached
[2020-07-29 12:40:36]  INFO sync: Requesting parent block currentSlot=67590 parentRoot=86fd2d8d917f
[2020-07-29 12:40:40] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:40:41] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:40:45] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:40:46] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:40:46] ERROR sync: Could not send recent block request: i/o deadline reached
[2020-07-29 12:40:46]  INFO sync: Requesting parent block currentSlot=67646 parentRoot=2914e3534f4f
[2020-07-29 12:40:50] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:40:51] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:40:55] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:40:56] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:40:56] ERROR sync: Could not send recent block request: i/o deadline reached
[2020-07-29 12:40:56]  INFO sync: Requesting parent block currentSlot=67681 parentRoot=87e4fe87c0c6
[2020-07-29 12:41:00] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:41:01] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:41:05] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:41:05]  INFO sync: Requesting parent block currentSlot=67490 parentRoot=70f8a3e2f405
[2020-07-29 12:41:10] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:41:13]  INFO sync: Requesting parent block currentSlot=67590 parentRoot=86fd2d8d917f
[2020-07-29 12:41:15] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:41:18] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:41:20] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:41:23] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:41:23] ERROR sync: Could not send recent block request: i/o deadline reached
[2020-07-29 12:41:23]  INFO sync: Requesting parent block currentSlot=67646 parentRoot=2914e3534f4f
[2020-07-29 12:41:25] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:41:28] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:41:30] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:41:35] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:41:40] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:41:42]  INFO sync: Requesting parent block currentSlot=67681 parentRoot=87e4fe87c0c6
[2020-07-29 12:41:45] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:41:47] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:41:50] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:41:52] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:41:52] ERROR sync: Could not send recent block request: i/o deadline reached
[2020-07-29 12:41:52] ERROR sync: Could not process block from slot 67489: could not process block: could not execute state transition: could not process slot: expected state.slot 67496 < slot 67489
[2020-07-29 12:41:52]  INFO sync: Requesting parent block currentSlot=67490 parentRoot=70f8a3e2f405
[2020-07-29 12:41:55] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:41:57] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:42:00] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:42:02] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:42:02] ERROR sync: Could not send recent block request: i/o deadline reached
[2020-07-29 12:42:02]  INFO sync: Requesting parent block currentSlot=67590 parentRoot=86fd2d8d917f
[2020-07-29 12:42:05] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:42:07] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:42:10] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:42:12] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached
[2020-07-29 12:42:12] ERROR sync: Could not send recent block request: i/o deadline reached
[2020-07-29 12:42:12]  INFO sync: Requesting parent block currentSlot=67643 parentRoot=67f04363b66f
[2020-07-29 12:42:17] ERROR sync: Unable to retrieve block from stream error=i/o deadline reached

Proofs

How to mitigate?

Prysm nodes should temporarily ban IPs that flood their P2P port with the garbage traffic.

Note:

Lighthouse and teku are likely vulnerable too.

AlexSSD7 commented 4 years ago

How severe it is?

Let's assume that the eth2 network has 5000 validating nodes out there, and the average internet speed worldwide is 11 mbps.

Attack scenario:

To be sure that the node would be DDoS'ed - we will dedicate 22 mbps (2x speed) to each one.

Note that in real world speed would be adjusted for every node by determining the ISP's average speed (is it a home ISP, hosting, business, etc).

DigitalOcean's average droplet speed is 300mbps, so it should be enough to just rent 5000*22/300 = 36 cheapest droplets ($5 a month) to perform a massive attack.

So the total price of a day doing the attack is 36*5/30.5 = $5.90.

Please note that this is just an example. A real world attack would look a bit different.

protolambda commented 4 years ago

Hey! Thanks for reaching out. At the moment we are looking into some other attacknet issues, so responses may be slower than normal. Some quick feedback: At 300mbps, for a month: 300 * 10e6 * 30 * 24 * 60 * 60 / 8 / 10e9 = 97200 GB of data, per month. The digital ocean pricing calculator says 5 USD per month. But almost 1k USD in bandwidth costs (only accounting for outwards traffic here). Now times 36 to have more nodes do the same. It's not completely out of reach, but not as cheap as a mere 6 dollars. With some more hardening in prysm, to make it less effective, cost would increase as well.

AlexSSD7 commented 4 years ago

Yeah, you're right. Obviously, no one would use digitalocean to perform DDoS attacks because they can suspend the account.

I used DigitalOcean to simulate the botnet that black hats would have used.

AlexSSD7 commented 4 years ago

Update: the network was unfinalized for 38 epochs despite I have stopped the attack on ~20 unfinalized epochs. Some validators are down too.

nisdas commented 4 years ago

To update we had block proposals timing out for an extended amount of time

Beacon Node:

time="2020-07-29 02:07:25" level=error msg="Failed to close stream" error="read tcp4 172.31.2.203:9000->69.178.221.103:1024: read: connection reset by peer" prefix=sync
time="2020-07-29 02:52:25" level=error msg="Failed to close stream" error="read tcp4 172.31.2.203:9000->82.64.27.147:13000: read: connection reset by peer" prefix=sync
time="2020-07-29 03:10:25" level=error msg="Failed to close stream" error="read tcp4 172.31.2.203:9000->54.87.247.218:17076: read: connection reset by peer" prefix=sync
time="2020-07-29 06:16:50" level=error msg="Failed to close stream" error="read tcp4 172.31.2.203:9000->82.64.27.147:13000: read: connection reset by peer" prefix=sync
time="2020-07-29 06:20:36" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 08:21:50" level=error msg="Failed to revalidate peer" error="i/o deadline exceeded" peer=16Uiu2HAmNNrpDyCHqtRGU9CnvHuwGaRBaTLN8HffA226MnsK4n2A prefix=sync
time="2020-07-29 08:25:02" level=error msg="Failed to revalidate peer" error="i/o deadline exceeded" peer=16Uiu2HAmNNrpDyCHqtRGU9CnvHuwGaRBaTLN8HffA226MnsK4n2A prefix=sync
time="2020-07-29 08:28:14" level=error msg="Failed to revalidate peer" error="i/o deadline exceeded" peer=16Uiu2HAmNNrpDyCHqtRGU9CnvHuwGaRBaTLN8HffA226MnsK4n2A prefix=sync
time="2020-07-29 08:31:26" level=error msg="Failed to revalidate peer" error="i/o deadline exceeded" peer=16Uiu2HAmNNrpDyCHqtRGU9CnvHuwGaRBaTLN8HffA226MnsK4n2A prefix=sync
time="2020-07-29 08:53:50" level=error msg="Failed to revalidate peer" error="i/o deadline reached" peer=16Uiu2HAkxvVfxCgL9oUJG3uULpq9rEwzxEcVBzgix6maxe7oXtyV prefix=sync
time="2020-07-29 09:06:38" level=error msg="Failed to revalidate peer" error="i/o deadline reached" peer=16Uiu2HAmVssyKiSKoSekvb35AFeFB8jEBsDVYx4U8r3NJVc2NZii prefix=sync
time="2020-07-29 09:07:12" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 09:07:24" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 09:07:48" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 09:08:00" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 09:09:00" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 09:15:00" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 09:18:48" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 09:21:12" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 09:22:38" level=error msg="Failed to revalidate peer" error="i/o deadline reached" peer=16Uiu2HAmBDM7zQgPhYLR4W97a4CeQR9wNLymXLQyekT7hz8RN1e5 prefix=sync
time="2020-07-29 09:23:24" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 09:24:00" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 09:24:48" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 09:25:00" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 09:29:48" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 09:32:12" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 09:32:24" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 09:35:00" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 09:35:36" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 09:38:24" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 09:38:36" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 09:44:12" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 09:44:36" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 09:44:57" level=error msg="Failed to revalidate peer" error=EOF peer=16Uiu2HAmRt42NX7EHMGVRPdLmcwtdSye9ezdTh2KUXeaKQWGT8Gn prefix=sync
time="2020-07-29 09:45:24" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 09:45:48" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 09:47:00" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 09:47:36" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 09:48:10" level=error msg="Failed to revalidate peer" error=EOF peer=16Uiu2HAmRt42NX7EHMGVRPdLmcwtdSye9ezdTh2KUXeaKQWGT8Gn prefix=sync
time="2020-07-29 09:48:10" level=error msg="Failed to revalidate peer" error=EOF peer=16Uiu2HAmDmNWARMgdSNAq29hcGU9pmFMxWtMNR65ch9QfWj12KKY prefix=sync
time="2020-07-29 09:48:13" level=error msg="Failed to revalidate peer" error=EOF peer=16Uiu2HAmJaGbUaHdxAHduoMpEmEemcYUx4CXAJzryd5A8sLv4j6Y prefix=sync
time="2020-07-29 09:48:14" level=error msg="Failed to revalidate peer" error="i/o deadline reached" peer=16Uiu2HAmVssyKiSKoSekvb35AFeFB8jEBsDVYx4U8r3NJVc2NZii prefix=sync
time="2020-07-29 09:48:24" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 09:49:48" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 09:50:12" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 09:51:22" level=error msg="Failed to revalidate peer" error=EOF peer=16Uiu2HAmDmNWARMgdSNAq29hcGU9pmFMxWtMNR65ch9QfWj12KKY prefix=sync
time="2020-07-29 09:51:22" level=error msg="Failed to revalidate peer" error=EOF peer=16Uiu2HAmRt42NX7EHMGVRPdLmcwtdSye9ezdTh2KUXeaKQWGT8Gn prefix=sync
time="2020-07-29 09:51:23" level=error msg="Failed to revalidate peer" error=EOF peer=16Uiu2HAmJaGbUaHdxAHduoMpEmEemcYUx4CXAJzryd5A8sLv4j6Y prefix=sync
time="2020-07-29 09:51:26" level=error msg="Failed to revalidate peer" error="i/o deadline reached" peer=16Uiu2HAmVssyKiSKoSekvb35AFeFB8jEBsDVYx4U8r3NJVc2NZii prefix=sync
time="2020-07-29 09:54:34" level=error msg="Failed to revalidate peer" error="wrong fork digest version" peer=16Uiu2HAmJaGbUaHdxAHduoMpEmEemcYUx4CXAJzryd5A8sLv4j6Y prefix=sync
time="2020-07-29 09:54:34" level=error msg="Failed to revalidate peer" error=EOF peer=16Uiu2HAmRt42NX7EHMGVRPdLmcwtdSye9ezdTh2KUXeaKQWGT8Gn prefix=sync
time="2020-07-29 09:54:35" level=error msg="Failed to revalidate peer" error=EOF peer=16Uiu2HAmDmNWARMgdSNAq29hcGU9pmFMxWtMNR65ch9QfWj12KKY prefix=sync
time="2020-07-29 09:55:36" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 09:57:46" level=error msg="Failed to revalidate peer" error="wrong fork digest version" peer=16Uiu2HAmDmNWARMgdSNAq29hcGU9pmFMxWtMNR65ch9QfWj12KKY prefix=sync
time="2020-07-29 09:57:48" level=error msg="Failed to revalidate peer" error=EOF peer=16Uiu2HAmRt42NX7EHMGVRPdLmcwtdSye9ezdTh2KUXeaKQWGT8Gn prefix=sync
time="2020-07-29 09:59:00" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 10:01:48" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 10:03:00" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 10:04:14" level=error msg="Failed to revalidate peer" error="i/o deadline reached" peer=16Uiu2HAmVssyKiSKoSekvb35AFeFB8jEBsDVYx4U8r3NJVc2NZii prefix=sync
time="2020-07-29 10:04:48" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 10:07:48" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 10:14:48" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 10:15:00" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 10:16:12" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 10:18:00" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 10:20:12" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 10:21:36" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 10:22:24" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 10:27:48" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 10:28:12" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 10:28:48" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 10:34:00" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 10:36:00" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 10:36:14" level=error msg="Failed to revalidate peer" error="i/o deadline reached" peer=16Uiu2HAmVssyKiSKoSekvb35AFeFB8jEBsDVYx4U8r3NJVc2NZii prefix=sync
time="2020-07-29 10:38:48" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 10:39:36" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 10:42:12" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 10:43:55" level=error msg="Could not get rough time result: lookup roughtime.chainpoint.org on 172.31.0.2:53: read udp 172.31.2.203:38221->172.31.0.2:53: i/o timeout" prefix=roughtime
time="2020-07-29 10:43:55" level=error msg="Could not get rough time result: no reply" prefix=roughtime
time="2020-07-29 10:47:24" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 10:49:12" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 10:51:36" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 10:52:10" level=error msg="Failed to revalidate peer" error=EOF peer=16Uiu2HAmRt42NX7EHMGVRPdLmcwtdSye9ezdTh2KUXeaKQWGT8Gn prefix=sync
time="2020-07-29 10:52:10" level=error msg="Failed to revalidate peer" error=EOF peer=16Uiu2HAm7ukQsQ3Vx9ButibFzQ6eCJBnDDpe6QEMtfQKpXetexYq prefix=sync
time="2020-07-29 10:52:13" level=error msg="Failed to revalidate peer" error="wrong fork digest version" peer=16Uiu2HAmKb72XTXHBYYsBYWYsWuE5AeBeFw2awt518cWrwh4Anvf prefix=sync
time="2020-07-29 10:52:36" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 10:53:36" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 10:54:24" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 10:54:36" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 10:55:23" level=error msg="Failed to revalidate peer" error=EOF peer=16Uiu2HAm7ukQsQ3Vx9ButibFzQ6eCJBnDDpe6QEMtfQKpXetexYq prefix=sync
time="2020-07-29 10:56:00" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 10:58:12" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 10:58:34" level=error msg="Failed to revalidate peer" error=EOF peer=16Uiu2HAm7ukQsQ3Vx9ButibFzQ6eCJBnDDpe6QEMtfQKpXetexYq prefix=sync
time="2020-07-29 11:00:12" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 11:01:46" level=error msg="Failed to revalidate peer" error=EOF peer=16Uiu2HAm7ukQsQ3Vx9ButibFzQ6eCJBnDDpe6QEMtfQKpXetexYq prefix=sync
time="2020-07-29 11:02:36" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 11:03:12" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 11:03:48" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 11:04:58" level=error msg="Failed to revalidate peer" error=EOF peer=16Uiu2HAm7ukQsQ3Vx9ButibFzQ6eCJBnDDpe6QEMtfQKpXetexYq prefix=sync
time="2020-07-29 11:06:24" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 11:11:00" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 11:16:12" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 11:20:00" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 11:20:24" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 11:27:36" level=error msg="Failed to get block number from timestamp" error="context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 11:31:36" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 11:38:00" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 11:38:48" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 11:41:36" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 11:43:00" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 11:44:24" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 11:45:36" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 11:47:36" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 11:48:00" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 11:55:24" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 11:56:48" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 11:57:12" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 12:00:12" level=error msg="Failed to get block number from timestamp" error="Post "https://goerli.prylabs.net": context deadline exceeded" prefix="rpc/validator"
time="2020-07-29 12:25:29" level=error msg="Failed to close stream" error="read tcp4 172.31.2.203:9000->117.102.178.66:13000: read: connection reset by peer" prefix=sync
time="2020-07-29 12:48:26" level=error msg="Could not verify attestation" error="signature did not verify" prefix=sync
time="2020-07-29 12:49:38" level=error msg="Could not verify attestation" error="signature did not verify" prefix=sync
time="2020-07-29 12:49:50" level=error msg="Could not verify attestation" error="signature did not verify" prefix=sync
time="2020-07-29 12:49:50" level=error msg="Could not verify attestation" error="signature did not verify" prefix=sync

Validator

time="2020-07-28 06:36:26" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = InvalidArgument desc = invalid request: attestation slot 59604 not within attestation propagation range of 59571 to 59603 (current slot)" prefix=validator pubKey=0xb05912611a8a slot=59604
time="2020-07-28 06:36:30" level=error msg="Could not submit slot signature to beacon node: rpc error: code = Internal desc = Could not find attestation for slot and committee in pool" prefix=validator slot=59604
time="2020-07-28 06:36:38" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = InvalidArgument desc = invalid request: attestation slot 59605 not within attestation propagation range of 59572 to 59604 (current slot)" prefix=validator pubKey=0xab96f762e34f slot=59605
time="2020-07-28 06:36:42" level=error msg="Could not submit slot signature to beacon node: rpc error: code = Internal desc = Could not find attestation for slot and committee in pool" prefix=validator slot=59605
time="2020-07-28 06:36:48" level=error msg="Failed to propose block" error="rpc error: code = Internal desc = Could not process beacon block: could not process block: could not process slot from the future, slot time 2020-07-28 06:36:46 +0000 UTC > current time 2020-07-28 06:36:42.97448128 +0000 UTC m=+716003.074494488" prefix=validator pubKey=0xab862426225a
time="2020-07-28 06:36:50" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = InvalidArgument desc = invalid request: attestation slot 59606 not within attestation propagation range of 59573 to 59605 (current slot)" prefix=validator pubKey=0x90c2d8caa017 slot=59606
time="2020-07-28 06:36:50" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = InvalidArgument desc = invalid request: attestation slot 59606 not within attestation propagation range of 59573 to 59605 (current slot)" prefix=validator pubKey=0x93fecf223672 slot=59606
time="2020-07-28 06:36:54" level=error msg="Could not submit slot signature to beacon node: rpc error: code = Internal desc = Could not find attestation for slot and committee in pool" prefix=validator slot=59606
time="2020-07-28 06:37:02" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = InvalidArgument desc = invalid request: attestation slot 59607 not within attestation propagation range of 59574 to 59606 (current slot)" prefix=validator pubKey=0x92391f2f8f70 slot=59607
time="2020-07-28 06:37:06" level=error msg="Could not submit slot signature to beacon node: rpc error: code = Internal desc = Could not find attestation for slot and committee in pool" prefix=validator slot=59607
time="2020-07-28 06:37:12" level=error msg="Failed to propose block" error="rpc error: code = Internal desc = Could not process beacon block: could not process block: could not process slot from the future, slot time 2020-07-28 06:37:10 +0000 UTC > current time 2020-07-28 06:37:06.946614948 +0000 UTC m=+716027.046628108" prefix=validator pubKey=0xb8c30fe77f4c
time="2020-07-28 06:37:14" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = InvalidArgument desc = invalid request: attestation slot 59608 not within attestation propagation range of 59575 to 59607 (current slot)" prefix=validator pubKey=0xa89b0f62f290 slot=59608
time="2020-07-28 06:37:18" level=error msg="Could not submit slot signature to beacon node: rpc error: code = Internal desc = Could not find attestation for slot and committee in pool" prefix=validator slot=59608
time="2020-07-28 06:37:38" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = InvalidArgument desc = invalid request: attestation slot 59610 not within attestation propagation range of 59577 to 59609 (current slot)" prefix=validator pubKey=0x8a2076344a20 slot=59610
time="2020-07-28 06:37:42" level=error msg="Could not submit slot signature to beacon node: rpc error: code = Internal desc = Could not find attestation for slot and committee in pool" prefix=validator slot=59610
time="2020-07-28 06:38:00" level=error msg="Failed to propose block" error="rpc error: code = Internal desc = Could not process beacon block: could not process block: could not process slot from the future, slot time 2020-07-28 06:37:58 +0000 UTC > current time 2020-07-28 06:37:54.895913532 +0000 UTC m=+716074.995926635" prefix=validator pubKey=0x8116f9d43628
time="2020-07-28 06:38:26" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = InvalidArgument desc = invalid request: attestation slot 59614 not within attestation propagation range of 59581 to 59613 (current slot)" prefix=validator pubKey=0xb637a6cce122 slot=59614
time="2020-07-28 06:38:30" level=error msg="Could not submit slot signature to beacon node: rpc error: code = Internal desc = Could not find attestation for slot and committee in pool" prefix=validator slot=59614
time="2020-07-28 06:38:38" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = InvalidArgument desc = invalid request: attestation slot 59615 not within attestation propagation range of 59582 to 59614 (current slot)" prefix=validator pubKey=0xa0b3af7ad720 slot=59615
time="2020-07-28 06:38:38" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = InvalidArgument desc = invalid request: attestation slot 59615 not within attestation propagation range of 59582 to 59614 (current slot)" prefix=validator pubKey=0x8ff351cec33c slot=59615
time="2020-07-28 06:38:38" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = InvalidArgument desc = invalid request: attestation slot 59615 not within attestation propagation range of 59582 to 59614 (current slot)" prefix=validator pubKey=0x8496e00cb85c slot=59615
time="2020-07-28 06:38:42" level=error msg="Could not submit slot signature to beacon node: rpc error: code = Internal desc = Could not find attestation for slot and committee in pool" prefix=validator slot=59615
time="2020-07-28 06:38:50" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = InvalidArgument desc = invalid request: attestation slot 59616 not within attestation propagation range of 59583 to 59615 (current slot)" prefix=validator pubKey=0xa89b0f62f290 slot=59616
time="2020-07-28 06:38:50" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = InvalidArgument desc = invalid request: attestation slot 59616 not within attestation propagation range of 59583 to 59615 (current slot)" prefix=validator pubKey=0xa070a71012c8 slot=59616
time="2020-07-28 06:38:54" level=error msg="Could not submit slot signature to beacon node: rpc error: code = Internal desc = Could not find attestation for slot and committee in pool" prefix=validator slot=59616
time="2020-07-28 06:39:00" level=error msg="Failed to propose block" error="rpc error: code = Internal desc = Could not process beacon block: could not process block: could not process slot from the future, slot time 2020-07-28 06:38:58 +0000 UTC > current time 2020-07-28 06:38:54.890112797 +0000 UTC m=+716134.990125974" prefix=validator pubKey=0x920fb32fc5a7
time="2020-07-28 06:39:02" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = InvalidArgument desc = invalid request: attestation slot 59617 not within attestation propagation range of 59584 to 59616 (current slot)" prefix=validator pubKey=0x920fb32fc5a7 slot=59617
time="2020-07-28 06:39:02" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = InvalidArgument desc = invalid request: attestation slot 59617 not within attestation propagation range of 59584 to 59616 (current slot)" prefix=validator pubKey=0xaaa15caad703 slot=59617
time="2020-07-28 06:39:06" level=error msg="Could not submit slot signature to beacon node: rpc error: code = Internal desc = Could not find attestation for slot and committee in pool" prefix=validator slot=59617
time="2020-07-28 06:39:38" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = InvalidArgument desc = invalid request: attestation slot 59620 not within attestation propagation range of 59587 to 59619 (current slot)" prefix=validator pubKey=0xb526e9a055a1 slot=59620
time="2020-07-28 06:39:42" level=error msg="Could not submit slot signature to beacon node: rpc error: code = Internal desc = Could not find attestation for slot and committee in pool" prefix=validator slot=59620
time="2020-07-28 06:39:50" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = InvalidArgument desc = invalid request: attestation slot 59621 not within attestation propagation range of 59588 to 59620 (current slot)" prefix=validator pubKey=0x88c6c308f5d5 slot=59621
time="2020-07-28 06:39:50" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = InvalidArgument desc = invalid request: attestation slot 59621 not within attestation propagation range of 59588 to 59620 (current slot)" prefix=validator pubKey=0xb637a6cce122 slot=59621
time="2020-07-28 06:39:54" level=error msg="Could not submit slot signature to beacon node: rpc error: code = Internal desc = Could not find attestation for slot and committee in pool" prefix=validator slot=59621
time="2020-07-28 06:40:12" level=error msg="Failed to propose block" error="rpc error: code = Internal desc = Could not process beacon block: could not process block: could not process slot from the future, slot time 2020-07-28 06:40:10 +0000 UTC > current time 2020-07-28 06:40:06.710014217 +0000 UTC m=+716206.810027394" prefix=validator pubKey=0x84dd17e4b9c8
time="2020-07-28 06:40:26" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = InvalidArgument desc = invalid request: attestation slot 59624 not within attestation propagation range of 59591 to 59623 (current slot)" prefix=validator pubKey=0xa690f6af92cb slot=59624
time="2020-07-28 06:40:30" level=error msg="Could not submit slot signature to beacon node: rpc error: code = Internal desc = Could not find attestation for slot and committee in pool" prefix=validator slot=59624
time="2020-07-28 06:40:38" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = InvalidArgument desc = invalid request: attestation slot 59625 not within attestation propagation range of 59592 to 59624 (current slot)" prefix=validator pubKey=0x8c618479377e slot=59625
time="2020-07-28 06:40:42" level=error msg="Could not submit slot signature to beacon node: rpc error: code = Internal desc = Could not find attestation for slot and committee in pool" prefix=validator slot=59625
time="2020-07-28 06:40:50" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = InvalidArgument desc = invalid request: attestation slot 59626 not within attestation propagation range of 59593 to 59625 (current slot)" prefix=validator pubKey=0x8116f9d43628 slot=59626
time="2020-07-28 06:40:54" level=error msg="Could not submit slot signature to beacon node: rpc error: code = Internal desc = Could not find attestation for slot and committee in pool" prefix=validator slot=59626
time="2020-07-28 06:41:02" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = InvalidArgument desc = invalid request: attestation slot 59627 not within attestation propagation range of 59594 to 59626 (current slot)" prefix=validator pubKey=0x8a2076344a20 slot=59627
time="2020-07-28 06:41:06" level=error msg="Could not submit slot signature to beacon node: rpc error: code = Internal desc = Could not find attestation for slot and committee in pool" prefix=validator slot=59627
time="2020-07-28 06:41:14" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = InvalidArgument desc = invalid request: attestation slot 59628 not within attestation propagation range of 59595 to 59627 (current slot)" prefix=validator pubKey=0x8f8829801df1 slot=59628
time="2020-07-28 06:41:14" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = InvalidArgument desc = invalid request: attestation slot 59628 not within attestation propagation range of 59595 to 59627 (current slot)" prefix=validator pubKey=0xab96f762e34f slot=59628
time="2020-07-28 06:41:18" level=error msg="Could not submit slot signature to beacon node: rpc error: code = Internal desc = Could not find attestation for slot and committee in pool" prefix=validator slot=59628
time="2020-07-28 06:41:38" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = InvalidArgument desc = invalid request: attestation slot 59630 not within attestation propagation range of 59597 to 59629 (current slot)" prefix=validator pubKey=0x879bd7308de1 slot=59630
time="2020-07-28 06:41:42" level=error msg="Could not submit slot signature to beacon node: rpc error: code = Internal desc = Could not find attestation for slot and committee in pool" prefix=validator slot=59630
time="2020-07-28 06:41:50" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = InvalidArgument desc = invalid request: attestation slot 59631 not within attestation propagation range of 59598 to 59630 (current slot)" prefix=validator pubKey=0xab862426225a slot=59631
time="2020-07-28 06:41:54" level=error msg="Could not submit slot signature to beacon node: rpc error: code = Internal desc = Could not find attestation for slot and committee in pool" prefix=validator slot=59631
time="2020-07-28 06:42:26" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = InvalidArgument desc = invalid request: attestation slot 59634 not within attestation propagation range of 59601 to 59633 (current slot)" prefix=validator pubKey=0xad9db6ff6f77 slot=59634
time="2020-07-28 06:42:30" level=error msg="Could not submit slot signature to beacon node: rpc error: code = Internal desc = Could not find attestation for slot and committee in pool" prefix=validator slot=59634
time="2020-07-28 06:42:38" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = InvalidArgument desc = invalid request: attestation slot 59635 not within attestation propagation range of 59602 to 59634 (current slot)" prefix=validator pubKey=0x90c2d8caa017 slot=59635
time="2020-07-28 06:42:38" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = InvalidArgument desc = invalid request: attestation slot 59635 not within attestation propagation range of 59602 to 59634 (current slot)" prefix=validator pubKey=0x827a91c95bf7 slot=59635
time="2020-07-28 06:42:38" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = InvalidArgument desc = invalid request: attestation slot 59635 not within attestation propagation range of 59602 to 59634 (current slot)" prefix=validator pubKey=0x93fecf223672 slot=59635
time="2020-07-28 06:42:42" level=error msg="Could not submit slot signature to beacon node: rpc error: code = Internal desc = Could not find attestation for slot and committee in pool" prefix=validator slot=59635
time="2020-07-28 06:42:48" level=error msg="Failed to propose block" error="rpc error: code = Internal desc = Could not process beacon block: could not process block: could not process slot from the future, slot time 2020-07-28 06:42:46 +0000 UTC > current time 2020-07-28 06:42:42.715815143 +0000 UTC m=+716362.815828289" prefix=validator pubKey=0xab862426225a
time="2020-07-28 06:42:50" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = InvalidArgument desc = invalid request: attestation slot 59636 not within attestation propagation range of 59603 to 59635 (current slot)" prefix=validator pubKey=0xb8c30fe77f4c slot=59636
time="2020-07-28 06:42:54" level=error msg="Could not submit slot signature to beacon node: rpc error: code = Internal desc = Could not find attestation for slot and committee in pool" prefix=validator slot=59636
time="2020-07-28 06:43:00" level=error msg="Failed to propose block" error="rpc error: code = Internal desc = Could not process beacon block: could not process block: could not process slot from the future, slot time 2020-07-28 06:42:58 +0000 UTC > current time 2020-07-28 06:42:54.713747693 +0000 UTC m=+716374.813761079" prefix=validator pubKey=0xb05912611a8a
time="2020-07-28 06:43:14" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = InvalidArgument desc = invalid request: attestation slot 59638 not within attestation propagation range of 59605 to 59637 (current slot)" prefix=validator pubKey=0x86e7ab2e6027 slot=59638
time="2020-07-28 06:43:14" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = InvalidArgument desc = invalid request: attestation slot 59638 not within attestation propagation range of 59605 to 59637 (current slot)" prefix=validator pubKey=0x84dd17e4b9c8 slot=59638
time="2020-07-28 06:43:18" level=error msg="Could not submit slot signature to beacon node: rpc error: code = Internal desc = Could not find attestation for slot and committee in pool" prefix=validator slot=59638

The validator errors are interesting as we potentially have a internal clock mismatch, this will need more exploration on the root cause.

nisdas commented 4 years ago

Looking through our code the root cause seems to be a skewed internal clock rather than any issue with goerli timeouts when proposing a block.

AlexSSD7 commented 4 years ago

FYI, I tested the teku attacknet DoS issue on the prysm attacknet. It does not work as was working with teku.

prestonvanloon commented 4 years ago

Regarding clock skew, I would expect this to resolve itself after 1 to 2 hours after the attack stops. Perhaps we can resync the clock when we start to see a high volume of errors around clock related issues.

AlexSSD7 commented 4 years ago

@protolambda, @djrtwo, waiting for your review 😁

AlexSSD7 commented 4 years ago

@nisdas fixed it in this PR: https://github.com/prysmaticlabs/prysm/pull/6780

djrtwo commented 4 years ago

Great work @AlexSSD7! This qualifies for the $5k beta-0 prysm attacknet bounty. The clock skew is very interesting :)

We're a bit busy with preparations for the medalla testnet, but I'll circle back with you early next week to discuss payment and get you the leaderboard