how does pos ethereum prevent bribery for late block proposal?

TLDR: There exist a financial incentive for the execution payload to be proposed later than the anticipated time window for it. We worry that this might lead to favoring larger validators, encourage attester bribing infrastructure and might contribute to making consensus attacks described in recent papers more likely.

Context: The beacon chain has slots. A slot is currently every 12s. Within this slot exists an execution chain block as well as consensus objects. While there is theoretically (12s-epsilon) time to put the beacon block together within the slot, there are actually sub-slots within this slot that are softly enforced by consensus clients. These sub-slots are necessary in order to account for propagation time (incl. network delays) to attesters and for the proper construction of a beacon block.

Afaik - an execution chain block is supposed to be proposed in the first 4s of the 12s slot. This gives 8s for it to be voted on by attesters and in MEV-Boost, leaves enough for the body to be revealed by the end of the slot. For eg. see waitToOneThirdOrValidBlock() in Prysm's client here: https://github.com/prysmaticlabs/prysm/blob/8c8f1bb9c18ebbc42241848a328ed275b12587e5/validator/client/attest.go#L250 (thanks @terencechain )

Worry 1: Late execution block proposals: In theory, the proposer has until 12s-epsilon to submit a beacon block. This means the execution chain block could be constructed maybe at 11s rather than 4s in. This time difference means the builder would have an additional 7s to construct a block. In those 7s, new transactions could’ve come in and made the block more profitable.

The proposer and builders now have an incentive to delay the block submission as much as possible.

This is problematic because:

a proposer that controls a lot of validators can inherently force them to vote later (by modifying their consensus client source codes) vs a solo validator. This means there is an advantage to controlling a lot of validators that are well-connected to the p2p network.
another flavor of this problem is that there might exist an incentive to bribe attesters in order to delay their attestations.

Questions:

on average, how much more profitable are blocks after X seconds from Flashbots data? This will help us understand how big of an advantage would larger validators get.
what is the technical complexity of either bribing attesters and making sure the attesters attest later than expected?
what is the financial risk encountered by doing this (later attestations get less rewards + there is a risk of missing the slot afaik which would incur a slashing penalty)? can we surface and calculate the expected value of doing this?
Given this information, how expensive would it be to bribe attesters?

The good news is that we have numbers to quantify this, in particular, Flashbots finds that on average, each additional second of waiting is worth 0.034 eth in additional miner payment (incl. 1559 tips). The outlier numbers we have (top 0.1% of blocks excluding tips - only bundle payments) are around 1 eth/s (thx @metachris and @Ruteri ). These numbers generally seem too small for this worry to be relevant.

The bad news is that this quantification is imperfect. In particular, if someone can delay their ultimate proposal to a few seconds later, they are able to do cross-domain arbitrage with low-latency domains more effectively. Priced in, this might be worth large sums of money.

To-do:

[ ] poll relevant parties (searchers who run these type of strategies) to understand what this could amount to for them.

Worry 2: Reorgs Two papers (https://arxiv.org/pdf/2203.01315.pdf and https://eprint.iacr.org/2021/1413.pdf) were recently released detailing 5 attacks on PoS consensus (co-authored by @casparschwa @barnabemonnot et.al). One attack in particular relies on the timing of a block proposal.

We worry wether the financial incentives outlined above exert an upwards pressure on the likeliness of such a timing attack happening in the wild. I understand these attacks have been fixed and suspect this worry can be easily alleviated by @casparschwa sharing how these attacks were fixed.

We open the discussion here and look forward to your questions and comments :)!

Worry 1: Late execution block proposals:

Thanks for bringing up this issue on timeliness of block proposals. This is a known issue with the protocol and active exploit of this issue will lead to accelerated mitigation. However, it's a known problem for which we have a solution in mind already. The solution is (block, slot)-voting, it just needs some details figured out.

In this HackMD you'll find a much more detailed write-up of the problem, what this means for proposers/builders and what solutions could look like (there are more ideas!).

tldr:

A block proposer can release as late as 11s into the slot and still end up canonical. The only constraint is that the next block proposer hears the late block so that they build on top of it (as opposed to the late block's parent).
However, if a proposer chooses to release very late virtually no attesters will vote for their late block (because they will have attested already). This opens the late block up to reorg attacks. The next proposer can use something called Proposer LMD Score Boosting (essentially as if some validators had already attested to this block) to reorg out a preceeding that has less votes accumulated than the size of the boost. This effectively introduces a soft-cap for the block releasing deadline around the 4s mark (which is the time when hoenst validators attest, if they havent heard a valid block already).
Both builders and proposers are interested in colluding since more tx mempool listening time implies more MEV, which is good for both parties involved. The questions is whether they can collude without trusting each other? And I think the answer is yes: Builders can simply send new bids as time goes by and the proposer just picks the best whenever they decide to finally build and release their block. This is good in so far that there is no advantage to being a large, reputable staker (builders trust you) vs being a solo staker (builders don’t trust you), but it is bad in so far that it is therefore more likely to happen in the wild.
One may wonder whether a relay could somehow enforce honest release timing? For example, the relay could simply refuse to forward late builder bids. However, this incentivizes the creation of a new relay that is trusted by builders/proposers and allows untimely timing strategies. Relays only need to be trusted by builders and proposers, and it’s irrelevant whether they’re value aligned. Generally this competitive pressure should make them stay away from being too restrictive. Further, relays are overall quite limited in what they can enforce, e.g. builders can release their payload themselves (should the relay refuse to because the proposer was late)…

Worry 2: Reorgs

The three attacks mentioned in our paper are fixed by the aforementioned Proposer LMD Score Boosting. One of the attacks in the other paper by Neu et al. is not an attack on PoS Ethereum. The other one is fixed by removing equiovcating validators from fork choice consideration. I am happy to go into more detail on this here or elsewhere, but didn't want to go beyond scope in this comment.

Some other comments

There is actually one more "sub-slot":

At the beginning of the slot a block proposer is supposed to propose a block.
Committee members are supposed to attest when they hear a valid block or 4s into the slot -- whatever comes first.
Aggregators are supposed to release their attestation aggregates 8s into slot.

This means that if the aggregators don't hear your attestations some time before the 8s mark they will likely not include them, meaning you're incentivized to release them before the 8s mark (if you want them to be included asap).

A proposer that controls a lot of validators can inherently force them to vote later (by modifying their consensus client source codes) vs a solo validator. This means there is an advantage to controlling a lot of validators that are well-connected to the p2p network.

Both solo and multi-validators can modify the source code. It is not clear to me how there is an advantage only available to one or the other. Controlling lots of validators does not mean you control lots of nodes on the p2p layer. You can, but typically you'll use the same beacon node that handles the p2p stuff. And again, both a solo- and multi-validator could run multiple p2p nodes.

One advantage that someone controlling many validators has: They are more likely to also control an aggregator in the relevant (committee, slot)-pair in question. This means a large enough adversary could potentially circumvent the 8s deadline without foregoing inclusion rewards.

And finally, one question for you: Rather than an average value for what each additional second of waiting is worth, do you have data on the distribution of this?

I wanted to post this already, but I'll think more about some of the other questions mentioned. Let me know, if there are follow-up questions etc.

However, if a proposer chooses to release very late virtually no attesters will vote for their late block (because they will have attested already). This opens the late block up to reorg attacks.

To expand on this.

If proposer of slot N is regularly willing to be "dishonest" (deviating from spec and normal software) and wait well past the 4s mark, then it is reasonable to believe the proposer of slot N+1 will (also be dishonest) utilize their proposer boost to re-org that proposer (by building on N's parent) and to steal MEV from proposer at slot N.

So it is not actually clear that waiting near or past the 4s mark is a viable strategy to get more MEV due to this strategy being attackable by the next slot proposer.

Beyond that. A general note here on this security issue. Mentioned by @casparschwa but worth being very clear what's at stake here.

Ethereum PoS has a number of known upgrades in the pipeline. Some are feature enhancements (e.g. sharded data) while some are security enhancements (e.g. proof of execution, DAS, etc). While we (researchers, developers, community) would like to prioritize shipping features, if known security issues become active targets, security enhancements will have to be prioritized

Manipulating the release of blocks between 4 and 11s in a slot is a known degenerate case with a proposed fix (called (block, slot) fork choice). This is explicitly against the "honest" spec and would (1) fairly attributable and (2) seen as nefarious behavior. In the event that flashbots, some large pool, or some other mechanism begins to exploit this issue to shift the timing equillibrium, it will accelerate the priority of shipping (block, slot) at the expense of important (desired?) scalability features.

The good news is that we have numbers to quantify this, in particular, Flashbots finds that on average, each additional second of waiting is worth 0.034 eth

I'm curious about how much the honest block that builds on top of the late block would degrade and whether that's something we can quantify. For example, say at slot N, a dishonest proposer releases block at 11s and gains X more MEV. Would it be fair to estimate that at slot N+1, the honest proposer releases block at 0s and loses X MEV that it would have gained? Assuming both proposers use mev-boost and connect to different relays or same relay (would that make a big difference?)

There's also an active prototype by the lighthouse team to combat this: https://github.com/sigp/lighthouse/pull/2860 If it goes well, it'll likely land itself into the honest validator spec and all clients will implement it.

Hey everyone! Thanks a lot for you comments here, and thanks in particular to @casparschwa and @fradamt for the conversations offline and the detailed answers above. They've been critical in writing this reply!

TLDR:

there are some mitigations for late block proposals including Proposer LMD score boosting, and in the future (block,slot) voting.
- however none of them protect against a degenerate case where the network without needing coordination pushes block proposal and attestation times further and further.
- this is an issue to further study, as well as to potentially monitor for by setting up infrastructure to measure time at which blocks are proposed within a slot and to call out any misbehavior publicly.
other tangential research questions to study are: retroactive rewards for timely block proposals, quantifying how much money can be made from delaying block proposals, mev oracles as a scaling factor to help retroactive timely block proposal rewards and whether deterministic block times are a fundamental flaw of PoS Ethereum.

One misunderstanding corrected privately that I'll relay here is an attestation of slot n can only get included in the block from slot n+1 (the earliest).

You'll find below further details on all of the above:

Mitigations (summarizing the above - feel free to skip to the 'additional comments' and later sections)

Currently, there is no network-wide mitigation for late block proposals that's implemented and all a proposer has to do is to make sure that the next proposer hears their block (block n) by the time they start constructing their block (block n+1). This means proposers today can propose a block very late in their slot.

One immediate mitigation that's implemented in some clients is Proposer LMD score boosting. This was implemented for another issue than late block proposals but turns out to be helpful for our issue too. In a nutshell, this would allow the next proposer to re-org the previous one out if they proposed their block too late.

How does it work? If you don't accumulate any fork choice weight because you’re releasing your block so late that the entire attester committee doesn’t see it in time, then you’re prone to be reorged out by the next proposer. If you want to be reorg-resistant then you need to release it in such a way that your late block accumulates at least PROPOSER_BOOST votes, which brings you back to around a 4s deadline. One outstanding question here is how to set the boost value correctly. Read more about it here

An advanced version of that mitigation that avoids relying on the proposer but instead on the attester committee is (block,slot) voting. This currently has a rough spec but if i understand correctly needs more work before being implemented. In (block,slot) voting, committee members can vote on an empty slot (which they weren't able to previously), meaning the empty slot now accumulates fork choice weight making it harder for a late block to become canonical. This enshrines the 4s rule in-protocol, without needing proposer LMD score boost.

Finally, there are other mitigation discussions at an experimental stage like retroactive timely rewards for block proposers, effectively changing the components of the block proposer reward to include timeliness. One problem with this is that it is only effective if this reward is higher than the reward from a proposer stands to make from 'late MEV' which makes it problematic since MEV rewards can shift while this parameters may be static.

You can learn more about these mitigations in this document from @casparschwa here: (Un)-Timeliness in PoS Ethereum

Some additional comments:

On @casparschwa 's point about the potentiality of relays enforcing proposal deadlines on builders: I agree with your intuition. It would make such relays less competitive. We've already seen similar dynamics happen today among different relays on PoW Ethereum.
On @terencechain's point, and also @djrtwo's point: absolutely!
- if this is a network-wide behavior, large players (large wrt to stake ownership) would be stealing from one another.
- this is easily detectible, the one who is constantly spoiling the pot should be able to get told off.
- on the other hand, if it's a solo validator it's totally fine because they're a proposer so rarely that such a behaviour wouldn't have a material impact unless it becomes endemic.

how would you be able to detect this behavior?

you could detect it by the value of the proposer's MEV and the content of their block. if their block includes txs at the 4s mark and beyond then you know they were still constructing it then. this would require a node setup to benchmark when txs are received in the mempool and some statistical analysis.
- one heuristic that can work to mitigate edge cases of this first one would be that if it's a big validator who's constantly late, it's worth asking why.
meta-question: should monitoring infrastructure be set-up for this/should this be part of the monitoring infrastructure on the network? (tagging @elopio, @epheph and @taarushv here)

So, can we close this issue?

I don't think so! There is one oustanding concern worth discussing:

Even if proposer-boost is implemented network-wide, and even with (block,slot) voting, there is still an incentive for proposers to propose as late as they can within the boundaries given to them (ie. as close as possible to the 4s mark).

Because the block would be proposed around the 4s deadline, some attesters would likely miss it as they vote on the latest head they've seen. This would affect their total rewards since voting on timely head gives 14/64 of total validator rewards (so roughly 10% of their reward).

The ideal strategy for attesters is that if they're protected from 'proposer_boost', they might as well wait 0.5s later to make sure they're in the group that sees the proposer slot. The future proposers now might push their proposal to 4.5s rather than 4s. You can imagine this game being iterated, pushing block proposal deadlines further and further without any network-wide coordination needed. (Read more about this in here by @fradamt https://notes.ethereum.org/HO8UXG4rSdmH4Sv16iWKuw?view#Shifting-timing-equilbria-of-block-releasing-amp-attestations )

There is more complexity to this problem to think about such as:

if the network doesn't move in unison when they push this limit, it becomes quite complex to think about attestations and things like timing attacks.
there are more network-dynamics to consider, like:
- solo validators having to monitor the network to figure out that this behavior is happening,
- solo validators potentially not being able to catch up because they would constantly have to download a client that's incentive compatible (ie. with updated deadlines)
- the advantage of large validators who would likely control an attestation aggregator and are able to play around the aggregator deadline that's at 8s in the slot which solo validators wouldn't be able to

If the network does move in unison doing this behavior, should we expect a drift over time of block proposals and attestations?

So where does that leave us?

There are multiple solutions to mitigate late block proposals. Yay!
This behavior is detectible and can be addressed publicly, where players doing this can be called out.
However, none of these solutions are silver bullets and there is still an underexplored worrisome degenerate case.

Further research

Quantifying MEV: Having a better idea of the amount of MEV players stand to make would allow us to think about these mitigation solutions better. In particular it would help us think about the usefulness of retroactive timely rewards for block proposals. Generally, estimating what isn't in the numbers which I mentioned above would be helpful although seemingly difficult to achieve. (ie. quantifying cross-domain MEV for later block proposals)

Understanding the degenerate case better: A bunch of questions were listed above as part of understanding the dynamics of the iterated game better.

It's also worth asking the question, why is it bad if blocks are proposed later? I'm asking this question because I'd like to be able to point to something written to explain why a validator/validator pool who's acting under economically rational assumption shouldn't do this. From all our conversation above, the answer seems to be:

it would significantly complicate reasoning about balancing attacks.
it is likely useless if the network does it in unison (it just delays slots in perpetuity)
it may enforce higher hardware & software requirements on validators and less tolerance on network delays and propagation. This means the network becomes less accessible to players that don't have the best connections or professional set-ups. This goes against the ethos of the system we're building.
it is unwanted behavior that if noticed, will become a higher priority to solve and as @djrtwo mentioned, will be prioritized against efforts that stand to make Ethereum better by adding new features to it.

Against deterministic block times? I wonder if this is simply a fundamental flaw of chains with deterministic block times, and that current PoW Ethereum's variable block times is actually better for decentralization. If you start from the assumption that users will always want to push the limit of how far they can wait to propose a block because of economic incentives, variable block times introduce uncertainty which was helpful to prevent this behavior. While I don't have more thoughts here at the moment, I wonder if ethereum will be the first big test of a sophisticated (wrt to MEV activity) PoS network with deterministic block times and whether there may be conclusions to draw on the experiment. I also know of efforts like Algorand's PoS chain that mimics PoW-like probabilistic leader election that may be worth studying further.

flashbots / mev-boost