Open ImTei opened 1 month ago
Thanks @ImTei for tracking the issue on this edge case we found!
I do agree this might not be handled automatically in first instance.
I think could be a good opportunity to spin-up a playbook/runbook section in the docs to describe this and other scenarios that might realize, providing best practices (at conduit we have internally).
Another example, blob congestion => switch to calldata, make a blob transaction type to cancel the pending one (see LZ airdrop).
Another example, blob congestion => switch to calldata, make a blob transaction type to cancel the pending one (see LZ airdrop).
@emilianobonassi This is already done automatically by the batcher since https://github.com/ethereum-optimism/optimism/pull/10941
High level description
If op-batcher stops batch submission for a while and the sequence window is expired, op-node starts to generate empty blocks. But even if the op-batcher becomes operational, it's hard to recover batch submission and chain derivation because new batches will be likely dropped. This issue includes details of the situation and proposed solution.
Details of the incident
Repeat 2~6
Currently, we have to do following things manually to recover the chain from the incident.
We may add these steps to the runbook, but we can improve system to automate these steps.
Solution
incident mode
for now.incident mode
is enabled when the op-node generates empty block. and disabled when the op-node derives new block from L1 batch.incident mode
, sequencer builds empty blocks by settingNoTxPool
astrue
.incident mode
is included as a boolean value in theoptimism_syncStatus
RPC response.incident mode
bysyncStatus
RPC. If the op-node is inincident mode
, op-batcher builds singular batch even if it's running as a span batch mode. (Or we can make it build span batch far from the current safe head to avoid sequence window expiration)Discussion
This change can automate the incident recovery of OP stack chains, but it may be a bit risky because it's touching a lot of important features like sequencing and batch submission. Because this incident situation is very unlikely, we can consider more manual way to recover the system from the incident.