Add `op-node-derivers` design doc

[protolambda]

Description

Design doc for refactor of the Driver/Derivation in op-node, and introduction of "Deriver" to make the core chain validation processes of the rollup-node more extensible. This supports new projects such as interop by providing cleaner abstractions for major new functionality.

[mslipper]

I'm curious what the plan is to roll this out. Is there a way we can roll this refactor out incrementally in order to reduce review burden and risk?

[sebastianst]

I'm curious what the plan is to roll this out. Is there a way we can roll this refactor out incrementally in order to reduce review burden and risk?

@mslipper I believe @protolambda presents a nice rollout plan in three phases towards the end of the proposal, where the first removes the EngineQueue, the second introduces the new pattern into the Deriver without changing underlying concurrency, and the third would then incrementally add concurrency.

[tynes]

I'm curious what the plan is to roll this out. Is there a way we can roll this refactor out incrementally in order to reduce review burden and risk?

@mslipper I believe @protolambda presents a nice rollout plan in three phases towards the end of the proposal, where the first removes the EngineQueue, the second introduces the new pattern into the Deriver without changing underlying concurrency, and the third would then incrementally add concurrency.

A POC implementation also hid the new logic behind a flag and kept the old logic such that its possible to incrementally roll out the software to production, ie we run some nodes that opt into the new code and then after being sure that its safe we can eventually remove the old code

[protolambda]

Alignment on:

• We address the EngineQueue situation first.
• We can take more inspiration from the reth builder API.
• Instrumentation feature of the deriver is good to have.
• We want an event-based API for derivers

Aware of:

• Need to keep fault-proofs synchronous. Code that we lift out of the derive package into the driver package, unless not used by Fault-Proofs, should be either synchronous or have a simple synchronous alternative.

Contention on:

• (Josh/Axel) How we adapt the specs. See here Two viewpoints:
  • Do not enshrine implementation in the spec, or anything like this structure.
  • Do not enshrine implementation, but do enshrine the abstract state-changes, such that we have an overview of these in the specs.

Confusion on: the exact usage of Resource / OnEvent:

• Axel:
  • Should we call Resource an Inbox?
  • A: A resource is not the final receiver of the events, but rather ideal executing thread.
  • Should we call event Message?
  • A: Not every event has a prescribed destination, so no.

  • The Resource worker is essentially a router/prioritizer here? This helps explain my earlier confusion about the relationship between Resources and Workers a bit.

  • A: It "routes" in the sense that it runs the deriver stack. It's parallel to other workers, who can focus on modifying other resources.

  • Are there any cases where work has multiple contentious Resources that aren't managed by just a single Resource worker?

  • A: We can implement them, but ideally we create a new resource, to communicate state, rather than sharing state to communicate.
• Adrian:
  • Stacks:

  • It would be good to avoid having to have a interop+derive, interop+fp, altda+derive, altda+fp kind of stacks. So I think we want to decouple how message passing happens from where the message is handled (which this part implies but doesn't seem to fit with some of the stuff above).

  • A: Composability is meant to make these different stacks very lean / easy to create.
  • Event routing:

  • I'm unclear on how the OnEvent implementation is controlled.

  • Resource is meant to signal which thread runs the OnEvent call. There is still only one instance of the deriver stack.
  • Concurrency:

  • It would definitely be very useful to have a clear set of threading expectations around event handling. For example does OnEvent have to be handled synchronously or could it be async? Is that deriver dependent?

  • A: agreed, something that we can improve. The driver is meant to handle anything concurrent. The derivation is meant to handle anything synchronous, and there should be very minimal work to do between the derivation and the engine, such that almost none of the driver functionality has to be fault-proven.
• Seb:
  • So each resource has exactly one worker?
  • A: yes
  • What's the relationship between resource workers and derivers
  • A: worker executes, deriver maintains state and defines what happesn to an event.
  • Where are the derivers registered?
  • A: They are not "registered", but rather composed into a single root deriver.
• (Seb/Axel) Do we want a "decoupled pub/sub architecture"?
  • Would there be some registration process at the top level, and then on an incoming event a loop would go over all registrants and call OnEvent?
  • A: No, composition of event-handlers can serve the same. The OnEvent should maybe be renamed, it's meant as a standard interface, such that derivers are all compatible and easy to wrap.
• "If we don't have registration, how would we achieve the decoupled modularity?"
  • A: Components can be composed, such that they receive the events that are meant for them. Components do not have to dynamically register themselves during runtime. The Deriver interface allows any component to be composed as a target for events to be emitted into.

Proposal:

• Don't use channels in derivers, keep them fault-provable.
• Introduce an interface for emitting events, attached to the root driver (not any deriver). This can then target the right worker-thread for the given resource, if async, or just execute synchronously.

  type Emitter interface {
  EmitEvent(event *Event, dest Deriver)
  }

• Rename Resource to something that better describes "thread concerned with single L1 or L2 block-tag label".