RFC: Top-level rollup API (Sequencer)

[rpanic]

This will be the "Shell" of the rollup, which will allow the user to configure non-circuit components and also pass in the actual AppChain. This API will then take care of:

• Starting up the rollup services, including but not exhaustive:
  • GraphQL
  • Database
  • Workers
• Deployment of the on-chain contracts
• etc.

[rpanic]

As a starting point, I am suggesting following high-level API:

Every component exposed to the sequencer will have the following interface

/**
 * Lifecycle of a SequencerModule
 *
 * bind(): Executed after creating the module and allowing the module inject additional dependencies / configuration
 *
 * start(): Executed to execute any logic required to start the module
 */
export abstract class SequencerModule<Config> {
  public abstract start(runtime: Sequencer): Promise<void>;
  public abstract bind(
    builder: SequencerBuilder<any>,
    config: Config
  ): void;
  public abstract get name(): string;

  public abstract defaultConfig(): Config;
}

const sequencerModules: SequencerModules = {
  mempool: PrivateMempool,
  modules: [GraphQLModule, PostgresModule, AWSWorkerPool] //In practice these would be enabled by default or provided via template modules (basically packages of modules)
}

sequencerModules.loadConfigFile("config.yml")
//or
sequencerModules.pushConfig({ graphql: { port: 8080 } }) //Type checked

Sequencer.fromModules(sequencerModules)
  .start()

Why seperate it in that way? In order for the types to be passed on, we need the Definition first (i.e. have a list of modules)

If a user doesn't need to manipulate the config, it can be shortened to

Sequencer.fromModules({
  mempool: PrivateMempool,
  modules: [GraphQLModule, PostgresModule]
}, config)

Pros:

• Type safety for config manipulation
• Concice API which makes starting a sequencer really easy and still customizable

Questions / Conerns:

• Is type safety that big of a benefit that we need to complicate the Sequencer building that much?
• If a SequencerModule injects own dependencies in bind(), it has to extend the Config Type, otherwise the outside config type will not know the config properties of the child-module. This could be enforced with generics though
• The sequencer depends on the DI-container it received (through whatever means) to contain the protocol and appchain instances

[maht0rz]

I think we should look at the sequencer package design from top to bottom, rather than from bottom to top. What this means is i'd personally start by determining what the responsibilities of a sequencer are and how does it fit into the bigger picture.

The following separation is what comes to my mind:

• AppChain is the highest level entity in our system, consisting of Runtime and Sequencer
• Runtime consists of runtime modules which hold all the required business logic
• Sequencer is the orchestration layer, providing end user access to the runtime via e.g. our gql API, which can be separated into mostly reading data and sending transactions

AppChain itself is just a wrapper API, which could look like the following:

const appChain = AppChain.from({
  runtime: Runtime.from({ modules: [Balances, Governance] }),
  sequencer: Runtime.from({
    storage: PostgresStorage,
    mempool: PrivateMempool,
    api: GraphQLAPI
  }),
  config: {} // shall we pass a config here?
})

Alternatively with in-line config:

const appChain = AppChain.from({
  runtime: Runtime.from({ modules: [Balances, Governance] }),
  sequencer: Runtime.from({
    storage: PostgresStorage.from({
      withP2PReplication: true
    }),
    mempool: PrivateMempool.from({
      withInstantConfirmation: false
    }),
    api: GraphQLAPI.from({
      plugins: [GraphQLCORSPlugin]
    })
  }),
})

Question now is, what kind of modularity do we want to offer? From your modules design, i'm not sure what the lifecycle looks like and what the 'injection/hook' points are for these modules to come into effect - and i think thats where we should start thinking about modules/extensions/plugins. Hence i recommend going top-to-bottom and understanding what customisation can we offer at certain architectural components.

One way to think about it is to let AppChain accept a certain shape of a config, where things like Runtime or Mempool are interfaces to which the components must adhere to, that would be the first layer of modularity.

Second layer of modularity, is allowing config or rich injection points for plugins/modules into certain lifecycles of each plugin. I can imagine we allow a plugin architecture, my hunch is that re-using the name modules is a bad idea since we already have runtime modules, and calling it again an appchain module might be confusing. Perhaps we call it a lifecycle module?

Anyways, here's how i imagine a lifecycle plugin could look like:

AppChain.from({
  ..theUsualConfig,
  plugins: [Logger]
})

class Logger implements MempoolLifecycle {
  // this method gets called from the actual PrivateMempool implementation, or any other Mempool-like implementation that respects the lifecycle plugin architecture
  public onTransactionReceived(transaction: Transaction): Bool {
    return Bool(true);
  }
}

After we agree on how do we want to separate architectural responsibilities to achieve the optimal modularity, then we can focus on how to enable e.g. JSON-ish stringified configuration files that could persist the entire AppChain configuration. But i don't think we should prioritise config serialisation over a simple .ts file to create and start the whole AppChain, including the sequencer parts.

To sum things up:

• I suggest we define 'what components does an app chain need' to function, e.g. runtime, sequencer
• then its up to the sequencer to define what kind of components does it need to function, e.g. mempool, public facing api, storage
• and then its again up to these individual adapters to define what kind of config do they support etc.
• and the cherry on top is that the AppChain could accept lifecycle plugins, that can tap into any part of the lifecycle of either AppChain or Runtime

[rpanic]

Above spec was agreed on, with following changes:

Sequencer.fromModules(
 {PrivateMempoolModule, GraphQLModule, PostgresModule}
, config)

Runtime injection The sequencer needs access to the Runtime, and ideally the runtime modules. This can be accomplished by resolving all modules via tsyringe- therefore it can be injected with @inject(Runtime)

Additionally, we want to provide a default sequencer configuration called DefaultSequencer that has certain components already pre-configured. This can be PrivateMempool, DefaultGraphQLModule, PostgresStorageModule, etc.

The top-level AppChain API will take Sequencer as a parameter, if it is not set, the DefaultSequencer will be used.

Sequencer interface The sequencer interface has only one method

interface Sequencer {
  start: () => Promise<void>
}

Therefore, the Module-based Sequencer implementation will have the responsibility to ensure the SequencerModule lifecycles. Sequencers will always be resolved with tsyringe and should resolve dependencies with tsyringe.