(1-2) Extracting sequencer code from the gateway

[ppedziwiatr]

The sequencer code should be fully extracted from the gateway code and considered as an independent application. The extracted version should be rewritten in Go. Original version is available here: https://github.com/warp-contracts/gateway/blob/main/src/gateway/router/routes/sequencerRoute.ts

Instead of arweave-js - use goar lib https://github.com/everFinance/goar Instead of vrf-js - use https://github.com/google/keytransparency/tree/master/core/crypto/vrf

Instead of server timestamp (duh...), use a dedicated database sequence - obviously it MUST start from the latest value created with the current solution (this might be a pain during production deployment). Using database sequence will allow us to run more sequencers and rotate them periodically (or simply use round-robin or whatever) - https://community.optimism.io/docs/protocol/#decentralizing-the-sequencer - as a first step to make sequencers a bit more decentralized (or at least remove the single point of failure - from the application perspective).

There's also an option to use a server timestamp - but then it MUST be fully synced among all the sequencers.

In case of Arweave block height - use Warp gateway https://gateway.redstone.finance/gateway/arweave/info and https://gateway.redstone.finance/gateway/arweave/block The value needs to cached for some 'short' - 10 or 20s - period of time!

Logging is very important - the performance, different log levels, etc.

[ppedziwiatr]

Writing down stuff after today's discussion... The part of the sequencer extracting task should be a change in how transactions are being sent to Bundlr.

Current flow:

1. generate sort key and tags for the bundle data-item
2. set original tx as "data" of the data-item
3. sign and send transaction to Bundlr
4. after getting proper response from Bundlr - the tx data is indexed in Warp Gw.

This solution is simple and..works, but has few issues:

1. TPS is limitted by response times from Bundlr (usually ~250ms)
2. it's difficult to properly implement the "last_tx" feature (i.e. each newly created tx should have a previous tx id set in tags - this will allow the clients to properly validate whether they received all the required transactions - and is crucial for implementing the pub/sub model from https://github.com/warp-contracts/gateway/issues/88) With the default model we would need to store somewhere in the "global" sequencer state the lastly assigned value. But even then we might have an issue in some edge cases, e.g.: 1. the current last_tx is set to tx1 2. user1 one sends the tx2 - the tx2.last_tx is set to tx1 and the current last_tx is set to tx2 3. user2 sends the tx3 - the t3.last_tx is set to tx2 and the current last_tx is set to tx3 4. sequencer sends the tx2 to bundlr 5. sequencer sends the tx3 to bundlr 6. bundlr fails for the tx2 7. bundlr responds properly for the tx3 - which now has the last_tx set to tx2 - which wasn't properly bundled. 8. 💥

In order to fix these issues, we must split the sequencing into to two separate, independent functions.

1. The first function will simply register the transactions send by users to the sequencer - i.e. it will generate the sort_key and immediately index the new tx in GW's cache/db. The new tx should have the "bundling status" set to not_bundled
2. The second function should run periodically, grab all transactions (starting from the "oldest") that are in not_bundled status, and send them to Bundlr(*). I.e. it should:

1. grab the transaction 2. set its status in db to processing 3. if point 2. fails, retry n-times. If it fails after n-th try, move to next tx. 4. if point 2. succeeds, send the transaction to Bundlr 5. if point 4. succeeds (i.e. we get a proper response from Bundlr), update the "bundling status" in db to bundled. 6. if point 4. fails,, update the "bundling status" in db to not_bundled 7. if either point 5 or 6 fails (i.e. updating the status in our db) - retry n times. If it fails after n-th time - log error, send notification (via email, discord, etc) and leave the transaction in the processing status

The likelihood of a failing status update in our db is very low - but it's not impossible. The processing status will inform us, which transactions should be either verified manually - or automatically, by another scheduled task, that would:

1. grab all the tx in processing status (once a day)
2. for each of such transaction - verify if it is are available either in arweave.net cache or Bundlr node cache
3. if it is available - set its status to processed
4. if not - set its status to not_bundled

[ppedziwiatr]

Re. Bundlr(*) - this would allow us to (at some point ;-)) effectively implement our custom bundling process, without having to rely on Bundlr (e.g. using the the https://github.com/everFinance/arseeding#arseeding library). I.e. we could create bundles that would contain only the data items with contract interactions, generate tags that would greatly simplify indexing/searching of the bundled interactions via arweave.net GQL endpint, etc.

[Max-Levitskiy]

We use github.com/dappley/go-dappley/crypto/keystore/secp256k1 instead of github.com/google/keytransparency/tree/master/core/crypto/vrf. google/keytransparency doesn't have secp256k1 support, but JS implementation uses secp256k1 and we have to use it in the golang implementation for backward compatibility with existing transactions. It's still under question if we shouldn't fork the github.com/dappley/go-dappley implementation, reimplement the needed parts on our own or find an alternative, because the library is not super-popular and doesn't have commits for 2 years.

[szynwelski]

The new decentralization of the Warp sequencer is being implemented in a dedicated repository. A high-level description of the process can be found in the article.