RFC: Multichain deployments

[dzhelezov]

Motivation

• For many dApps, the data is fragmented across multiple chains, and it's essential to track and aggregate events on transactions
• Running multiple processors within a single squid has significant drawbacks:
  • impossible to track the sync status (and even define it properly)
  • failure of any of the processors halts the whole squid
  • not scalable for many networks (e.g. if one wants to track all parachains)
• Stitching multiple GraphQL endpoints requires an external service (e.g. Hasura) and is not feasible for aggregations. For example, implementing a list of transactions for a given account would require client-side aggregation and sorting. Pagination becomes too complex: one needs to keep a cursor for each endpoint and perform the sorting/aggregation on the client.
• Sharing a database between squids will significantly reduce the consumed resources

Proposal

Introduce namespaces and parent-child relations in the deployment manifest. Within each namespace the addons can be referenced by an ID. Only the parent squid can define the api service, and then the status is tracked only for the child squids. Example

## deploy-main.yml
manifestVersion: subsquid.io/v0.1
name: transactions 
version: 1
build:

deploy:
  addons:
     postgres:
         id: db
  api:
    migrations: true                                        # run the db migrations on start
    cmd: [ "npx", "squid-graphql-server" ]  # for the main squid only the `api` is defined.

## deploy-arbitrum.yml
manifestVersion: subsquid.io/v0.1
name: transactions/arbitrum 
version: 1
build:

deploy:
  addons:
     postgres:
         schema: `arbitrum`
         ref: db
  processor:   # only the processor, saves transactions on arbitrum
     migrations: false # do not run the migrations

## deploy-kusama.yml
manifestVersion: subsquid.io/v0.1
name: transactions/kusama 
version: 1
build:

deploy:
  addons:
     postgres:
         schema: `kusama`
         ref: db
  processor:  # only the processor, saves transactions on kusama
      migrations: false # do not run the migrations

• Addons may only be referenced within the same namespace
• namespace is derived from name
• api can be defined only for the parent endpoint
• each processor is granted the write access only to the schema, defined in the postgres addon
• when a child squid is dropped, only the schema is dropped

Points to discuss

• Can we make addons shareable between squids in different namespaces? This will further reduce the resource footprint
• Can we do better with the way migrations are handled?