RFC: deployment manifest for squids

[dzhelezov]

Motivation

1) The existing way of deploying squids with sqd deploy is not suitable for a growing list of deployment options Aquarium supports 2) We want to make each squid portable and easy to deploy on any platform aside from Aquarium. That's why all squids will be built into a single Docker image which is available via IPFS.

We also support the following user stories:

• create a squid from a specified context (e.g. when you have a single repo for multiple networks):
  • current root
  • a tar.gz archive
  • a git repo url
• Update an existing squid with a fix
• Restart an already deployed squid with new env variables (e.g. set logging flags to debug)
• Have an identical build and execution environment for a squid deployed to Aquairum
  • to reproduce bugs spotted in the wild
  • to be able to self-host squids easily
• Create a new squid version (e.g. for new experimental features)
• Scale up an existing squid (e.g. add more storage to Postgres, add Redis cache or add gateway replicas)
• Replace an existing squid version with a new one (hard reset)
• Deploy a squid without GraphQL API (if the data is written into a file and no API is needed)

Manifest file format

The file consists of the following sections

• Header with the squid metadata
• build -- only supports builds from a strictly defined Docker file. All services (the processor, API) will now use the same image but different commands
• deploy -- define the squid services (processor and/or api) and a set of addons . We currently support a Postgres addon In the future we will also support Redis and other services that one can run along the squid.
• scale -- this section defines how the squid services should be scaled up.

A minimal manifest file is as follows:

manifestVersion: subsquid.io/v0.1
name: sample-squid
version: 1
description: |-
  The very first squid from manifest  

build: 

deploy:
  addons:
    postgres: 
  processor:
    cmd: [ "node", "lib/processor" ] 
  api:
    cmd: [ "npx", "squid-graphql-server"]

A more advanced will be as follows:

manifestVersion: subsquid.io/v0.1
name: sample-squid
version: 1
description: |-
  The very first squid from manifest  

build: 

deploy:
  addons:
    postgres: 
         # additional configs for addons
         version: 14
 # set of secrets that must be defined by the environment
  secrets:
    - ACALA_RPC_ENDPOINT
    - COINGECKO_API_KEY 
  processor:
    environment:
        SQD_DEBUG=sqd:mapping
    env:
       - .deploy.env  # add env variables from a file
    cmd: [ "node", "lib/processor" ] 
  api:
    # custom command
    cmd: [ "npx", "squid-graphql-server", "--subscriptions", "--max-root-fields", "10", "--sql-statement-timeout", "1000" ] 

scale:
  addons:
     postgres:
         read-replicas: 1
         storage: 100G
  processor:
     profile: p1.medium
  api:
     profile: a1.large
     # add three replicas
     replicas: 3

CLI deploy commands

Deploy

sqd deploy <source> [--update] [--manifest] [--hard-reset]. <source> can be as follows:

• a path to a local folder (e.g. .)
• a url to a .tar.gz archive
• a git-url to a git repo with an optional branch or commit tag

The deployment flow is as follows:

• If the existing version already exists, prompt the user if it should be updated (or rewrite if the --update flag is passed).
• Otherwise, create a new version as specified in the manifest.
• Build a Docker image using a standardized pre-defined Docker file (bound to the manifest version) and the provided context
• Publish the image to the registry (with later support of IPFS by nerdctl
• If scale is defined in the manifest, update the deployment specs
• Deploy to Aquarium

Environment variables interpolation

sqd deploy will interpolate${} placeholders in the manifest file before making a deployment request. E.g. running sqd deploy manifest.yaml -e NETWORK=kusama will replace all occurrences of ${NETWORK} with `kusama

Accessing squid images

All images of the deployed squids (previously built by Aquairum) will be available from the container registry and later via IPFS. List all images Aquarium: sqd images get [--name]

[aintnomoreloyalty]

If a developer wants to create a new version, do they need to create a separate manifest file or how does that work?

[dzhelezov]

Yes normally you just bump the version in the file

[aintnomoreloyalty]

Hi there What should happen if developer deploys a squid, then changes its name and deploys it from the same directory? @mo4islona

[mo4islona]

@aintnomoreloyalty a new squid will be created

[icikibung]

How to add rpc