Deployer interface for Substreams Services or Sinks

[abourget]

Add in substreams:proto/sf/substreams/sink/service/v1/service.proto

message DeployRequest {
  sf.substreams.v1.Package substreams_package = 1;

  bool development_mode = 1;

  repeated EnvironmentVariable environ = 2;
}
message EnvironmentVariable {
  string key = 1;
  string value = 2;
}

Mutation of the alpha commands to:

substreams alpha service deploy
# and eventually:
substreams service deploy
substreams service info
substreams service list

From the SinkConfig objects, we remove all things that can be related to development. The goal of those Config objects is define the strict service we want in PRODUCTION environments. Which is the lowest denominator that all service providers MUST serve in order to be compliant with the service definition.

The development features are to be passed out-of-band with the development_mode flag and other environment variables.

Prod vs dev

• One principle is the full specification of a service as defined by a model like sf.sub.webhook.v1.Service
  • When you ask for a given service, it is THAT service that you will get.
  • Whever you deploy, you'll get a DEFAULT experience that is the SAME.
  • That can be -e http://localhost:8080 or -e deploy.streamingfast.io.

If a provider doesn't support, it simple says so:

$ substreams service deploy -e deploy.pinax.network ./substreams.sql.yaml
PINAX: Sorry, the service `sf.substreams.sql.v1.Service` is not supported here.

When the development_mode == False:

• Postgres will NOT expose any wonky ports.
  • We can remove the wire_protocol_access..

When development_mode == True

• Expose postgres ports and all
• DBT will deploy an empty can, do some validation perhaps, but mainly print an MOTD to tell you how to use the remote database below.

substreams.clickhouse+dbt.yaml is reduced to its bare essentials:

sink:
  config:
    dbt:
      engine: clickhouse
      rest_frontend:
        enabled: true
      dbt:
        files: ./dbt
        run_interval_seconds: 3600

This is an example deployment locally:

By DEFAULT we deploy with --prod=false, which sets the development_mode = True

$ substreams service deploy ./substreams.clickhouse+dbt.yaml
LOCAL DOCKER: 
- Spinning up `clickhouse`
  - Exposing port 8125 (http clickhouse protocol)
  - Exposing port 5432 (pgsql compatible endpoint)
  - Exporing port 9000 (admin interface)
- Spinning up `rest_frontend`
- Spinning up `sql-sink`
- Spinning up `dbt`
  - Automatic DBT disabled. You can do one of two things: Set OOB var DBT_RUN=true if you still want to run it.
    1. Run `dbt` via Docker:

      export SET_PROFILE=bayc_dbt=clickhouse://asldkfj:alsdkfj@localhost:8135
      alias my_dbt=docker run -v ./dbt:/dbt/files -ti -e SET_PROFILE=$SET_PROFILE dbt-runner:latest run

    2. Or run locally:

    pip install dbt
    dbt setprofile bayc_dbt=clickhouse://asldkfj:alsdkfj@localhost:8135
    dbt run  # iterate here

With an explicit deployment to production:

$ substreams service deploy --prod ./substreams.clickhouse+dbt.yaml
LOCAL DOCKER: 
- Spinning up `clickhouse`
- Spinning up `rest_frontend`
- Spinning up `sql-sink`
- Spinning up `dbt`

Some deployment services could honor other variables to further refine the experience, either in dev or production:

$ substreams service deploy -e deploy.pinax.network -H CLICKHOUSE_OPEN_PORTS=false DBT_ENABLED=false ./substreams.clickhouse+dbt.yaml
PINAX: 
- Spinning up `clickhouse`
  - Not exposing clickhouse ports, as per `CLICKHOUSE_OPEN_PORTS`
- Spinning up `rest_frontend`
- Spinning up `sql-sink`
- Spinning up `dbt`
  - Automatic DBT disabled (DBT_ENABLED=false)

[colindickson]

Tasks so far:

• [x] update DeployRequest with definition given above
• [ ] change cmd to alpha service instead of alpha deploy
• [x] remove pgweb and wire protocol access from SinkConfig

[sduchesneau]

@abourget a few things to specify and discuss possibly:

The only things that are not production in our definition are: 1) wire_protocol 2) pgweb

I'm certain that some users WILL WANT wireprotocol access on their production environment, but we would either: a) tell them no (usually a bad idea) b) do it "out of band" -- they'll probably ask for it after the fact. But it could be a special magic "streamingfast..." environment variable that enables it. It wouldn't be part of the spkg definition. If it's in an environment variable, we'll need to support an "update" method on the deployment endpoint. an "update" on production may be refused if the modules or schema change, or other kind of business rules like that.

Comments ?

[sduchesneau]

Also, maybe some users will want DBT to run automatically in development.... It would require a more complex system, so I suggest we don't do any of that now: development means run your own (damn) dbt (from your workstation)

[sduchesneau]

@abourget

Another thing:

I don't believe that environment_variables name make sense anymore.

What about custom_parameters ? or just parameters ?

They are not environment variables, they are not used as such... it would be a bit like substreams params...

[sduchesneau]

chosen: deployment_parameters

it will contain either

• "provider-specifics params" (ex: streamingfast_enable_logrotate: true)
• secrets (ex: webhook_url: https://my.web.hook)

todo:

• [x] rename environment_variables to deployment_parameters
• [x] add flag -p (--deployment-parameter) as StringArray to substreams alpha service deploy

[colindickson]

• [x] activate pg web by default for dev mode
• [x] make clickhouse work with datagrip for docker deployment