Params to modules / dynamic inputs

[jubeless]

Currently Substreams do not allow the abiliy to add dynamic inputs when running a substreams. To accomodate this feature we suggest the following modification to the substreams manifest

imports:
  ethereum: substreams-ethereum-v1.0.0.spkg

modules:
  - name: mymod
    doc: """
      The params are URL-encoded key/value and deal with it. Two are supported: contract & account. They mean whatever
    """
    inputs:
      - params: string
      - source: sf.ethereum.type.v1.Block

params:
  "ethereum:othermod": override-params-optionally-copy-the-original-value-dude
  mymod: params

and the CLI could:

-p ethereum:othermod=original-value-plus-yours -p mymod=0x123123123

To be parameterizable, the params input needs to be present, and be the first input.

In the Rust module, it's a new input, decoded as a simple String (we need to tweak the substreams-rs macro to adapt to that).

The wasm runtime needs to pass the params value, like another input.

For now, we only support string as a params, and encourage people to use URL-encoding for more complex parameterization. However, a simple contract address for instance, is totally fine.

---

The modifications to the [substreams/v1/modules.proto](https://github.com/streamingfast/substreams/blob/develop/proto/sf/substreams/v1/modules.proto) file woud look like this:

  message Input {
    oneof input {
      Source source = 1;
      Map map = 2;
      Store store = 3;
      Params params = 4;  # This added
    }

    message Params {  # This added
      string value = 1;
    }

This way, there is a way for default values to be lodged when publishing a module. It's already at the right place for the runtime to consume. And it can be overridden by client code as such:

Python code to tweak params would live here and look like:

[mod for mod in pkg.modules.modules if mod.name == "store_pools"][0].inputs[0].params.value = "myvalue"

[abourget]

Another option is to use params as a string, the manifest be:

modules:
  - name: mymod
    doc: """
      The params are URL-encoded key/value and deal with it. Two are supported: contract & account. They mean whatever
    """
    inputs:
      - params: string
        default: "default string" # Option 1
      - source: sf.ethereum.type.v1.Block
  params: "default string" # Option 2

params: # Option 3
  othermod: override-default-params-here

with support on the CLI for something like:

-p mymod=hello=world&other=var \
-p othermod=0x123123123

---

Second way to override params:

imports:
  ethereum:
    url: substreams-ethereum-v1.0.0.spkg
    params:
      othermod: override-params-in-there-keep-original-if-you-want

and CLI:

-p ethereum:othermod=this-is-another-override

---

A third way (preferable by abourget, deniscarriere) to override params, is its own section:

imports:
  ethereum: substreams-ethereum-v1.0.0.spkg

modules:
  - name: mymod
    doc: """
      The params are URL-encoded key/value and deal with it. Two are supported: contract & account. They mean whatever
    """
    inputs:
      - params: string
      - source: sf.ethereum.type.v1.Block

params:
  "ethereum:othermod": override-params-optionally-copy-the-original-value-dude
  mymod: params

and the CLI could:

-p ethereum:othermod=original-value-plus-yours -p mymod=0x123123123

[abourget]

Use case with Antelope:

• goal is to add a field in the final output module that's going to a sink, to add the chain_id to certain objects.
• the consuming process is not aware of the endpoint.. and the substreams module is made to be totally generic, so upon running the substreams, it'd be useful to be able to inject some metadata, from the caller, that ends up inside the stream.

[abourget]

Second use case:

• You have a general purpose token balance tracker substreams
  • Your goal is to track only a SUBSET of those assets. You could specify the contract address, token symbol, whatever.

[abourget]

Another question, is where it lands in the protobuf model:

https://github.com/streamingfast/substreams/blob/develop/proto/sf/substreams/v1/modules.proto

  message Input {
    oneof input {
      Source source = 1;
      Map map = 2;
      Store store = 3;
      Params params = 4;  # This added
    }

    message Params {  # This added
      string value = 1;
    }

Python code to tweak params would do:

[mod for mod in pkg.modules.modules if mod.name == "store_pools"][0].inputs[0].params.value = "myvalue"

[abourget]

Previous design:

Currently Substreams do not allow the abiliy to add dynamic inputs when running a substreams. To accomodate this feature we suggest the following modification to the substreams manifest

modules:
  ...

  - name: map_transfers
    kind: map
    initialBlock: 12287507
    inputs:
      - params: sf.substreams.v1.ListParams
        default: {"my": "default", "value": "world"} // optional, let's start without.
      - source: sf.ethereum.type.v1.Block
    output:
      type: proto:eth.erc721.v1.Transfers
  ...

A few things to note:

1) There needs to be a params as an input, and it needs to be the first one. For the time being we will only support string 2) the params key that is on the same "level" as inputs is a default value. The user running the substreams will have the abiliyt to override it via the CLI or in code while performing his GRPC request

substreams run -p -e localhost:9000 substreams.yaml map_transfer \
  -p map_transfer={"json": "representation", "of": "the 'param' type"} \
  -p other_module={"bob": "dylan"} \
  --start-block 12375267 \
  -t +1

The serialized inputs will need to be added to the modules' hash for cache purposes.

message MapParams {
  map<string,string> params = 1;
}
message ListParams {
  repeated string = 1;
}

On the command line, we could also help with the known type sf.substreams.v1.Params:

substreams run ... -p map_transfer=bob=123 -p other_module=hello=world&my=friend
substreams run ... -p map_transfer=0x123123,0x234234 -p other_module=hello=world&my=friend

the moment a --param module=HERE param HERE doesn't start with a {.