Configuration override in separate file

[abourget]

Support a separate file like substreams.polygon.yaml that ends with:

package:
  name: my-package-polygon

network: polygon
initialBlocks:
  mod2: 123123123
params:
  mod1: "123123"

and can be packed as an additional spkg with:

# // where --config, -c
substreams pack substreams.yaml -c substreams.polygon.yaml

Try to find the algorithm to resolve the initalBlocks just like the Manifest algorithm does, but when we're given an spkg and asked to override initialBlocks.

!! We also want to be able to override the sink_config and sink_module.

• [x] Think how to map the blocks that are identical, and see how that integrates with the initialBlock resolution algorithm (that checks for dependencies).. Perhaps merge the file before doing the descent. See https://github.com/streamingfast/substreams/issues/207 for a block mapping proposition on the command-line, to stay consistent.

[maoueh]

@abourget What about run/gui? Were you expecting people to perform a .pack to use the override feature? I discussed with Colin this morning and it was weird to me to force developers to have a pack.

[abourget]

Would it make sense to rework it so we have a different kind of manifest, we'd pick up on the difference using something like package.override: path/to/module ? And in that case, we'd laod the overridden module. This would allow us to always have a single file to point to.

It makes a lot of sense because an override always needs to know what it is overriding. It doesn't make sense that it doesn't know, and leave it to the command-line runner!

@maoueh @colindickson

[jubeless]

Add a deriveFrom field in the Manifest...

specVersion: v0.1.0

deriveFrom: https://github.com/streaming/substreams-core/v1/file.spkg

package:
  name: "substreams-polygon"
  version: v0.4.3

network: polygon
initialBlocks:
  mod2: 123123123
params:
  mod1: "123123"

You can override certain field

So we remove the -c in pack or any other flags

[jubeless]

Validation:

package.name OPTIONAL package.version OPTIONAL network: OPTIONAL initialBlocks: OPTIONAL params: OPTIONAL