Cannot set `startBlock` for Substreams data source

DenisCarriere commented 1 year ago

Which packages are impacted by your issue?

@graphprotocol/graph-cli

Describe the issue

Cannot provide startBlock: 1397553 for when using Substreams data source.

Reproduction

https://github.com/pinax-network/substreams-cookbook/tree/main/erc20

Steps to Reproduce the Bug or Issue

graph build

Error message:

✔ Apply migrations
✖ Failed to load subgraph from subgraph.yaml: Error in subgraph.yaml:

  Path: dataSources > 0 > source
  Unexpected key in map: startBlock
Error: Error in subgraph.yaml:

  Path: dataSources > 0 > source
  Unexpected key in map: startBlock

Expected behavior

Additionally to Substreams startBlock, should be able to provide startBlock to Subgraph when using Substreams data source.

Screenshots or Videos

No response

Platform

$ graph --version @graphprotocol/graph-cli/0.51.2 darwin-arm64 node-v20.2.0

Subgraph Manifest

not sure where to include: startBlock: 1397553

dataSources:
  - kind: substreams
    name: erc20
    network: Mainnet
    source:
      package:
        moduleName: graph_out
        file: ./erc20-v0.1.0.spkg
    mapping:
      kind: substreams/graph-entities
      apiVersion: 0.0.5

Subgraph GraphQL Schema

No response

Additional context

Alternative way, must provide initialBlock: 1397553 to Substreams.yaml

azf20 commented 1 year ago

@DenisCarriere I think this is currently expected behaviour (rather than a bug), the substreams package dictates the starting block. I can see the argument for setting this in the subgraph.yaml, but I can also see potential confusion arising with the same setting in two places. I guess interested in thinking through the different permutations:

If the substreams initialBlock < subgraph startBlock, you should still process the earlier blocks in the substreams, but only stream the entity changes from the startBlock - this could lead to issues if there are some invalid changes based on some of history not being processed
If the subgraph startblock < substreams initialBlock, you might expect to get entity changes for blocks where the substreams modules aren't running as it is still prior to their initialBlock

Maybe it might be cleanest if the startBlock overrides the initialBlock (e.g. as a parameter), but that could also be troublesome if there are different initialBlocks across different modules? @maoueh interested in your thoughts here

maoueh commented 1 year ago

I agree that it's currently a feature. I also agree that the source of truth will be the .spkg and that subgraph start block < substreams start block should be forbidden if we ever allow startBlock in the Subgraph manifest.

What is the use case here exactly you tried to achieve @DenisCarriere? Testing purposes I imagine?

graphprotocol / graph-tooling