Update manifest to incorporate block and transaction handlers

Here is a proposal for a new manifest structure. Some of the goals I had in mind are:

Enable the addition of new triggers, in the short term block and transaction triggers
Enable the addition of new chains/networks/plugins/datasources
Add a place for configuration that varies in relevance depending on environment 
Add a place to enable experimental features before including them in the proper subgraph schema
Design the schema to transparently describe the system and its hierarchy of objects (referenced the network information model)
Ensure current manifests can be upgraded programatically graph upgrade ./path/to/manifest.yaml

With regards to domain design, I just used the naming in the network information model and the spec field is lifted from k8s manifests.

Updated manifest examples are included in the comments

apiVersion: subgraphs/0.0.1
metadata:
  description: A decentralized virtual world that runs on open standards. Find districts, parcels, auctions, and more.
  repository: https://github.com/graphprotocol/decentraland-subgraph
spec:
  schema: ./schema.graphql
  plugins:
  - name: ipfs
    version: 0.0.2
    metadata:
    spec:
      objects:
      - name: ensRecordSetAddress
        value: /ipfs/0xedf4...
      - name: ensConfigAddress
        value: /ipfs/0xedf4...
      triggers:
      - name: ensRecordSet
        kind: line
        spec:
          address: ensRecordSetAddress
      - name: ensConfig
        kind: bulk
        spec:
          address: ensConfigAddress
      mapping:
        runtime: wasm/typescript
        version: 0.0.2
        spec:
          module: ./src/mappings/ipfs_logic.ts
          handlers:
          - trigger: ensRecordSet
            handler: handleEnsRecord
          - trigger: ensConfig
            handler: handleEnsConfig
  - name: ethereum
    version: 0.0.1
    metadata:
      experimental/triggerOrdering: "block,tx,event"
      experimental/trailHead: 60 // Subgraph should execute trailing the chain head by x blocks
      network: mainnet
    spec:
      abis:
      - name: LANDRegistry
        file: ./abis/UpdatedLANDRegistry.json
      sources:
      - address: 0xf87e31492faf9a91b02ee0deaad50d51d56d5d4d
        abi: LANDRegistry
      triggers:
      - name: transfer
        kind: event
        spec:
          eventSignature: Updated(address,uint256)
      - name: approval
        kind: transaction
        spec:
          to: 0x9f...
          functionSignature: approve(uint256,uint256)
      - name: epoch
        kind: block
        spec:
          modulo: 10 // Trigger every time the block number is divisible by x
      mapping:
        runtime: wasm/typescript
        version: 0.0.1
        spec:
          module: ./src/mappings/logic.ts
          handlers:
          - trigger: transfer
            handler: handleTransfer
  - name: tezos
    version: 0.0.1
    metadata:
      pauseOnNetworkUpgrade: true
    spec:
      triggers:
      - name: proposalAccepted
        kind: upgradeProposalAccepted
        spec:
          voteThresholdLE: .55 // Trigger if the vote passed with approval lower or equal to x
          voteThresholdGT: .9 // Trigger if the vote pass with approval greater than or equal to x
      ...
  - name: polkadot
    version: 0.0.1
    metadata:
    spec:
      triggers:
      - name: adExMessage
        kind: parachainMsg
        spec:
          fromParachain: 0x8ef...
      - name: newParachain
        kind: parachainCreated
        spec:
      - name: blockFinalised
        kind: block
        spec:
          parachain: 0xef56...
      ...

I generally like this a lot. It is more verbose but it clearly separates common and specific fields at every level.

A few comments:

I'm not sure I like plugins. Whether something is a plugin or not feels like an implementation detail. Why not keep dataSources?
I'm not sure I like how the ipfs example mixes multiple objects and sources. IMHO these should be separate data sources, each with their own trigger(s). Again, I'd prefer thinking of "data sources" before architectural structure "plugins".
IMHO, the network under the Ethereum example really belongs next to the contract address. Again, combining multiple contracts here doesn't feel right to me. If we did it this way, I'd want each trigger to be linked to a specific contract from the list.

@lutter posted in slack:

Looking at it, I Was wondering if you could maybe add another comment 
to make it clear what's part of the core format, and what is brought in by the plugins?

Like, the stuff for each plugin, is that completely up to 
each plugin to define or are there things (besides name and version) 
that each plugin has to provide?

And at the danger of bikeshedding, but I 
liked 'dataSources' where you have plugin now

Every plugin/datasource needs to provide a name, version, metadata map, and spec. The combination of name and version determines the schema under the spec field.

A plugin's version also specifies the triggers being offered and the arguments passed to handlers subscribed to a trigger.

As far as dataSources vs. plugins, my goal would be to make sure our domain design matches the naming we use in the manifest. The network information model Brandon put out uses the terms plugins and network adapters, although these are likely still up for change.

I personally like plugins since it puts the subgraph at the center of the mental model when writing a manifest and that hierarchy matches well with the value proposition of the Graph (i.e. forget about the chain specifics, your subgraph is all that matters.) I'd be in favor of any word which achieves that same effect though.

plugin appears to be subjective. If i am building a subgraph that is 90% ethereum data, and 10% tezos, or 10% IPFS, Im gonna wanna say the ETH part of the subgraph is the datasource, and the other parts are plugins. but someone could make a subgraph that is 90% tezos, and 10% ethereum. It looks like offchainstorage in brandons model is considered to be a plugin. This makes sense to me that IPFS, or other off chain storage, would be a plugin. And then all blockchains would be datasources.
i think network belongs at top level beside ethereum. I cant see how anyone would combine different ethereum networks in a subgraph. but if they did, then they should just have another ethereum datasource, where the network at the top level is ropsten, or whichever
I think under etheretum:spec: there should be a field name right above abis. same with all others. I will show what I mean below:

      - name: ethereum
        version: 0.0.1
        metadata:
          experimental/triggerOrdering: "block,tx,event"
          experimental/trailHead: 60 // Subgraph should execute trailing the chain head by x blocks
          network: mainnet
        spec:
          - name: DataSource1
              abis:
                - name: LANDRegistry
                  file: ./abis/UpdatedLANDRegistry.json
              sources:
                - address: 0xf87e31492faf9a91b02ee0deaad50d51d56d5d4d
                  abi: LANDRegistry
              triggers:
                - name: transfer
                  kind: event
                  spec:
                    eventSignature: Updated(address,uint256)
                - name: approval
                  kind: transaction
                  spec:
                    to: 0x9f...
                    functionSignature: approve(uint256,uint256)
                - name: epoch
                  kind: block
                  spec:
                    modulo: 10 // Trigger every time the block number is divisible by x
              mapping:
                runtime: wasm/typescript
                version: 0.0.1
                spec:
                  module: ./src/mappings/logic.ts
                  handlers:
                    - trigger: transfer
                      handler: handleTransfer
          - name: DataSource2
              abis:
                  - name: SOMETHING
                    file: ./abis/SOMETHING.json
              sources:
                - address: 0xf87e31492faf9a91b02ee0deaad50d51d56d5d4d
                  abi: SOMETHING
              triggers:
                - name: SOMETHING
                  kind: event
                  spec:
                    eventSignature: SOMETHING(address,uint256)
                - name: SOMETHING
                  kind: transaction
                  spec:
                    to: 0x9f...
                    functionSignature: SOMETHING(uint256,uint256)
                - name: epoch
                  kind: block
                  spec:
                    modulo: 10 // Trigger every time the block number is divisible by x
              mapping:
                runtime: wasm/typescript
                version: 0.0.1
                spec:
                  module: ./src/mappings/logic.ts
                  handlers:
                    - trigger: SOMETHING
                      handler: handleTransfer

Or something along those lines. The way I see it right now, It isn't clear to me how you would have two datasources within ethereum, or the others.

Overall it looks really good!

@davekaj I disagree about plugins vs. data sources. In Brandon's information model, they are still all the same. What the information model refers to as storage covers both, blockchains and IPFS in the same way.

IMHO they should be treated identical. As soon as a subgraph with an IPFS data source is published on chain, the IPFS file is anchored on chain and consensus can be formed around it.

I addressed a few of the concerns in the manifest below, although I think the more valuable discussion right now is conceptual.

Concerns addressed:

Leo mentioned that the indirection in the IPFS plugin isn't worth much, so I moved the object addresses directly into the triggers
Jannis noted that the Ethereum triggers should be related to a contract, so I updated the sources property to accounts and added a reference to an account for each trigger

Some concerns I didn't address in this manifest, which I think need more discussion:

networks/adapters/plugins vs datasources
- Right now the manifest object heirarchy
- datasource
  - mapping
    - handler
- In this proposal the object heirarchy is
- plugin
  - trigger
  - mapping
  - handler (although this concept is slimmer in this model since it relates to a trigger)
- One shortcoming I see with the current datasources orientation is that while in the system it seems that the plugin will be versioned as a whole, datasources (i.e. ethereum/contract, ethereum/transaction, ethereum/x) are versioned individually in the manifest. It probably shouldn't be possible to have separate ethereum/contract datasources pegged to different versions in the same manifest, and the current hierarchy doesn't give us an intuitive way to blanket version them all
- The entities are meant to be disjoint in the protocol by plugins. Having many different instances of datasource at the same level leveraging different plugins makes it clunky to represent that constraint
network being at the top level
- It is at the top level right now, it's just that in this model you don't have multiple instances of a plugin. All triggers for that plugin are defined under its one instance. This could be an organizational downside for the proposed heirarchy, especially for plugins with many triggers.
enforcing disjoint entities across plugins
- right now there is one schema and it lives at the same level as plugins, in the future we could require each plugin have its own schema file and ensure schema names don't overlap across files

apiVersion: subgraphs/0.0.1
metadata:
  description: A decentralized virtual world that runs on open standards. Find districts, parcels, auctions, and more.
  repository: https://github.com/graphprotocol/decentraland-subgraph
spec:
  schema: ./schema.graphql
  plugins:
  - name: ipfs
    version: 0.0.2
    metadata:
    spec:
      triggers:
      - name: ensRecordSet
        kind: line
        spec:
          object: /ipfs/0xedf4...
      - name: ensConfig
        kind: bulk
        spec:
          object: /ipfs/0xedf4...
      mapping:
        runtime: wasm/typescript
        version: 0.0.2
        spec:
          module: ./src/mappings/ipfs_logic.ts
          handlers:
          - trigger: ensRecordSet
            handler: handleEnsRecord
          - trigger: ensConfig
            handler: handleEnsConfig
  - name: ethereum
    version: 0.0.1
    metadata:
      experimental/triggerOrdering: "block,tx,event"
      experimental/trailHead: 60 // Subgraph should execute trailing the chain head by x blocks
      network: mainnet
    spec:
      abis:
      - name: LANDRegistry
        file: ./abis/UpdatedLANDRegistry.json
      accounts:
       - name: registry
         address: 0xf87e31492faf9a91b02ee0deaad50d51d56d5d4d
         abi: LANDRegistry
      triggers:
      - name: transfer
        kind: event
        spec:
          eventSignature: Updated(address,uint256)
          from: registry
      - name: approval
        kind: transaction
        spec:
          to: registry
          functionSignature: approve(uint256,uint256)
      - name: epoch
        kind: block
        spec:
          modulo: 10 // Trigger every time the block number is divisible by x
      mapping:
        runtime: wasm/typescript
        version: 0.0.1
        spec:
          module: ./src/mappings/logic.ts
          handlers:
          - trigger: transfer
            handler: handleTransfer

graphprotocol / research

Update manifest to incorporate block and transaction handlers #89