proposal: merge gridproxy and graphql into a new Grid-API

[Omarabdul3ziz]

Overview

to summarize how our data/api currently works. we handle two types of data in our apis:

1. chain data: this data is related to transactions and states on the chain. It includes contracts, billings, reserved capacity, farms, twins, and IP addresses, ...etc.
2. grid data: this data is related to the node itself, such as its capacity, uptime reports, public configuration, hardware information like GPU/DMI, and some performance test results.

How is this data handled?

• the chain data is purely stored on the chain. each contract created or each newly registered twin is an extrinsic call to the chain and is totally reserved there
• the grid data:
  • part of the node info is stored on the chain, mostly small-size numeric data like the last uptime, the capacity, ...etc.
  • another part of the grid data can't be stored on the chain, and it is for keeping the data stored on the chain minimal. tt will not be efficient to store DMI data, for example, on the chain for each node. Here comes the gridproxy/node-indexer It is a service that runs periodically based on a configured interval to ask the node for the needed data and store it directly on a db to be available after for the API

Current flow: graphql repository:

• graphql/chain-indexer: listens to chain events and dumps them into indexer-db
• graphql/indexer-gateway: graphql api for the data stored in indexer-db
• graphql/processor: watches the indexer api/db and stores a meaningful information like farm/twin/contract/node in processor-db. kind of normalized database
• graphql/processor-gateway: graphql api that serves data on processor-db with basic filtering on the db table.

gridproxy repository:

• gridproxy/processor: sum of triggers placed on the normalized tables in processor-db and aggregate them in a denormalized cache table.
• gridproxy/node-indexer: periodically calls nodes and stores fetched data in processor-db

• gridproxy/server: rest api to serve data from processor-db

  flowchart LR
  c((chain))
  n((node))
  
  id[(indexer-db)]
  pd[(processor-db)]
  
  qi([graphql/chain-indexer])
  qp([graphql/processor])
  
  gp([gridproxy/triggers])
  gi([gridproxy/node-indexer])
  
  qip([graphql/indexer-api])
  qpp([graphql/processor-api])
  gpp([gridproxy/server-api])
  
  subgraph graphql
      direction TB
      c --> qi
      subgraph indexer
      qi --> id --> qip
      end
      subgraph processor
      qip --> qp --> pd --> qpp
      end
  end
  
  subgraph gridproxy
  pd --> gp --> pd
  n --> gi
  gi --> pd
  pd --> gpp
  end
  
  subgraph clients
      user --> gpp
      user --> qpp
  end

Motivation for the proposal

1. graphq api problem

   • the graphql/processor-api is missing all the grid data coming from the indexer in the proxy because it only serves the tables made by the processor.
   • also it doesn't have good filtering system there is no such an inter-tables filtering. tor example, in most cases, filtering a farm based on its nodes fields or vice versa.

2. rest api problem

   • due to the repetitive need to add more fields on the rest response, especially the /nodes endpoint, it has become quite large. after adding the DMI fields i was suggesting to add vertical filtering to return only some fields of the response instead of all of them. this would look like /nodes?select=node_id,twin_id. this is where GraphQL can help fix the over-fetching/under-fetching problem by defining what is actually needed in the request

3. Multiple processing logic

   • merging the processing logic is important since there are two places that process the data: the processor in graphql and the triggers on the proxy. this sometimes make things unclear while debugging an issue. It would be easier if the logic is in one place.
   • also it would be easier to write/debug the plain sql code for trigger functions in Go

Proposal

by integrating the functionalities of both projects, we can achieve the following:

• consolidate all chain and grid data into a single database/cluster. this database should implement role-based permissions, allowing different levels of access for querying and mutating data. with this it will be easier to mock clients for testing purposes. also role-based permissions will enable read-only access to production data, this is regularly needed for debugging and testing changes

• implement a unified processor responsible for handling raw chain events and processing them into the final table. this eliminates the need for writing plain SQL triggers, simplifying the processing logic.

• introduce a centralized database query client to serve as the primary interface for various operations. This client can be utilized as:

  • a package for grid-client usage
  • serve rest api
  • serve graphql api: as opposed to directly integrating graphql api to the database.

  this will ensure uniformity across different interfaces

New structure

graph LR
    subgraph mutation
  n((node))
    c((chain))

    s([syncer])
    p([processor])
    i([indexer])

    dbm{{db-mutate-client}}

    n --> i --> dbm
    c --> s --> dbm
    dbm --> p --> dbm
    end

    subgraph query
    gc[go-client]
    rest[rest-api]
    gql[graphql-api]

    dbq{{db-query-client}}

    dbq --> gc
    dbq --> rest
    dbq --> gql
    end

    db[(grid-db)]
    dbm --> db --> dbq

Plan:

considering the heavy usage of proxy/graphql by the clients, we need to implement this as a separate package, which can gradually replace the old api.

here are the steps:

• [ ] develop the syncer code (may use a different name) to query chain events in a database.
• [ ] rewrite the processing logic (graphql processor/proxy triggers) in Golang
• [ ] Implement database clients for mutation/query operations.
• [ ] develop api servers for both graphql and rest interfaces using the database client.
• [ ] testing.

[muhamadazmy]

I think i understand the overall purpose of this proposal and I agree with the issues you stated above. I just have some "clarification" questions:

• Is this a complete rewrite for the indexer -> processor pipeline? or it builds on top? I mean we still can reuse the indexer part since it just indexes raw events, and we can run our custom processors on top of that events log to rebuild the grid-db
• I understand that the grid-db is where all mutations gonna happen, from both the events, and collected information from the nodes. Is that correct?
• In the second graph, shouldn't the processor be between the syncer and the mutation-client ?
• The mutation-client is just a sql client with write access to the grid-db, correct?

[Omarabdul3ziz]

• Is this a complete rewrite for the indexer -> processor pipeline? or it builds on top? I mean we still can reuse the indexer part since it just indexes raw events, and we can run our custom processors on top of that events log to rebuild the grid-db

yes for the indexer part, we can keep the usage as is. since we are using a ready subsquid image for ingesting the chain event, but for the processor part i think it will need to be rewritten to cover the new tables. maybe in go the current code base written on typescript

• I understand that the grid-db is where all mutations gonna happen, from both the events, and collected information from the nodes. Is that correct?

yes, it will be a central db for all data

• In the second graph, shouldn't the processor be between the syncer and the mutation-client ?

i was thinking of storing the raw ingested chain data in the same database, so the processor will read from and store in the same database. i am not fully aware of how we can built the processor, should we actually read the raw chain data from the database or can we instead use the subsquid-gateway API. i will look at this part.

• The mutation-client is just a sql client with write access to the grid-db, correct?

yes

[muhamadazmy]

I think the indexer-db (which contains the raw data) should be separate from the "view" db. Since the indexed data can be huge, it can be slow and can affect the db? also will make it easier to reset the view/grid-db database and start over without losing the indexed data which is expensive to rebuild from the chain