Introduce an indexer for faster lookups and aggregate data

[adamsoffer]

What's the problem?

In addition to providing a friendly user interface for token holders to delegate, the explorer plays an important role in helping token holders make informed decisions on which transcoders to delegate towards by exposing relevant transcoder data. Currently, the explorer gets this data directly and exclusively off the blockchain resulting in a less than desirable user experience for a couple reasons:

1. Aggregate data that ought to play a crucial role in a delegator’s decision-making process such as “total missed reward calls over n rounds” or “total segments claimed over n rounds” is precluded.
2. Querying data directly from the blockchain and formatting it client side to fit the requirements of the application is not performant. The transcoders view, for example, takes up to 20 seconds to render on initial page load.

Describe a possible solution you've considered

An indexer would provide the explorer with both faster lookups and valuable aggregate data. I’ve been experimenting with The Graph, a query protocol for getting data off blockchains and storage networks through a peer-to-peer network of indexing nodes using GraphQL.

How The Graph works:

At a high level The Graph data flow is as follows:

1. A decentralized application creates/modifies data on Ethereum through a transaction to a smart contract.
2. The smart contract emits one or more events (logs) while processing the transaction.
3. Graph Node listens for specific events and fires handlers in a user-defined mapping.
4. The mapping is a WASM module which runs in a WASM runtime. It creates one or more store transactions in response to Ethereum events.
5. The store is updated along with indexes.
6. A decentralized application queries--via a GraphQL endpoint--the Graph Node, which in turn queries the store, for data which was ingested from the blockchain. This may include complex queries which take advantage of the store's indexes.
7. The decentralized application displays this data in a rich UI, which an end-user leverages in making new transactions against the Ethereum blockchain.
8. The cycle repeats.

Creating a Livepeer Subgraph

In The Graph, a project's GraphQL endpoint is referred as a subgraph, because once deployed to the decentralized network, it becomes just one subset of a global GraphQL endpoint. The subgraph is defined as a YAML file called a subgraph manifest. It comprises a schema, source data and mappings which are used to deploy an endpoint.

I propose creating a Livepeer subgraph for the transcoder view to speed up the initial time to render and using it to add the following information to the transcoder card:

• total missed reward calls (over last 30 rounds)
• total segments claimed (over last 30 rounds)

I started working on a Livepeer subgraph which you can check out here. It's currently deployed to a Google Cloud instance if you’d like to play around with the GraphiQL interface. Right now it simply indexes transcoder data (stored in a PostgreSQL database).

Additional context

• The Graph Protocol is still heavily WIP. Its standalone indexer has been open sourced, but the decentralized network is yet to be launched, so the subgraph would be self-hosted to start.
• I’ve been in touch with The Graph team and they’re keen on making the software work well for Livepeer.
• GraphQL schema stitching can be used to combine the existing Livepeer GraphQL schema with the one generated by The Graph so everything can still be exposed over a single API endpoint.
• The transcoder card design would have to be updated to make room for the new information. Here’s a mockup of what that could look like.

[dob]

I am very excited about this! I think we can add a ton of user value by exposing indexed data to our delegators. My one suggestion would be to try to keep the UI resillient in the case of failures from The Graph node, since the project is so early in its development, and is not yet a reliable decentralized network. I'd expect the node to occasionally crash, or be unresponsive, or be slow, and I'd hate for the UI to be completely unusable in those scenarios.

[adamsoffer]

Happy to hear you're excited about this! Agreed, data availability is super important and would have to be paid close attention to. I'll make sure to configure a server for high availability along with a monitoring agent. I've had the node running over the last 24 hours and so far so good. In the likely event of one of those scenarios playing out as The Graph matures, to ensure the UI is never in an unusable state we can probably check network conditions and fall back to querying the blockchain directly if a connection is lost.

[adamsoffer]

Following up with where things are at with the Graph Protocol + Livepeer Explorer integration:

Todos

• [x] Add all Bonding Manager events necessary to the Livepeer subgraph in order to recreate the current state of the transcoder set
• [x] Set up a Google Cloud instance running a Graph Node for indexing incoming event data (as specified in the subgraph), storing data in a Postgres database, and serving it over a GraphQL endpoint.
• [x] Update the apollo package to include GraphQL schema generated by the subgraph using schema stitching.
• [x] Update explorer transcoders view to use new subgraph query responsible for fetching transcoder data as opposed to current query which fetches this data directly from blockchain and confirm performance improvement
• [x] Fall back to querying blockchain directly if graph node connection is lost
• [x] Add transcoder "missed reward calls" field to subgraph
• [ ] Add transcoder "segments claimed" field to subgraph

Blockers

• Infura Websockets Unfortunately, I’ve run into an issue with the Graph Node where it throws an error after running for a short while due to Infura's unreliable WebSocket endpoint. The Graph team is currently working on a solution to more reliably fetch data from Ethereum and make it available to subgraph mappings, but until that's ready we'll have to wait a little bit before we can roll this out. I'll follow up when that's fixed.

Links:

• Livepeer Subgraph
• Livepeer Subgraph GraphQL endpoint
• My Current LivepeerJS working branch

[dob]

Two points about the blocker with unreliable infura ws endpoints...

1) I assume that this not a client-specific connection in the browser, but instead a connection from the graph indexing node to infura? If so, is there a configuration option to point it at a locally running geth (light) node for example?

2) I got a note from The Graph team that they were aware of the issue and has a fix coming out very shortly.

[adamsoffer]

Ah there is indeed a configuration option to point it at a locally running geth node. I'll give that shot. Thanks.

[Randy1Burrell]

@adamsoffer I am not able to access Livepeer Subgraph GraphQL endpoint. I wanted to run some tests on this.

[adamsoffer]

Oops, looks like the instance ran out of space and crashed. I'll add more storage space and rerun.

[Randy1Burrell]

@adamsoffer a concern with this you might be able to address for me is the memory utilization that causes it to crash.

• How long have this instance been running?
• Under what load and time period does it crash?
• Any ideas about what can fix this issue for these metrics?

I ask because I would not want to see this happening in production under similar loads.

[adamsoffer]

@Randy1Burrell Yeah uptime is super important. It's been running for a few weeks. Must of crashed this morning. I've been using the smallest possible instance size on Google Cloud while testing.

TBH I haven't paid much attention to the devops side of things yet - just trying to get everything working first. I can ask the Graph team what type of server environment they recommend for running a Graph Node reliably. Also, if you have have some devops experience, I'd love some help.

I'm traveling right now, but I'll let you know when I get chance to get the endpoint back up and running. Would love some feedback.

Update: Confirmed it simply ran out of space. Only had 10gb of storage on this instance.

[Randy1Burrell]

@adamsoffer Safe travels.

I would love to hear what the Graph team recommends for this. Availability and reliability will make or break any platform.

Let me know what you need in regards to devops and I will see what I can do. I would be happy to give you my feedback and full support in getting this ready for deployment.

[adamsoffer]

@dob Update: pointing at a locally running ethereum node in fast mode does seem to fix that Websocket issue. However, in order to access smart contract state in subgraph mappings we would need to run an archive node which takes weeks to sync. If all the indexed data needed for the explorer was on events / transactions we could get away with a regular full node, but there is some transcoder contract state we need access to such as isActive that is not available without calling a contract method (correct me if I'm wrong here).

Good news is the Graph has a pull request in for a fix and I see it's pending review, so like you said it shouldn't be much longer before it's out.

[dob]

Yah, good news about the open PR, because running an archive node would be a non-starter. Even if you can get it synced, it "breaks", and then is useless, without proper operational overhead and redundancies

[adamsoffer]

Update:

• I've moved the logic responsible for schema stitching the subgraph api into our private apollo package. I originally had this logic inside the graphql-sdk package but realized it makes more sense to push the responsibility of schema stitching on to the developer since it gives the developer more flexibility. We can add instructions to the docs on how developers can easily combine the graphql-sdk with the livepeer subgraph or any other GraphQL API.
• I've set up error tracking with sentry.io for monitoring and fixing graph node crashes in real time. We can integrate this with discord for real time notifications in one of the channels.
• We have graceful degradation 🎉 When visiting the explorer, the apollo client first checks the network conditions of the Livepeer subgraph api and only attempts to stitch it if it returns a 200 status code.

Below is a screen recording that shows how the explorer responds if the graph node goes down. Notice when the graph node is running the transcoder view loads near instantly since it's fetching data from an indexed PostgreSQL db. When the graph node is manually shut down the explorer is still accessible, falling back to fetching data directly from the blockchain as it currently does with a ~15 second delay.

[adamsoffer]

Update:

The Graph team released a fix for the unreliable infura endpoints. I noticed there were still a couple issues causing the node to error out, but they were resolved over the weekend at ETHSF with the help of the Graph team. That, along with some updates made to the Livepeer subgraph, means we can now index all relevant events from genesis to the most recent block and accurately recreate the current state of the transcoder set which I'm pretty excited about.

A couple things left to do:

1. Put in place a solid deployment / hosting strategy for serving the graph node. (Jannis, the Graph's tech lead, kindly offered to share with us his team's set up.)
2. Go over how the pieces fit together for anyone who's interested in a Google hangout.

cc: @dob @ericxtang @Randy1Burrell

[dob]

Excellent, count me in for the Goog hangout walkthrough. Agree that at this point it seems to be more about dev ops of adding this piece of infrastructure than maintaining the actual functionality, but the fact that we have graceful fallbacks to blockchain looks is great!

[Randy1Burrell]

@adamsoffer sounds great.