Epic: migrate to Polkascan

[pmespresso]

Rationale

Nodewatcher is useable but as I need to add a bulk more heavy tasks it's increasingly becoming difficult to keep it fast enough to keep up with the 6 second block time.

so, as discussed on Riot, the plan is to gradually shift over to using Polkascan, who are an open source team specializing in doing nodewatcher stuff. This would help to focus in on the UX rather than dealing with unfamiliar devops tools.

Foreseen challenge

Gatsby is super opinionated and you must use Graphql to generate a static site at build time. This makes genernating nested routes based on results from a REST api at runtime that much more...not straightforward, but not impossible.

cc @amaurymartiny, I would love to get your thoughts on this, maybe you've had experience doing so already, or see a better way?

helpful: https://www.gatsbyjs.org/docs/using-gatsby-without-graphql/, https://www.gatsbyjs.org/blog/2018-10-25-using-gatsby-without-graphql/ maybe helpful: https://medium.com/@christinavhastenrath/rest-apis-in-gatsby-at-runtime-89a89dd976d7, https://www.gatsbyjs.org/docs/why-gatsby-uses-graphql/#generate-pages-with-graphql

Roadmap

Both Polkassembly and Nomidot rely on the existing nodewatcher plus graphql api built on top of it. This switch will replace the planned v1.5 sync of our own nodewatcher, which would have been a major migration.

who hosts it There is a public api that can be used for development purposes, we won't be making a whole lot of queries so it won't affect their bills by more than a few cents a week.

Medium-long term, as discussed with Felix, we want to run their harvester code and host our own db+api so that the costs dont just fall on the Polkascan team.

What needs to change to get to parity with v1-alpha and then get to a v1-beta

a) Nomidot will need the following REST endpoints to make up the data we have through out graphql api:

Accounts: https://github.com/paritytech/Nomidot/issues/359

• https://api-01.polkascan.io/kusama/api/v1/accounts/
• https://api-01.polkascan.io/kusama/api/v1/account/{accountId}

Header: #360

• https://api-01.polkascan.io/kusama/api/v1/eras/
• https://api-01.polkascan.io/kusama/api/v1/block/{blockNumber}
• https://api-01.polkascan.io/kusama/api/v1/session/session/{sessionNumber}

Validators: #361

• https://api-01.polkascan.io/kusama/api/v1/session/validators
• https://api-01.polkascan.io/kusama/api/v1/nominators
• https://api-01.polkascan.io/kusama/api/v1/runtime-event?filter[latestRuntime]=true&filter[module_id]=imonline
• https://api-01.polkascan.io/kusama/api/v1/event?filter[module_id]=offences
• https://api-01.polkascan.io/kusama/api/v1/event?filter[module_id]=staking
• https://api-01.polkascan.io/kusama/api/v1/event?filter[module_id]=vesting

[amaury1093]

I have a couple of questions:

There is a public api that can be used for development purposes, we won't be making a whole lot of queries so it won't affect their bills by more than a few cents a week.

• Is Polkascan aware that we will be using their infra?
• Is this endpoint a production or a staging endpoint? Polkassembly (now live with users) will also depend on this. It's imo risky to rely on someone else's infrastructure, especially if 1/ it's free 2/ they don't have any service level and 3/ there's nothing we can do in case their infra breaks or stops.

IMO: if we decide to use their harvester, we should host it ourselves. Putting @Tbaut as cc.

REST vs graphql

If I remember correctly, the frontend would only talk to one GraphQL endpoint, which would be an umbrella above the AppDB and the ChainDB endpoints. Is this architecture still relevant? If yes, it would make sense to build a thin GraphQL wrapper on top of Polkascan's REST API (not-so-uncommon pattern), and make gatsby talk to it.

If we do use their REST endpoint, re the links you posted: why can't we just use fetch/axios like we would normally do in a CRA app? Right now, anyways, we are using Apollo Client instead of their built-in graphql client. We would just need to replace Apollo Client with an axios to the REST endpoint. For the static site generation at build time, we can just put placeholders instead of real values.

[pmespresso]

IMO: if we decide to use their harvester, we should host it ourselves

yes of course...but while we're getting that up/waiting for a devops person's priority list to get down to this....what would I do? That's why I'm proposing to use their production endpoint to build the frontend on top of it, then by the time when it gets to a point where the app would be making any significant amount of queries, we'd hopefully have our own deployment ready. (btw, not sure if it's staging or production, but it's the endpoint in the Network tab from polkascan.io). The point is not to freeload, rather to keep moving forward and not be stagnated despite the setback....

Is this architecture still relevant? well, no. They are only an extension of nodewatcher, at least for Nomidot.

, it would make sense to build a thin GraphQL wrapper on top of Polkascan's REST API

So as I brought this up in our Riot chat, is that really the best use of our time? My hunch is no, it's a lot of ongoing work for not much payoff, but I know that I don't have the best compass regarding "how long/how much effort does this take", so would definitely appreciate input from you @amaurymartiny, @Tbaut , and @niklabh.

why can't we just use fetch/axios like we would normally do in a CRA app?

As is discussed in those links I posted, Gatsby is incredibly opinionated - one must for the most part use Graphql.

The main problem is to do with the route creation. Because Gatsby is a static site generator, it needs some values in order to create routes....for example, maybe a dumb question, but if we want to create a Details page for each account or each validator, how would we create a accounts/{accountId} from an extension account at buildtime?

For the static site generation at build time, we can just put placeholders instead of real values.

Not sure what you mean, how would that work?

[amaury1093]

Not sure what you mean, how would that work?

See https://www.gatsbyjs.org/docs/data-fetching. Let's take the validator list. At build time, we show an empty <ValidatorsTable>. At runtime, it uses fetch or axios to fetch Polkascan's endpoint to populate the table.

Gatsby is opiniated for build time data fetching: you must use GraphQL. For runtime, it's as flexible as a regular CRA.

For using Polkascan's API: I believe Nikhil and Thibaut will have a better idea than I do.

[Tbaut]

I'm sold to the 1 graphQL interface to rule them all. We are currently aggregating data from 3 DB (with Hasura as a gateway) and handling this on the front-end is a great breeze. Hasura does support stitching with REST (here is some doc for the record but it's an alpha, I can't tell how much hassle it represents and if it'll ever make it to a release).

We'll definitely keep the node-watcher for a while in our case, because we're relying on it heavily not only for the data, but for events. We've a middleware service (chain-db-watcher) that maintains a constant graphQL subscription to the chain-db. Meaning there would be quite a lot to do to migrate for us. Now that we finally have the node watcher stable, I'm pretty happy with it.

I totally get the move to another data harvesting, because Nomidot needs much more data, and if you're not happy with what you have.. that's obviously not cool. Not sure what will be the trigger for us to move as well, but one thing I'm pretty sure is that I'll want graphQL. So a thin self made graphQL server (unless there's something à la prisma on MySql supporting subscriptions (iirc primsa v2 doesn't care about subscription :man_shrugging:). Also if the Archive node with graphQL is getting to the finish line within a couple months, we might want to go this way and wait a bit.

TL;DR do what you feel is the best for you, we'll keep nodewatcher alive for quite some time because we can't afford to refactor this much now.

[pmespresso]

Thanks for your input @Tbaut, that makes the path a bit clearer in my mind.

I think the best next task for me then is to work on a GraphQL layer for Polkascan.

See https://www.gatsbyjs.org/docs/data-fetching. Let's take the validator list. At build time, we show an empty . At runtime, it uses fetch or axios to fetch Polkascan's endpoint to populate the table.

This makes sense of course but what I was talking about was not about that, but routing 😅 - e.g. to create a route like /accounts/G1R1PJd3gdDLTT6LyYdhXEz3Cjrj5XvForrMUXW2bx4nfcP. https://www.gatsbyjs.org/docs/node-apis/#createPages

[pmespresso]

as promised https://github.com/yjkimjunior/Polkascan-GQL