Create local archive endpoint to allow updating indexers before enactment of runtime upgrades

[ebma]

Context

In https://github.com/pendulum-chain/tasks/issues/186 we did some research how we can improve the indexer downtime whenever a runtime upgrade happens. In the past we noticed that the metadata changes introduced by runtime upgrades make our indexers unable to decode some of the events properly.

We now want to implement the agreed outcome of the research task. The idea is the following:

Local archive endpoint

We create a new command or script that:

• Assumes you have the new runtime.wasm file somewhere in the local directory
• The script calls the subwasm tool on runtime.wasm to extract the metadata from it. The metadata is then stored in a file runtime.metadata
• The script starts a new local HTTP server that:
  • Queries the respective production metadata archive, e.g. https://v2.archive.subsquid.io/metadata/foucoco
  • Reads the content of the local runtime.metadata file we created earlier and creates a new JSON object of the form {"blockNumber":0,"blockHash":"xxx","specName":"foucoco","specVersion":<the_new_spec_version>,"metadata":"<content_of_runtime.metadata>"} with it
  • Serves GET requests with the concatenation of the original metadata archive response + the custom JSON

TODO

• Implement the server for the local archive endpoint. The server can be implemented using express or just the http-server package. It should live in an extra folder. It does not need to live in the src folder with the rest of the squid files so we can reduce the overhead by using Javascript over Typescript.
• Add a command to commands.json that first runs subwasm to extract the metadata and then starts the local server.
• (Optional) Find a way to automatically run the respective typegen-{network} command to regenerate the types. This might be tricky as the typegen commands need to know the network they are used for and we need to adjust the contained specVersions endpoint eg. here.

[ebma]

@pendulum-chain/product this ticket is relevant for improving the downtime of our indexers when runtime upgrades happen.

[TorstenStueber]

@ebma the ticket is quite clear but a I have some question about the context:

• How does subsquid extend the response returned by https://v2.archive.subsquid.io/metadata/foucoco? Is it always generated from the current production version of the indexer so that after deploying the indexer that additionally supports the new metadata version (generated using the approach from this ticket here), then that endpoint would also respond with the additional metadata?
• You specify "blockNumber": 0 above, however, the data returned by the endpoint seems to contain the actual block number for the runtime upgrades. Do we have reason to believe that it works when just specifying "blockNumber": 0?
• The tool described here in this ticket is just one step of preparing the squid for a runtime upgrade. What would we need to do next, just run typegen? I assume that this is not enough yet: in the repository there is currently only code for Foucoco spec versions 1, 4 and 13 although the metadata endpoint returns metadata for all spec versions from 1 to 18 (except versions 5 and 6 for some reason).

[prayagd]

Hey team! Please add your planning poker estimate with Zenhub @ebma @gianfra-t @TorstenStueber @bogdanS98

[ebma]

The archive endpoints are not linked to any of our deployed squids. I checked the docs and apparently they know call these archives, the 'Subsquid network'.

Subsquid Network is a distributed query engine and data lake that the Subsquid ecosystem is built around. Currently it serves historical blockchain data for 80+ networks.

As far as I understand, they have a service that reads our blockchain data and derives info like the one you can see at the archive endpoint. So it doesn't matter what's happening in our indexers; once a runtime upgrade goes live on our production endpoints, the archive endpoint should provide the new metadata soon after.

Do we have reason to believe that it works when just specifying "blockNumber": 0?

I checked the code of the typegen package but I didn't find any meaningful mention of the blocknumber so I would assume it's safe to ignore. As long as the spec versions are numbered in ascending order, the block number should not matter.

What would we need to do next, just run typegen?

Yes, I think re-generating the types with typegen and then adding the new version number (if any) to the respective array here is enough. As far as I know, the typegen command only focuses on the metadata changes to events, calls and storage items we define in the respective typegen file, see eg. here. That's why it's not problematic that some runtime upgrades seem to be ignored and we don't have type files for each version number.