search

paritytech / substrate-api-sidecar

REST service that makes it easy to interact with blockchain nodes built using Substrate's FRAME framework.
https://paritytech.github.io/substrate-api-sidecar/dist/
GNU General Public License v3.0
246 stars 153 forks source link

readme



@substrate/api-sidecar

REST service that makes it easy to interact with blockchain nodes built using Substrate's FRAME framework.

npm Github Actions GPL-3.0-or-later



Prerequisites

<= v15.0.0

This service requires Node versions 14 or higher.

Compatibility: Node Version Stablility
v14.x.x Stable
v16.x.x Stable
v17.x.x Stable
v18.x.x Stable
v19.x.x stable

>= v16.0.0

This service requires Node versions 18.14 or higher.

Compatibility: Node Version Stablility
v18.14.x Stable
v20.x.x Stable
v21.x.x Pending

NOTE: Node LTS (long term support) versions start with an even number, and odd number versions are subject to a 6 month testing period with active support before they are unsupported. It is recommended to use sidecar with a stable actively maintained version of node.js.

Table of contents

NPM package installation and usage

Global installation

Install the service globally:

npm install -g @substrate/api-sidecar
# OR
yarn global add @substrate/api-sidecar

Run the service from any directory on your machine:

substrate-api-sidecar

To check your version you may append the --version flag to substrate-api-sidecar.

Local installation

Install the service locally:

npm install @substrate/api-sidecar
# OR
yarn add @substrate/api-sidecar

Run the service from within the local directory:

node_modules/.bin/substrate-api-sidecar

Finishing up

Jump to the configuration section for more details on connecting to a node.

Click here for full endpoint docs.

In the full endpoints doc, you will also find the following trace related endpoints :

To have access to these endpoints you need to :

  1. Run your node with the flag —unsafe-rpc-external
  2. Check in sidecar if BlocksTrace controller is active for the chain you are running.

Currently BlocksTrace controller is active in Polkadot and Kusama.

Source code installation and usage

Quick install

Simply run yarn.

Rust development installation

If you are looking to hack on the calc Rust crate make sure your machine has an up-to-date version of rustup installed to manage Rust dependencies.

Install wasm-pack if your machine does not already have it:

cargo install wasm-pack

Use yarn to do the remaining setup:

yarn

Running

# For live reload in development
yarn dev

# To build and run
yarn build
yarn start

Jump to the configuration section for more details on connecting to a node.

Configuration

To use a specific env profile (here for instance a profile called 'env.sample'):

NODE_ENV=sample yarn start

For more information on our configuration manager visit its readme here. See Specs.ts to view the env configuration spec.

Express server

Substrate node

Metrics Server

Custom substrate types

Some chains require custom type definitions in order for Sidecar to know how to decode the data retrieved from the node. Sidecar affords environment variables which allow the user to specify an absolute path to a JSON file that contains type definitions in the corresponding formats. Consult polkadot-js/api for more info on the type formats (see RegisteredTypes). There is a helper CLI tool called generate-type-bundle that can generate a typesBundle.json file for you using chain information from @polkadot/apps-config. The generated json file from this tool will work directly with the SAS_SUBSTRATE_TYPES_BUNDLE ENV variable.

You can read more about defining types for polkadot-js here.

Connecting a modified node template

Polkadot-js can recognize the standard node template and inject the correct types, but if you have modified the name of your chain in the node template you will need to add the types manually in a JSON types file like so:

// my-chains-types.json
{
  "Address": "AccountId",
  "LookupSource": "AccountId"
}

and then set the enviroment variable to point to your definitions:

export SAS_SUBSTRATE_TYPES=/path/to/my-chains-types.json

Logging

Log levels

Log levels in order of decreasing importance are: error, warn, info, http, verbose, debug, silly.

http status code range log level
code < 400 http
400 <= code < 500 warn
500 < code error

RPC logging

If looking to track raw RPC requests/responses, one can use yarn start:log-rpc to turn on polkadot-js's logging. It is recommended to also set SAS_LOG_STRIP_ANSI=true to increase the readability of the logging stream.

N.B. If running yarn start:log-rpc, the NODE_ENV will be set to test. In order still run your .env file you can symlink it with .env.test. For example you could run ln -s .env.myEnv .env.test && yarn start:log-rpc to use .env.myEnv to set ENV variables. (see linux commands ln and unlink for more info.)

Prometheus server

Prometheus metrics can be enabled by running sidecar with the following env configuration: SAS_METRICS_ENABLED=true

You can also expand the metrics tracking capabilities to include query params by adding to the env configuration: SAS_METRICS_INCLUDE_QUERYPARAMS=true

The metrics endpoint can then be accessed :

A JSON format response is available at http://127.0.0.1:9100/metrics.json.

That way you will have access to the default prometheus node instance metrics and the following metrics will be emitted for each route:

The blocks controller also includes the following route-specific metrics:

The metrics registry is injected in the Response object when the SAS_METRICS_ENABLED flag is set to true in the .env file, allowing to extend the controller based metrics to any given controller from within the controller functions.

To successfully run and access the metrics and logs in Grafana (for example) the following are required:

For mac users using homebrew:

brew install prometheus loki promtail

Debugging fee and staking payout calculations

It is possible to get more information about the fee and staking payout calculation process logged to the console. Because these calculations happens in the statically compiled web assembly part, a re-compile with the proper environment variable set is necessary:

CALC_DEBUG=1 sh calc/build.sh

Available endpoints

Click here for full endpoint docs.

Chain integration guide

Click here for chain integration guide.

Docker

With each release, the maintainers publish a docker image to dockerhub at parity/substrate-api-sidecar

Pull the latest release

docker pull docker.io/parity/substrate-api-sidecar:latest

The specific image tag matches the release version.

Or build from source

yarn build:docker

Run

# For default use run:
docker run --rm -it --read-only -p 8080:8080 substrate-api-sidecar

# Or if you want to use environment variables set in `.env.docker`, run:
docker run --rm -it --read-only --env-file .env.docker -p 8080:8080 substrate-api-sidecar

NOTE: While you could omit the --read-only flag, it is strongly recommended for containers used in production.

then you can test with:

curl -s http://0.0.0.0:8080/blocks/head | jq

N.B. The docker flow presented here is just a sample to help get started. Modifications may be necessary for secure usage.

Contribute

Need help or want to contribute ideas or code? Head over to our CONTRIBUTING doc for more information.

Notes for maintainers

Commits

All the commits in this repo follow the Conventional Commits spec. When merging a PR, make sure 1) to use squash merge and 2) that the title of the PR follows the Conventional Commits spec.

Updating polkadot-js dependencies

  1. Whenever the polkadot-js ecosystem releases a new version, it's important to keep up with these updates and review the release notes for any breaking changes or high priority updates. In order to update all the dependencies and resolutions, create a new branch, such as yourname-update-pjs, and then run yarn up "@polkadot/*" in that branch.

  2. Ensure everything is up to date and working by running the following:

    yarn
yarn dedupe
yarn build
yarn lint
yarn test
yarn test:historical-e2e-tests
yarn test:latest-e2e-tests

  3. Commit the dependency updates with a name like chore(deps): update polkadot-js deps (adjust the title based on what was updated; refer to the commit history for examples). Then, wait for it to be merged.

  4. Follow RELEASE.md next if you're working through a full sidecar release. This will involve creating a separate PR where the changelog and versions are bumped.

Maintenance Guide

A more complete list of the maintainer's tasks can be found in the MAINTENANCE.md guide.

Hardware requirements

Disk Space

Sidecar is a stateless program and thus should not use any disk space.

Memory

The requirements follow the default of node.js processes which is an upper bound in HEAP memory of a little less than 2GB thus 4GB of memory should be sufficient.

Running sidecar and a node

Please note that if you run sidecar next to a substrate node in a single machine then your system specifications should improve significantly.

Benchmarks

During the benchmarks we performed, we concluded that sidecar would use a max of 1.1GB of RSS memory.

The benchmarks were:

Hardware specs in which the benchmarks were performed:

Machine type:
n2-standard-4 (4 vCPUs, 16 GB memory)

CPU Platform:
Intel Cascade Lake

Hard-Disk:
500GB

Benchmarks are automatically published in Github pages under the url https://paritytech.github.io/substrate-api-sidecar/dev/bench/. The data in the graphs are updated with every new commit/push in the master branch (refer to the benchmark.yml for more details).