A set of GO and HTTP tools for testing implementation compliance with https://specs.ipfs.tech
Gateway Conformance test suite is a set of tools for testing implementation
compliance with a subset of IPFS Specifications. The
test suite is implementation and language-agnostic. Point gateway conformance test
at HTTP endpoint and specify which tests should run.
IPFS Shipyard uses it for ensuring specification compliance of the boxo/gateway
library included in Kubo, the most popular IPFS implementation,
that powers various public gateways, IPFS Desktop, and Brave.
Some scenarios in which you may find this project helpful:
The gateway-conformance
can be run as a standalone binary, a Docker image, or a part of Github Action.
Some of the tests require the tested gateway to be able to resolve specific fixtures CIDs or IPNS records.
Two high level commands exist:
$ # Install the gateway-conformance binary
$ go install github.com/ipfs/gateway-conformance/cmd/gateway-conformance@latest
$ # skip path gateway tests, and run subdomain-gateway tests against endpoint at http://127.0.0.1:8080 and use *.ipfs.example.com subdomains, output as JSON
$ gateway-conformance test --gateway-url http://127.0.0.1:8080 --subdomain-url http://example.com:8080 --json report.json --specs +subdomain-gateway,-path-gateway -- -timeout 5m
[!TIP] If want skip individual tests, or only run specific ones based on a regex, see
/docs/examples
.
The gateway-conformace
requires golang runtime to be present to facilitate go test
.
If you want to run it on a box without having to instal golang runtime, prebuilt image at ghcr.io/ipfs/gateway-conformance
is provided.
It can be used for both test
and extract-fixtures
commands:
$ # extract fixtures to ./extracted-fixtures directory under the current user's permissions
$ docker run -u "$(id -u):$(id -g)" -v "$(pwd):/workspace" -w "/workspace" ghcr.io/ipfs/gateway-conformance:latest extract-fixtures --directory extracted-fixtures
$ # skip path gateway tests, and run subdomain-gateway tests against endpoint at http://127.0.0.1:8080 and use *.ipfs.example.com subdomains, output as JSON
$ docker run --net=host -u "$(id -u):$(id -g)" -v "$(pwd):/workspace" -w "/workspace" ghcr.io/ipfs/gateway-conformance:latest test --gateway-url http://127.0.0.1:8080 --subdomain-url http://example.com:8080 --json report.json --specs +subdomain-gateway,-path-gateway -- -timeout 5m
[!IMPORTANT]
- for stable CI/CD, replace
latest
with a semantic version version you want to test against-u
ensures extracted fixtures and created report files can be read by your local user, make sure to adjust it to suit your use case
Common operations are possible via reusable GitHub actions:
ipfs/gateway-conformance/.github/actions/test
ipfs/gateway-conformance/.github/actions/extract-fixtures
To learn how to integrate them in the CI of your project, see real world examples in:
kubo/../gateway-conformance.yml
(fixtures imported into tested kubo node that exposes HTTP gateway feature)boxo/../gateway-conformance.yml
(fixtures imported into a sidecar kubo node that is peered with small HTTP server used for testing boxo/gateway
library)rainbow/../gateway-conformance.yml
(fixtures imported into a kubo node that acts as a remote block provider, than tested against different boxo/gateway
backends)See test
and extract-fixtures
documentation at /docs/commands.md
Want to test mature specs, while disabling specific specs?
Or only test a specific spec (like trustless gateway), while disabling a sub-part of it (only blocks and CARS, no IPNS)?
See /docs/examples.md
The main
branch may contain tests for features and IPIPs which are not yet
supported by stable releases of IPFS implementations.
Due to this, implementations SHOULD test themselves against a stable release of this test suite instead.
See /releases
for the list of available releases.
Want to improve the conformance test suite itself?
See documentation at /docs/development.md
Interested in write a new test case?
Test cases are written in Domain Specific Language (DLS) based on Golang.
More details at /docs/test-dsl-syntax.md
Conformance test suite output can be plain text or JSON, which in turn can be represented as a web dashboard which aggregates results from many test runs and renders them on a static website.
Experimental Implementation Dashboard instance at conformance.ipfs.tech is a view that showcases some of well known and complete implementations of IPFS Gateways in the ecosystem.
Learn more at /docs/web-dashboard.md
This project is dual-licensed under Apache 2.0 and MIT terms: