Gateway Conformance

A set of GO and HTTP tools for testing implementation compliance with https://specs.ipfs.tech

About
Usage
- CLI
- Docker
- Github Action
Commands
- Examples
Releases
Development
- Test DSL Syntax
- Web Dashboard
License

About

Gateway Conformance test suite is a set of tools for testing implementation compliance with a subset of IPFS Specifications. The test suite is implementation and language-agnostic. Point gateway conformance test at HTTP endpoint and specify which tests should run.

IPFS Shipyard uses it for ensuring specification compliance of the boxo/gateway library included in Kubo, the most popular IPFS implementation, that powers various public gateways, IPFS Desktop, and Brave.

Some scenarios in which you may find this project helpful:

You are building an product that relies on in-house IPFS Gateway and want to ensure HTTP interface is implemented correctly
You are building an IPFS implementation and want to leverage existing HTTP test fixtures to tell if you are handling edge cases correctly
You want to test if a trustless retrieval endpoint supports partial CARs from IPIP-402
You want to confirm a commercial service provider implemented content-addressing correctly

Usage

The gateway-conformance can be run as a standalone binary, a Docker image, or a part of Github Action.

Some of the tests require the tested gateway to be able to resolve specific fixtures CIDs or IPNS records.

Two high level commands exist:

test (test runner with ability to specify a subset of tests to run)
extract-fixtures (allowing for custom provisioning of how test vectors are loaded into tested runtime)

CLI

$ # Install the gateway-conformance binary
$ go install github.com/ipfs/gateway-conformance/cmd/gateway-conformance@latest

$ # skip path gateway tests, and run subdomain-gateway tests against endpoint at http://127.0.0.1:8080 and use *.ipfs.example.com subdomains, output as JSON
$ gateway-conformance test --gateway-url http://127.0.0.1:8080 --subdomain-url http://example.com:8080 --json report.json --specs +subdomain-gateway,-path-gateway -- -timeout 5m

[!TIP] If want skip individual tests, or only run specific ones based on a regex, see /docs/examples.

Docker

The gateway-conformace requires golang runtime to be present to facilitate go test. If you want to run it on a box without having to instal golang runtime, prebuilt image at ghcr.io/ipfs/gateway-conformance is provided.

It can be used for both test and extract-fixtures commands:

$ # extract fixtures to ./extracted-fixtures directory under the current user's permissions
$ docker run -u "$(id -u):$(id -g)" -v "$(pwd):/workspace" -w "/workspace" ghcr.io/ipfs/gateway-conformance:latest extract-fixtures --directory extracted-fixtures

$ # skip path gateway tests, and run subdomain-gateway tests against endpoint at http://127.0.0.1:8080 and use *.ipfs.example.com subdomains, output as JSON
$ docker run --net=host -u "$(id -u):$(id -g)" -v "$(pwd):/workspace" -w "/workspace" ghcr.io/ipfs/gateway-conformance:latest test --gateway-url http://127.0.0.1:8080 --subdomain-url http://example.com:8080 --json report.json --specs +subdomain-gateway,-path-gateway -- -timeout 5m

[!IMPORTANT]

for stable CI/CD, replace latest with a semantic version version you want to test against

-u ensures extracted fixtures and created report files can be read by your local user, make sure to adjust it to suit your use case

Github Action

Common operations are possible via reusable GitHub actions:

To learn how to integrate them in the CI of your project, see real world examples in:

kubo/../gateway-conformance.yml (fixtures imported into tested kubo node that exposes HTTP gateway feature)
boxo/../gateway-conformance.yml (fixtures imported into a sidecar kubo node that is peered with small HTTP server used for testing boxo/gateway library)
rainbow/../gateway-conformance.yml (fixtures imported into a kubo node that acts as a remote block provider, than tested against different boxo/gateway backends)

Commands

See test and extract-fixtures documentation at /docs/commands.md

Examples

Want to test mature specs, while disabling specific specs? Or only test a specific spec (like trustless gateway), while disabling a sub-part of it (only blocks and CARS, no IPNS)? See /docs/examples.md

Releases

The main branch may contain tests for features and IPIPs which are not yet supported by stable releases of IPFS implementations.

Due to this, implementations SHOULD test themselves against a stable release of this test suite instead.

See /releases for the list of available releases.

Development

Want to improve the conformance test suite itself? See documentation at /docs/development.md

Test DSL Syntax

Interested in write a new test case? Test cases are written in Domain Specific Language (DLS) based on Golang. More details at /docs/test-dsl-syntax.md

Web Dashboard

Conformance test suite output can be plain text or JSON, which in turn can be represented as a web dashboard which aggregates results from many test runs and renders them on a static website.

Experimental Implementation Dashboard instance at conformance.ipfs.tech is a view that showcases some of well known and complete implementations of IPFS Gateways in the ecosystem.

Learn more at /docs/web-dashboard.md

License

This project is dual-licensed under Apache 2.0 and MIT terms:

Apache License, Version 2.0, (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)

ipfs / gateway-conformance

readme