This repository is a collection is different scenarios and tests performed on different implementations of GraphQL gateways.
This goals of this repo:
Provide a transparent, accurate and descriptive benchmark testing for different tools:
➡️ All tests are running in Docker containers, with the same configuration.
➡️ During the test, tools like cadvisor are monitoring stats.
➡️ Promethues and Grafana are stores results and stats for every run, and we export and embed it as PNG.
➡️ Every scenario have a different report, based on what it measures.
➡️ We are using a dedicated GitHub Actions runners, with concurrency=1
to make sure tests are running as standalone.
Always be up to date:
➡️ This allows tools to improve over time.
➡️ All code in the repository is open and can be changed by the community.
➡️ Scenarios are executed for every PR, and reports are generated automatically per scenario.
➡️ Renovate keeps all dependencies up-to-date.
Various scenarios
➡️ We are trying to create real-life scenarios based on our experience and our customers.
➡️ Each scenario has differnt setup and measure different stats.
fed-v1-constant-vus-over-time
This scenario runs the following:
This measures the following:
This scenario uses Federation v1 spec with all gateways that supports this kind of specification (not all gateways supports v2 spec).
fed-v1-constant-vus-subgraphs-delay
This scenario runs the same flow as fed-v1-constant-vus-over-time
but with an intentional delay on each upstream Subgraph. This creates more stress and increases memory in the server due to the more inflight requests.
fed-v1-constant-vus-subgraphs-delay-resources
This scenario runs the same flow as fed-v1-constant-vus-subgraphs-delay
but with more resources (CPU and RAM) allocated for the gateway.
fed-v1-ramping-vus
This scenario runs the following:
This measures the following:
This scenario uses Federation v1 spec with all gateways that supports this kind of specification (not all gateways supports v2 spec).