Our mandate is:
To verify the successful end-to-end outcome of the filecoin protocol and filecoin implementations, under a variety of real-world and simulated scenarios.
➡️ Find out more about our goals, requirements, execution plan, and team culture, in our Project Description.
These are the topics we are currently centering our testing efforts on. Our testing efforts include fault induction, stress tests, and end-to-end testing.
This repository consists of test plans built to be run on Testground.
The source code for the various test cases can be found in the lotus-soup
directory.
If you are unfamiliar with Testground, we strongly suggest you read the Testground Getting Started guide in order to learn how to install Testground and how to use it.
You can find various composition files describing various test scenarios built as part of Project Oni at lotus-soup/_compositions
directory.
We've designed the test cases so that you can run them via the local:exec
, local:docker
and the cluster:k8s
runners. Note that Lotus miners are quite resource intensive, requiring gigabytes of memory. Hence you would have to run these test cases on a beafy machine (when using local:docker
and local:exec
), or on a Kubernetes cluster (when using cluster:k8s
).
Here are the basics of how to run the baseline deals end-to-end test case:
Compile and Install Testground from source code.
Run a Testground daemon
testground daemon
lotus-soup
test planmake pull-images
Alternatively you can build them locally with
make build-images
lotus-soup
test plan into your Testground home directorytestground plan import --from ./lotus-soup
filecoin-ffi
Git submodule in the extra
folder.git submodule update --init --recursive
filecoin-ffi
version locally (necessary if you use local:exec
)cd extra/filecoin-ffi
make
testground run composition -f ./lotus-soup/_compositions/baseline-docker-5-1.toml
The Oni testkit supports range parameters, which test cases can use to generate random values, either at the instance level (each instance computes a random value within range), or at the run level (one instance computes the values, and propagates them to all other instances via the sync service).
For example:
latency_range = '["20ms", "500ms"]'
loss_range = '[0, 0.2]'
Could pick a random latency between 20ms and 500ms, and a packet loss
probability between 0 and 0.2. We could apply those values through the
netclient.ConfigureNetwork
Testground SDK API.
Randomized range-based parameters are specially interesting when combined with batch runs, as it enables Monte Carlo approaches to testing.
The Oni codebase includes a batch test run driver in package lotus-soup/runner
.
You can point it at a composition file that uses range parameters and tell it to
run N iterations of the test:
$ go run ./runner -runs 5 _compositions/net-chaos/latency.toml
This will run the test as many times as instructed, and will place all outputs
in a temporary directory. You can pass a concrete output directory with
the -output
flag.
lotus-soup
deals-e2e
- Deals end-to-end test case. Clients pick a miner at random, start a deal, wait for it to be sealed, and try to retrieve from another random miner who offers back the data.drand-halting
- Test case that instructs Drand with a sequence of halt/resume/wait events, while running deals between clients and miners at the same time.deals-stress
- Deals stress test case. Clients pick a miner and send multiple deals (concurrently or serially) in order to test how many deals miners can handle.paych-stress
- A test case exercising various payment channel stress tests.lotus-soup
baseline-docker-5-1.toml
- Runs a baseline
test (deals e2e test) with a network of 5 clients and 1 miner targeting local:docker
baseline-k8s-10-3.toml
- Runs a baseline
test (deals e2e test) with a network of 10 clients and 3 miner targeting cluster:k8s
baseline-k8s-3-1.toml
- Runs a baseline
test (deals e2e test) with a network of 3 clients and 1 miner targeting cluster:k8s
baseline-k8s-3-2.toml
- Runs a baseline
test (deals e2e test) with a network of 3 clients and 2 miner targeting cluster:k8s
baseline.toml
- Runs a baseline
test (deals e2e test) with a network of 3 clients and 2 miner targeting local:exec
. You have to manually download the proof parameters and place them in /var/tmp
.deals-stress-concurrent-natural-k8s.toml
deals-stress-concurrent-natural.toml
deals-stress-concurrent.toml
deals-stress-serial-natural.toml
deals-stress-serial.toml
drand-halt.toml
local-drand.toml
natural.toml
paych-stress.toml
pubsub-tracer.toml
Find commands and how-to guides on debugging test plans at DELVING.md
Querying the Lotus RPC API
Useful commands / checks
Making sure miners are on the same chain
Checking deals
Sector queries
Sector sealing errors
Our current test plan lotus-soup
is building programatically the Lotus filecoin implementation and therefore requires all it's dependencies. The build process is slightly more complicated than a normal Go project, because we are binding a bit of Rust code. Lotus codebase is in Go, however its proofs
and crypto
libraries are in Rust (BLS signatures, SNARK verification, etc.).
Depending on the runner you want to use to run the test plan, these dependencies are included in the build process in a different way, which you should be aware of should you require to use the test plan with a newer version of Lotus:
local:docker
The Rust libraries are included in the Filecoin FFI Git submodule, which is part of the iptestground/oni-buildbase
image. If the FFI changes on Lotus, we have to rebuild this image with the make build-images
command, where X is the next version (see Docker images changelog
below).
local:exec
The Rust libraries are included via the extra
directory. Make sure that the test plan reference to Lotus in go.mod
and the extra
directory are pointing to the same commit of the FFI git submodule. You also need to compile the extra/filecoin-ffi
libraries with make
.
cluster:k8s
The same process as for local:docker
, however you need to make sure that the respective iptestground/oni-buildbase
image is available as a public Docker image, so that the Kubernetes cluster can download it.
Additional to the Filecoin FFI Git submodules, we are also bundling proof parameters
in the iptestground/oni-runtime
image. If these change, you will need to rebuild that image with make build-images
command, where X is the next version.
v1
=> initial image locking in Filecoin FFI commit ca281af0b6c00314382a75ae869e5cb22c83655b.v2
=> no changes; released only for aligning both images to aesthetically please @nonsense :Dv3
=> locking in Filecoin FFI commit 5342c7c97d1a1df4650629d14f2823d52889edd9.v4
=> locking in Filecoin FFI commit 6a143e06f923f3a4f544c7a652e8b4df420a3d28.v5
=> locking in Filecoin FFI commit cddc56607e1d851ea6d09d49404bd7db70cb3c2e.v6
=> locking in Filecoin FFI commit 40569104603407c999d6c9e4c3f1228cbd4d0e5c.v7
=> add Filecoin-BLST repo to buildbase.v8
=> locking in Filecoin FFI commit f640612a1a1f7a2d.v9
=> locking in Filecoin FFI commit 57e38efe4943f09d3127dcf6f0edd614e6acf68e and Filecoin-BLST commit 8609119cf4595d1741139c24378fcd8bc4f1c475.v1
=> initial image with 2048 parameters.v2
=> adds auxiliary tools: net-tools netcat traceroute iputils-ping wget vim curl telnet iproute2 dnsutils
.v3
=> bump proof parameters from v27 to v28v1
=> initial imagev2
=> locking in Lotus commit e21ea53v3
=> locking in Lotus commit d557c40v4
=> bump proof parameters from v27 to v28v5
=> locking in Lotus commit 1a170e18a