Open ericsson49 opened 4 years ago
The test scenario files are generated by augmenting our current Kotlin fork choice tests, the code is not part of a public repo yet.
I'm currently splitting the tests into generator
and executor
parts, so that generator
produce test cases and executor
runs them.
I'm also going to add invocation of the executable python spec code, so that "golden" answers are produced by the spec. And probably it makes sense to re-write the Kotlin generator code in Python as well.
I will post a link to a repo, when the generator
part is ready (working on it right now).
I've pushed test executor and other test updates. https://github.com/harmony-dev/beacon-chain-java/tree/tests/fork-choice-integration-tests/qa/src/test/kotlin/qa
There are two Kotlin files with tests, one for block tests and another one for attestations. They can be run using IDEA.
When run, they will generate yaml test scenario files. Which can be run by test_executor.
Currently, we are using pseudo BLS signature implementation so that tests run faster. There are also other sources of incompatibilities (like file format extra fields), but I am working on fixing them.
New test scenarios attached fork_choice_integration_tests.zip
I've finally adapted fork choice tests so that their format is aligned with the existing eth2.0-spec-tests. Tests are for v0.10.1 tag - I have validated them with Teku, which currently supports v0.10.1. Tests are validated against v0.10.1 specification too. I've put tests here
There are problems to be resolved yet:
However, it closely matches the existing tests format, so, it will be easier to adopt.
In general, I would propose another format, where parts are assembled into one file. Another idea is to move common test parts to a common directory. Test format becomes more like a graph. However, it will occupy much less disk space and will require less I/O, which is important if many tests are to be generated.
@ericsson49 Great, thanks for this work :).
Regarding the problems:
If we want to ship this fast I think it's best to provide states and blocks as .ssz
as well, so that each client can enjoy the tests. But if you have more time to put into this, we could rethink the testing format to solve the size/duplication issues before releasing the tests.
@protolambda Thanks for the comments!
I do want to put more time. I want to develop or generate more tests to improve coverage and disk space. Most tests do not require BLS and so can be run pretty fast. So the main problem is disk usage. I will convert tests to a new format then in near future.
@protolambda I've converted tests to a new format and put in a new branch.
Two main ideas:
cache
. A test description contains references to the parts instead of the parts themselves (a path to file currently, i.e. ../cache/block_0.yaml
).At the moment, parts are blocks, states and attestations. But they can be any top-level containers, defined by eth2.0 specs. Sub-parts like AttestationData will be more difficult to handle.
Test description is a dict
. For example, for fork choice tests there are following keys:
meta
- can contain bls_setting
, for examplegenesis
steps
- a list of steps, which are applied sequentially:
An additional optimization is that cache
directory can archived and compressed. It saves disc space considerably. Client impelementers can unpack it, if they like or load the whole archive in memory (should probably change archive format to tar.gz
or zip
).
The link above illustrates the ideas. The storage savings are significant:
genesis.yaml
is duplicatedAs a result, there is about 1.5K data per test on average, plus cached stuff, which is additionally 1K or 5K on average, depending on whether cache is compressed or not. Cache size is likely to grow sublinearly when new tests are added.
Are you hand specifying the tests to ensure reuse of the cache or do you have a method to deduplicate given a set of tests?
I record events during a test run, and when converting the trace to a file, I'm using a de-dup cache. It's not an external procedure currently.
In general, I'm thinking about three-four levels of de-duplication:
The proposed test format solves the first issue. The second one may be interesting, since implementations can cache Store
instances and thus, avoid, re-applying same sequences again and again.
I've generated a version fork choice tests for v012x branch and put them here.
Code to generate and to run the tests against pyspec phase0
are put into the branch.
The tests currently do not support optional checks. Optional checks may be required, if some client do not buffer early blocks/attestations and just drop them. Currently, the tests assume such messages are buffered and re-inserted when appropriate (e.g. current slot incremented or a referenced block has arrived). Therefore, such checks can be made optional, to support variations in implementations, though I personally believe it's not a good idea.
My priority has been to agree on the fork choice test format and release some official fork choice integration tests first. Generating lots of tests has not been a priority, since it can be somewhat painful to run and debug fork choice tests. Therefore, it's better to keep the test suite size reasonably small for the beginning. I can generate more tests, after everything is settled with the initial version.
Test generation code also does not yet conform to beacon chain test generators rules and guidelines.
I've created a version of fork choice test format, which allows optional checks, e.g.:
block_in_store:
value: '0x654dd99167c3b155924aad2a19086a70f89daa2b0cb86effb07a3e0488c5669c'
optional: false
instead of just
block_in_store: '0x654dd99167c3b155924aad2a19086a70f89daa2b0cb86effb07a3e0488c5669c'
I've updated tests to match v0.12.2.
Tests are here and here (optional checks). Code to generate the tests is here and here (optional checks).
I've updated tests to match v0.12.2.
Tests are here and here (optional checks). Code to generate the tests is here and here (optional checks).
Thanks! This is great work. Are we planning to merge these into the official eth2.0-spec-tests repo?
Thanks! This is great work. Are we planning to merge these into the official eth2.0-spec-tests repo? Thanks! They were supposed to be merged, but I do not know current EF plans. @djrtwo could you please clarify?
The issue is opened to communicate the initial draft version of the proposed format for fork choice integration-style compatibility test suite. It's an early stage work in progress and will change. However, the main idea will stay the same. It is to be moved to a GitHub repo at some point, when it's mature enough.
There will be common files like spec constants and genesis states, but this is to be defined. There is a file for each test case (yaml format currently, but can be SSZ as well). Each test consists of test steps, each step being either an event (slot, block or attestation) or a condition to verify. There is also an initial state (should be spec constants plus genesis state, however the last one is lengthy, currently only amount of validators is specified in the file. Test file format:
A file with several examples attached Stale integration_tests_on_attestation.zip
New version fork_choice_integration_tests.zip