Open protolambda opened 5 years ago
This may also come close to getting #15 done.
block_transition
has blocks
defined rather than block
. This should be singular, no?
@djrtwo Good spot, yes, see comment. Will fix type description
Regarding versions: Do we want to support ranges? (e.g. >=0.5.0
or something)? This seems useful in the long run, but it might be overkill for now (as test runners likely won't implement it anyways).
First of all, credits to @djrtwo and others who have been working on the test formats. Much of the work in this issue is a copy/adaption of theirs.
Motivation:
Definitions
Test types
shuffling
: tests validator shufflingbls
: tests crypto functions of BLSssz
: tests serializationhash_tree_root
: hash-tree-roots (incl. and excl. of signatures)fork_choice
: tests the choice between blocks when determining the head of the chaindeltas
: tests the computation of rewards and penalties, as defined by the spec as deltas.state_transition
: originallybeacon_state
, contains big transition with list of blocksepoch_transition
: tests sub-transitions within an epoch transitionblock_transition
: tests sub-transitions within a block transitionTest suites
A test suite is simply a YAML file in the directory of its test-type.
Suites are:
Format
The choice for just a
fork
field, and noversion
field, stems from #27 and #28An clear example of the format:
Configs
Configs are defined separately, each in its own YAML file.
Since configs are re-used across tests and we don't want to read them as random variables, each field should have a comment. I.e. the default phase0 config should just copy the phase 0 spec comments, alternative versions, like a minimal config, should describe the choice for changes.
The format is really simple, just the names of the constants, followed by their value:
Test case types
shuffling
Tests validator shuffling. See: #10.
Format:
A validator is defined as:
A committee is defined as an inline list of integers, example:
bls
Tests crypto functions of BLS. See #16
ssz
Tests serialization. See #13
hash_tree_root
Tests hash-tree-roots (incl. and excl. of signatures). See #14
fork_choice
Tests the choice between blocks when determining the head of the chain. See #12. (note: proposing some opinionated changes here, to be discussed later, and
-
->_
)deltas
Tests the computation of rewards and penalties, as defined by the spec as deltas. This is a new idea, but worthwhile imho to get one of the most important parts readily testable. Clients that don't work with "deltas" vectors can just wrap their rewarding/slashing functions with a simple function that collects the calculations.
The signature is basically:
state -> (rewards, penalties)
(with state being the point of the state starting from the rewards/penalties entry in the epoch transition)Format to be proposed later.
state_transition
Tests the bigger transitions with a list of blocks, like a so-called "smoke-test". See:
beacon_state
, #21, same test_case format, except no per-case config (per suite now).Format:
epoch_transition
Tests sub-transitions within an epoch transition.
Similar to full
state_transition
test format:Format:
Epoch sub-transition handles:
eth1
justification
crosslinks
rewards_and_penalties
ejections
validator_registry
slashings
exit_queue
finish
block_transition
Tests sub-transitions within a block transition, with a single block input in addition to the initial state.
Similar to full
state_transition
test format:Format:
Block sub-transition handles:
header
randao
eth1
proposer_slashings
attester_slashings
attestations
deposits
voluntary_exits
transfers
Structure
The
configs
andtests
define the top-level folder contents, withtests
containing collections of typed test-suites, named by type.Such tree structure could look like this:
Now let's get something like this standardized, and I can start implementing it :)