search

status-im / nimbus-eth2

Nim implementation of the Ethereum Beacon Chain
https://nimbus.guide
Other
536 stars 231 forks source link

Add a automated test passed/skipped/failed report #744

Closed mratsim closed 4 years ago

mratsim commented 4 years ago

Discussion with @arnetheduck Similar to https://github.com/status-im/nimbus markdown files

zah commented 4 years ago

Can you clarify the purpose here? We use such a report in Nimbus only because we are not yet passing the entire test suite. Since we cannot reduce the CI result to a simple Good/Failed status, we use the file to track the progress towards "Good".

tersec commented 4 years ago

So, currently 4 tests fail -- https://github.com/status-im/nim-beacon-chain/pull/740 elaborates. Three of those look to be failing on the same not-yet-0.10.1 BLS verify routine. This might be premature.

Also, this should probably be built around https://github.com/status-im/nim-beacon-chain/tree/devel/nbench#running-the-whole-test-suite if it's built at all, not the current per-module setup.

mratsim commented 4 years ago

AFAIK it's because some tests are skipped and we forget to unskip them.

This also avoids trying to fuzz those and then realizing that they didn't work all along.

arnetheduck commented 4 years ago

yes - this is related to tracking precisely in what state we are at any given moment in time - we are not passing the entire eth2 test suite which led to some unnecessary confusion with initial fuzzing attempts

arnetheduck commented 4 years ago

ie when the test suite for a new spec version or feature comes out, we should be able to throw it in there and get a progress bar on how far away we are.

zah commented 4 years ago

To help the implementer, I'll clarify the goal here:

Running the test suite of the Nimbus project results in updating a number of Markdown files that we store in Git. For example: https://github.com/status-im/nimbus/blob/master/BlockchainTests.md

We benefit from the ability to look at the diff in each commit to understand better what tests were fixed or broken by it: https://github.com/status-im/nimbus/commit/15c9fa54ec64b44a518d8b70214fba458d68d624

You'll find the existing markdown generation code here: https://github.com/status-im/nimbus/blob/a70c9d5e10f66b57a1ac3c9a965a26155d7f5d30/tests/test_helpers.nim#L111-L148

See the usages of jsonTest to see all the markdown files that are being generated in this way.

The goal would be to extract this code and to reuse it here, where a similar json test suite exists. If you wish, you can place the extracted code in the testutils package under a new module named testutils/markdown_reports.nim that will provide a simple API allowing you instantiate the contents of the report as data structures that are passed to a generateReport proc.

The reward for this bounty will be $150 (paid in crypto).

zah commented 4 years ago

For the completion of this bounty and https://github.com/status-im/nim-chronicles/issues/73, @zedeus has been awarded 250 USD (paid in ETH):

https://etherscan.io/tx/0x6d7b2520500993209cf3b05d1401c3876122216800e9bbe41b2911019ae1cf19