search

ethereum / tests

Common tests for all Ethereum implementations
MIT License
560 stars 319 forks source link

Better test descriptions and tool to generate test list #441

Open axic opened 6 years ago

axic commented 6 years ago

The state tests have a _info section with a comment field, however that rarely contains anything.

It would be nice if there would be a mandatory description field explaining what the test supposed to do and a tool run on the CI would actually generate a table of test name + description + path 😉

axic commented 6 years ago

Example:

name description path
codesizeOOGInvalidSize Create oversized (400000 bytes) contract via a creation transaction stCodeSizeLimit/codesizeOOGInvalidSizeFiller.json
holgerd77 commented 6 years ago

I would very much support this and go as far to suggest that you should make this one of your projects with strategic priority (whatever this means, sounded great 😄 ).

This ethereum/tests repository is wonderful and one of technical parts of the Ethereum ecosystem I admire the most for its thought-throughness and clarity. And this was probably the starting point for various clients to be implemented.

In everyday live this excitement cools down a bit 😺 since I have already spent countless hours figuring out the purpose of test cases (especially if one is not a fluent LLL reader) to get a grip where to start debugging.

A simple comment on every test case would be extremely valuable here. I know this is not the most rewarding kind of work, but if this is done in some various not-too-extensive stages this should be manageable.

axic commented 6 years ago

Related: #340.

ehildenb commented 6 years ago

461 is now checking that the _info block with the fields source and sourceHash are present. @zixuanzh can look into supporting the remaining fields and fixing tests that do not have them.

Also, since the script ./test.py is now asserting that those fields exist, we can have a reliable way to list this information in the same tool, using ./test.py info instead of ./test.py list.

winsvega commented 3 years ago

that would be a great research work. I think with the coverage analizer we can run and see which tests are covering what exactly. and would be nice to have a quiery engine for that

winsvega commented 1 year ago

Since yul tests there are many description comments in the yul file itself outside of json fields. The tests classification remains open