Passed tests accrue a score instead of simple PASS/FAIL system

[escottalexander]

Matt had the idea that we could make tests less of a PASS/FAIL and instead score challengers based on some additional tests.

For instance, basic functionality will be required and may give the user 75 percent of their grade. But depending on the challenge we may check any number of other "good to have"-"important" tests. If all the tests pass then they would have a score of 100. This will require us to carefully consider each additional tests importance.

Some of these will be very specific to the challenge but others may be somewhat generic and we can record here any ideas for above and beyond scoring.

Here is one thought I had that may apply to several challenges:

• Reentrancy: we could check the contract to make sure it is not vulnerable to reentrancy as this would be a critical issue

Add any more a comments to this issue.

[steve0xp]

Implementation wise, would you think of doing this by having a script check how many tests pass and assigning scores to each respective test to compile out to a scoring?

[escottalexander]

Implementation wise, would you think of doing this by having a script check how many tests pass and assigning scores to each respective test to compile out to a scoring?

Yeah, something like that. Maybe the essential tests would be in one file and the extra credit tests would live in a second file. Wondering if we could bypass needing to parse the output of the tests by using vm.ffi or the file related cheatcodes to write to a file when each test passes. Thinking about this a lot.

In addition to checking for each test passing I would like to record the gas used for each test as well so that we have that data to further comprise scores.

[escottalexander]

The biggest hurdle IMO is being able to think of several tests that are above and beyond the basic functionality. The more advance the challenges are the easier they will come but some of these contracts are just too simple to have a lot of edge cases.

[swellander]

The biggest hurdle IMO is being able to think of several tests that are above and beyond the basic functionality. The more advance the challenges are the easier they will come but some of these contracts are just too simple to have a lot of edge cases.

I like the idea of scoring gas optimization on every challenge, and then adding bonus features on challenges where they make sense. That way people can still prove their above and beyond knowledge, even for the simpler challenges with no edge cases.