Open onbjerg opened 2 years ago
Any reason why we cannot avoid the abi encode like this?
struct Case {
uint256 a;
uint256 b;
uint256 expected;
}
function setUpTableSums() public returns (Case[] memory) {
Case[] memory entries = new Case[](2);
entries[0] = Case(1, 2, 3);
entries[1] = Case(4, 5, 9);
}
function testTableSums(Case memory case) public {
assertEq(case.a + case.b, case.expected);
}
Hmm, thinking about it, you're right. Since we have the ABI we would know how to serialize/deserialize the return data
Maybe you could use some magic modifiers that you pull out of the solidity AST at test time? The following could be interpreted as "fuzz testTableSums
, always using the provided concrete inputs as a part of the fuzzing campaign".
modifier inputSums(uint a, uint b, uint expected) { _; }
function testTableSums(uint a, uint b, uint expected)
public
inputSums(1, 2, 3)
inputSums(4, 5, 9)
{
assertEq(a + b, expected);
}
I guess it should also be possible to parse some custom natspec comments with a list of inputs?
+1 to this feature
as a workaround i am actually using a modifier iterating over cases
struct Case {
uint256 a;
uint256 b;
uint256 expected;
}
Case[] entries;
Case c;
modifier parametrizedTest() {
for (uint256 i = 0; i < entries.length; i++) {
c = entries[i];
_;
}
}
function setUp() public override {
entries.push(Case(1, 2, 3));
entries.push(Case(4, 5, 9));
}
function testTableSums() public parametrizedTest {
assertEq(c.a + c.b, c.expected);
}
I guess it should also be possible to parse some custom natspec comments with a list of inputs?
0.8.13 supports the natspec directive of custom now
// from https://github.com/contractshark/vscode-solidity-extenstion/pull/7/files#diff-9b75242a14d24953968395cc5a946df5d3bdda1043d761a91e8d3f10157aa016R117-R120
// https://docs.soliditylang.org/en/latest/natspec-format.html#tags
"natspec-tag-custom": {
"match": "(@custom:)\\b",
"name": "storage.type.custom.natspec"
},
something like @custom:requires $TEST_ACCOUNT
etc etc see https://github.com/gakonst/foundry/issues/979
+1 for this feature. I'm porting my math library to Foundry and there are lots and lots of situations where table tests would be useful.
Possibly dumb question, but couldn't we abstract away the ABI encoding in the case of integers? I imagine that the most needed variable type for table tests will be uint256
.
function setUpTableSums() public returns (uint256[] memory) {
uint256[] memory entries = new uint256[](2);
entries[0] = 4;
entries[1] = 5;
}
Couldn't Foundry ABI encode and decode these, under the hood?
In the meantime, if you're using @emilianobonassi's solution (which is great!) and compiling with Solidity v0.8, consider wrapping the increment in an unchecked
block:
modifier parametrizedTest() {
for (uint256 i = 0; i < entries.length) {
c = entries[i];
_;
unchecked {
i += 1;
}
}
}
This should make the execution slightly faster, especially if you have lots of test cases.
Any reason why we cannot avoid the abi encode like this?
struct Case { uint256 a; uint256 b; uint256 expected; } function setUpTableSums() public returns (Case[] memory) { Case[] memory entries = new Case[](2); entries[0] = Case(1, 2, 3); entries[1] = Case(4, 5, 9); } function testTableSums(Case memory case) public { assertEq(case.a + case.b, case.expected); }
This pattern looks good. It lets the user define their own table entry structure as fancy or dumbed down as they like.
We would definitely use table tests to set up test inputs that reside inside .csv files or json tables. You can do it now, but it is hard to parallelize and can take quite some time if you are forking on many blocks with the parallelization.
In the meantime, if you're using @emilianobonassi's solution (which is great!) and compiling with Solidity v0.8, consider wrapping the increment in an
unchecked
block:modifier parametrizedTest() { for (uint256 i = 0; i < entries.length) { c = entries[i]; _; unchecked { i += 1; } } }
This should make the execution slightly faster, especially if you have lots of test cases.
Unfortunately, I think this solution does not parallelize each entry in the evm test runner, right?
@ChainsightLabs it doesn't, but it's fast enough for my needs.
One downside to @emilianobonassi's solution is that it fudges the gas reports. Extra gas has to be spent on pushing the entries in the storage array.
Unfortunately I can't think of any workaround to this issue .. except for Foundry supporting native table tests and handling the gas metering for us.
One downside to @emilianobonassi's solution is that it fudges the gas reports. Extra gas has to be spent on pushing the entries in the storage array.
Unfortunately I can't think of any workaround to this issue .. except for Foundry supporting native table tests and handling the gas metering for us.
yep @paulrberg
not sure how much will save but you can delete the array entries after the execution in the modifier
another proposal would be do a smoketest which print how much gas a noop test will need for your cases so you can offset from the real tests (this is what i actually use)
another one, but we need to await for eip inclusion, is to support transient storage tload/tstore
Deleting the array doesn't reduce the gas cost significantly (due to EIP-3529).
Transient storage will be great when it is finally available.
Interesting idea with the smoketest but it wouldn't work in my case because I have many different numbers of array entries per test. There's no baseline "noop test".
not sure yet possible in foundry but a cheatcode to change gasleft offsetting in the modifier will make it
the cheatcode will be useful too to test gas-dependant logics
Big brain idea.
should not be hard
underlying evm supports gas reset
https://github.com/bluealloy/revm/blob/main/crates/revm/src/gas.rs#L45
should be properly exposed as cheatcode
https://github.com/foundry-rs/foundry/blob/master/evm/src/executor/inspector/cheatcodes/mod.rs#L177
related to #2429
Any reason why we cannot avoid the abi encode like this?
struct Case { uint256 a; uint256 b; uint256 expected; } function setUpTableSums() public returns (Case[] memory) { Case[] memory entries = new Case[](2); entries[0] = Case(1, 2, 3); entries[1] = Case(4, 5, 9); } function testTableSums(Case memory case) public { assertEq(case.a + case.b, case.expected); }
Feeling strongly about this being the "right" solution here but open minded to alternatives.
I keep coming upon use cases for this feature, does anyone else agree that the priority could be bumped up from Low to something higher?
Sounds good - happy to provide resources for reviewing the PR if somebody would be up for taking a first stab at it. Need to think about if we can actually prioritize this - please give me some time on that.
Just bumped into another limitation of the parametrizedTest
modifier solution proposed above - it doesn't work for tests that expect reverts!
function testTableSums() public parametrizedTest {
vm.expectRevert();
callSomeFunction(c.a, c.b);
}
The test will show as passed, but it will stop looking for additional reverts once it reaches the first revert.
This is probably related to https://github.com/foundry-rs/foundry/issues/3723.
Update: yes, can confirm that this problem is related to #3723, albeit in a very specific circumstance. When you are testing free functions, you cannot have multiple vm.expectRevert
statements for multiple calls to free functions.
bump
@ChainsightLabs in the meantime, take a look at how I implemented parameterized tests in my math library PRBMath:
I have extensively used the parameterizedTest
approach suggested above.
this looks a lot like the new fixtures feature that was introduced recently https://book.getfoundry.sh/forge/fuzz-testing#fuzz-test-fixtures can people check it and share thoughts?
bump, would be great to get feedback on https://book.getfoundry.sh/forge/fuzz-testing#fuzz-test-fixtures
Component
Forge
Describe the feature you would like
Table tests are a way to generate test cases based on a dataset of parameters (the table), enabling code reuse in tests that use the same assertions but with predefined data. An example of a table test can be found here.
There are a few ways to go about this (after brainstorming with @gakonst) these seem like the most viable options:
Setup/run functions
This approach uses a
setUpTable*
function that returns the table with testdata and atestTable*
that runs assertions on each entry in the table. Due to constraints with Solidity (again) the UX is going to suck a bit:The
setUpTable
functions are called once during setup and the table is stored in-memory. ThetestTable
function is then called in parallel with each entry in the table. If an entry fails we should mark the table test as a failure and provide some info on what entries failed.The
setUpTable
functions returns an array of bytes: each entry in this array is calldata we pass directly to the table test function by prepending the signature of the table test function.Pros:
Cons:
External files
Same as above, but the table is in a file named after some convention. Each entry is ABI encoded and then passed to the table test function
Pros:
Cons:
Additional context
No response