search

foundry-rs / foundry

Foundry is a blazing fast, portable and modular toolkit for Ethereum application development written in Rust.
https://getfoundry.sh
Apache License 2.0
7.89k stars 1.59k forks source link

Performance regression for invariant tests #7503

Closed frontier159 closed 3 months ago

frontier159 commented 3 months ago

Component

Forge

Have you ensured that all of these are up to date?

What version of Foundry are you on?

forge 0.2.0 (563e062 2024-03-26T00:16:13.950236000Z)

What command(s) is the bug in?

forge test

Operating System

macOS (Apple Silicon)

Describe the bug

A performance regression has been introduced to invariant tests - they've slowed down considerably with a recent foundryup.

Using a version from early March:

> foundryup --version nightly-de33b6af53005037b463318d2628b5cfcaf39916
> time forge test --mt invariant --offline
...
Suite result: ok. 5 passed; 0 failed; 0 skipped; finished in 13.90s (63.18s CPU time)

Ran 1 test suite in 13.91s (13.90s CPU time): 5 tests passed, 0 failed, 0 skipped (5 total tests)
forge test --mt invariant --offline  62.56s user 0.56s system 431% cpu 14.636 total

Using latest:

> foundryup # latest
> time forge test --mt invariant --offline
...
Suite result: ok. 5 passed; 0 failed; 0 skipped; finished in 80.10s (344.30s CPU time)

Ran 1 test suite in 80.12s (80.10s CPU time): 5 tests passed, 0 failed, 0 skipped (5 total tests)
forge test --mt invariant --offline  327.82s user 2.22s system 409% cpu 1:20.62 total
klkvr commented 3 months ago

Do you have any handlers using vm.assume? #7309 could've resulted in longer runs for such suites which is actually just happening because deeper individual runs are done more often

frontier159 commented 3 months ago

Do you have any handlers using vm.assume? #7309 could've resulted in longer runs for such suites which is actually just happening because deeper individual runs are done more often

I don't use vm.assume, I also don't think I use any of the StdCheats which call that either.

Unfortunately a private repo, would be tricky/time consiuming to try and repro it. Any other repos like sablier showing the symptoms?

frontier159 commented 3 months ago

Actually @klkvr you can reproduce in a slightly older public snapshot of the repo: https://github.com/TempleDAO/origami-public/tree/main/apps/protocol/

  1. clone with submodules
  2. nvm use
  3. yarn
  4. time forge test --mt invariant

Will also need $MAINNET_RPC_URL set

grandizzy commented 3 months ago

had a look at, looks like hashbrown performance better than std's is due to the introduction of indexset for state (in https://github.com/foundry-rs/foundry/pull/7395)

Ran 1 test suite in 21.24s (21.22s CPU time): 5 tests passed, 0 failed, 0 skipped (5 total tests)

vs

Ran 1 test suite in 81.53s (81.51s CPU time): 5 tests passed, 0 failed, 0 skipped (5 total tests)

CC @DaniPopes or could be the introduction of indexset for state (instead regular hashmap) wdyt? any thoughts if this could be improved somehow?

LE: it looks like using std BTreeSet instead IndexSet yields better performance....

Ran 1 test suite in 27.33s (27.30s CPU time): 5 tests passed, 0 failed, 0 skipped (5 total tests)
DaniPopes commented 3 months ago

hm yeah we do try to insert the entire EVM stack on step. I believe it's fxhash, I hadn't considered fxhash was this slow for 32-byte values, @grandizzy can you try by just removing BuildHasherDefault<fxhash> in the type alias so it defaults to the default std hasher?

grandizzy commented 3 months ago

BuildHasherDefault

@DaniPopes yep, much better (even than original run)

Ran 1 test suite in 19.11s (19.08s CPU time): 5 tests passed, 0 failed, 0 skipped (5 total tests)

want me to do a PR or will you?

DaniPopes commented 3 months ago

Go ahead

frontier159 commented 3 months ago

Fantastic I'll check on the next nightly thanks!