foundry-rs / foundry

Foundry is a blazing fast, portable and modular toolkit for Ethereum application development written in Rust.
https://getfoundry.sh
Apache License 2.0
8.35k stars 1.77k forks source link

Noticeable change in gas estimation when using `forge test --gas-report` #6762

Closed jalextowle closed 7 months ago

jalextowle commented 10 months ago

Component

Forge

Have you ensured that all of these are up to date?

What version of Foundry are you on?

forge 0.2.0 (68c3663 2024-01-11T00:37:40.946762000Z)

What command(s) is the bug in?

forge test --gas-report

Operating System

Linux

Describe the bug

After Monday's release (d46bcb3ddc768b35da8d1d29f85886476f974128), I noticed that our Solidity benchmarks in CI got noticeably worse on a PR that shouldn't have a large impact on gas costs. After noticing this and being puzzled, I opened another PR with similarly minimal changes, and the benchmark had changes on the same order of magnitude. The benchmark below shows the difference (left is the newer commit using the new version of foundry and right is the old benchmark using the old version of foundry):

Hyperdrive Gas Benchmark

Benchmark suite Current: 04cb6ca547fac568122625ee05c5189245314e37 Previous: aadaa9950e44c58da81ccfaacc5f71e7158dd286 Deviation Status
addLiquidity: min 1600 gas 1600 gas 0% 🟰
addLiquidity: avg 66750 gas 55098 gas 21.1478% 🚨
addLiquidity: max 275046 gas 232446 gas 18.3268% 🚨
checkpoint: min 1172 gas 1172 gas 0% 🟰
checkpoint: avg 48038 gas 47937 gas 0.2107% 🚨
checkpoint: max 202123 gas 202123 gas 0% 🟰
closeLong: min 1580 gas 1580 gas 0% 🟰
closeLong: avg 27900 gas 26510 gas 5.2433% 🚨
closeLong: max 147816 gas 139624 gas 5.8672% 🚨
closeShort: min 1549 gas 1549 gas 0% 🟰
closeShort: avg 29995 gas 28777 gas 4.2325% 🚨
closeShort: max 147189 gas 144389 gas 1.9392% 🚨
initialize: min 1538 gas 1538 gas 0% 🟰
initialize: avg 214974 gas 181786 gas 18.2566% 🚨
initialize: max 256760 gas 256760 gas 0% 🟰
openLong: min 1509 gas 1509 gas 0% 🟰
openLong: avg 50165 gas 40502 gas 23.8581% 🚨
openLong: max 185904 gas 162207 gas 14.6091% 🚨
openShort: min 1519 gas 1519 gas 0% 🟰
openShort: avg 47392 gas 47335 gas 0.1204% 🚨
openShort: max 161536 gas 161536 gas 0% 🟰
redeemWithdrawalShares: min 1575 gas 1575 gas 0% 🟰
redeemWithdrawalShares: avg 23340 gas 18465 gas 26.4013% 🚨
redeemWithdrawalShares: max 105958 gas 84767 gas 24.9991% 🚨
removeLiquidity: min 1661 gas 1661 gas 0% 🟰
removeLiquidity: avg 148623 gas 118880 gas 25.0193% 🚨
removeLiquidity: max 323560 gas 258848 gas 25% 🚨
Evalir commented 7 months ago

We've recently improved gas estimation considerably, with the --isolate flagβ€”Will close this for now, but feel free to reopen if there's a still an issue regarding this.