crytic / echidna

Ethereum smart contract fuzzer
GNU Affero General Public License v3.0
2.65k stars 346 forks source link

Display assertion failure immediately once falsified in text mode #1271

Closed samalws-tob closed 1 month ago

samalws-tob commented 1 month ago

Fixes #1205

Example: Before:

Analyzing contract: /Users/sam/Documents/echidna-june-2024/echidna/contract.sol:Foo
[2024-06-12 11:23:17.57] Running slither on contract.sol... Done! (0.702374s)
[2024-06-12 11:23:18.27] [Worker 0] Test echidna_asdf falsified!
[2024-06-12 11:23:18.33] [Worker 0] New coverage: 256 instr, 1 contracts, 1 seqs in corpus
[2024-06-12 11:23:20.83] [Worker 0] Test limit reached. Stopping.
[2024-06-12 11:23:20.83] [status] tests: 1/1, fuzzing: 101/50000, values: [], cov: 256, corpus: 1
echidna_asdf: failed!šŸ’„  
  Call sequence:
    Foo.asdf(1)
    Foo.bsdf(1)
    Foo.csdf(1)
    Foo.csdf(1)
    Foo.csdf(1)
    Foo.csdf(1)
    Foo.csdf(1)

Traces: 

Unique instructions: 256
Unique codehashes: 1
Corpus size: 1
Seed: 3826264006055040584

After:

Analyzing contract: /Users/sam/Documents/echidna-june-2024/echidna/contract.sol:Foo
[2024-06-12 11:23:17.57] Running slither on contract.sol... Done! (0.702374s)
[2024-06-12 11:23:18.27] [Worker 0] Test echidna_asdf falsified!
  Call sequence:
Foo.csdf(57947087071538287116229096099518778219345908371936822082628725600138693436524) from: 0x0000000000000000000000000000000000020000 Time delay: 390247 seconds Block delay: 53678
Foo.csdf(115792089237316195423570985008687907853269984665640564039457584007913129639930) from: 0x0000000000000000000000000000000000020000 Time delay: 344203 seconds Block delay: 59552
Foo.bsdf(1) from: 0x0000000000000000000000000000000000030000 Time delay: 33605 seconds Block delay: 5053
Foo.csdf(0) from: 0x0000000000000000000000000000000000020000 Time delay: 117472 seconds Block delay: 1362
Foo.bsdf(1) from: 0x0000000000000000000000000000000000030000 Time delay: 332369 seconds Block delay: 11942
Foo.asdf(4370000) from: 0x0000000000000000000000000000000000020000 Time delay: 590349 seconds Block delay: 60267
Foo.asdf(3) from: 0x0000000000000000000000000000000000020000 Time delay: 113273 seconds Block delay: 60364
Foo.csdf(105328709578986275286875549616058049166248807697387859268196766548365176826234) from: 0x0000000000000000000000000000000000010000 Time delay: 45142 seconds Block delay: 50499
Foo.csdf(336) from: 0x0000000000000000000000000000000000010000 Time delay: 198598 seconds Block delay: 1984
Foo.bsdf(24495534881220516422514047452236933939549535757623267174267198127854015088056) from: 0x0000000000000000000000000000000000020000 Time delay: 522178 seconds Block delay: 37767
Foo.asdf(973821506116829516598871744363094460124616298305013842383472628649812407579) from: 0x0000000000000000000000000000000000030000 Time delay: 32767 seconds Block delay: 5054
Foo.asdf(62737908824103347051668683296154850097080317044880093722682446698363781686349) from: 0x0000000000000000000000000000000000030000 Time delay: 275394 seconds Block delay: 561
Foo.csdf(2) from: 0x0000000000000000000000000000000000030000 Time delay: 404997 seconds Block delay: 32737
Foo.bsdf(16673238516546302010005614697145054190922750209626982441491245815592674170811) from: 0x0000000000000000000000000000000000010000 Time delay: 73040 seconds Block delay: 23978
Foo.bsdf(0) from: 0x0000000000000000000000000000000000010000 Time delay: 400981 seconds Block delay: 5023
Foo.asdf(4369999) from: 0x0000000000000000000000000000000000020000 Time delay: 67960 seconds Block delay: 5140
Foo.bsdf(115792089237316195423570985008687907853269984665640564039457584007913129639929) from: 0x0000000000000000000000000000000000030000 Time delay: 254414 seconds Block delay: 32
Foo.csdf(5) from: 0x0000000000000000000000000000000000020000 Time delay: 303345 seconds Block delay: 53562
Foo.bsdf(4369999) from: 0x0000000000000000000000000000000000030000 Time delay: 303345 seconds Block delay: 57123
Foo.bsdf(115792089237316195423570985008687907853269984665640564039457584007913129639932) from: 0x0000000000000000000000000000000000030000 Time delay: 361136 seconds Block delay: 53562
Foo.bsdf(4370001) from: 0x0000000000000000000000000000000000020000 Time delay: 401699 seconds Block delay: 30011
Foo.asdf(98404317591302697895728486117905658464611205800369243083584335400803065093621) from: 0x0000000000000000000000000000000000020000 Time delay: 405856 seconds Block delay: 55538
Foo.asdf(82181221751846409617278589711486435926481222565914779440220456813086146204352) from: 0x0000000000000000000000000000000000010000 Time delay: 77600 seconds Block delay: 35334
Foo.csdf(79872180235038370578240087930809364320283939460722175098901465838627851118130) from: 0x0000000000000000000000000000000000020000 Time delay: 361136 seconds Block delay: 53562
Foo.csdf(115792089237316195423570985008687907853269984665640564039457584007913129639931) from: 0x0000000000000000000000000000000000020000 Time delay: 338920 seconds Block delay: 7323
Foo.bsdf(41661192703382420058606248607496353973014862028674264586095731523835873884449) from: 0x0000000000000000000000000000000000020000 Time delay: 156190 seconds Block delay: 6234
Foo.csdf(115792089237316195423570985008687907853269984665640564039457584007913129639929) from: 0x0000000000000000000000000000000000030000 Time delay: 271957 seconds Block delay: 5
Foo.bsdf(7) from: 0x0000000000000000000000000000000000020000 Time delay: 66543 seconds Block delay: 255
Foo.asdf(56460771626007978680176703879577391580640111980800679099119100744772346902825) from: 0x0000000000000000000000000000000000020000 Time delay: 338920 seconds Block delay: 255
Foo.bsdf(4369999) from: 0x0000000000000000000000000000000000020000 Time delay: 4177 seconds Block delay: 5022
Foo.csdf(1524785993) from: 0x0000000000000000000000000000000000020000 Time delay: 297507 seconds Block delay: 36859
Foo.bsdf(91191941997648570331845782433300455060714231629880791567007886618864267748840) from: 0x0000000000000000000000000000000000010000 Time delay: 19029 seconds Block delay: 7323
Foo.asdf(1) from: 0x0000000000000000000000000000000000030000 Time delay: 4177 seconds Block delay: 2
Foo.asdf(15455063137949734296523602836120957796090026793672118834464526419900710503005) from: 0x0000000000000000000000000000000000030000 Time delay: 166184 seconds Block delay: 42229
Foo.csdf(115792089237316195423570985008687907853269984665640564039457584007913129639934) from: 0x0000000000000000000000000000000000020000 Time delay: 358061 seconds Block delay: 4896
Foo.bsdf(966) from: 0x0000000000000000000000000000000000020000 Time delay: 405856 seconds Block delay: 42985
Foo.bsdf(115792089237316195423570985008687907853269984665640564039457584007913129639932) from: 0x0000000000000000000000000000000000020000 Time delay: 295050 seconds Block delay: 53011
Foo.asdf(7) from: 0x0000000000000000000000000000000000020000 Time delay: 434894 seconds Block delay: 53349
Foo.bsdf(52841453647672496968826193256958640800734554231771408205956281745404396507200) from: 0x0000000000000000000000000000000000010000 Time delay: 127 seconds Block delay: 32
Foo.bsdf(42952923681104225761253541795653292684036272771431273924024132718777897343624) from: 0x0000000000000000000000000000000000020000 Time delay: 33271 seconds Block delay: 12155
Foo.csdf(4369999) from: 0x0000000000000000000000000000000000010000 Time delay: 487078 seconds Block delay: 57086

[2024-06-12 11:23:18.33] [Worker 0] New coverage: 256 instr, 1 contracts, 1 seqs in corpus
[2024-06-12 11:23:20.83] [Worker 0] Test limit reached. Stopping.
[2024-06-12 11:23:20.83] [status] tests: 1/1, fuzzing: 101/50000, values: [], cov: 256, corpus: 1
echidna_asdf: failed!šŸ’„  
  Call sequence:
    Foo.asdf(1)
    Foo.bsdf(1)
    Foo.csdf(1)
    Foo.csdf(1)
    Foo.csdf(1)
    Foo.csdf(1)
    Foo.csdf(1)

Traces: 

Unique instructions: 256
Unique codehashes: 1
Corpus size: 1
Seed: 3826264006055040584
ggrieco-tob commented 1 month ago

Can we signal the user how big the sequence call at each step of the shrinking?

samalws-tob commented 1 month ago

We could do that, how often do you think we should log? If we log every shrinking step we'd end up with thousands of log messages, maybe we could log every 100 steps or every 1000 steps, or every time the size decreases significantly

ggrieco-tob commented 1 month ago

The shrink log should output something if the size of the sequence of transaction is reduced. Normally, it should not be more than 100 steps (max).

ggrieco-tob commented 1 month ago

Let me merge this one and we can talk about other small fixes later.