search

JoranHonig / vertigo

Mutation Testing for Ethereum Smart Contracts
GNU Lesser General Public License v3.0
203 stars 40 forks source link

No mutants killed #28

Open nfurfaro opened 4 years ago

nfurfaro commented 4 years ago

@JoranHonig Hello. Great work on this project. I've just started playing with it a bit on a fairly large codebase. I ran a full test (--sample-ratio = 1.0) last night, and the results were 0/247 mutants killed. I was suprised by this, so I picked 1 mutation at random and manually tested it by switching the operator and running some tests targeting this bit of code. I quickly had 5 failing tests, which makes me question the results from this.

I'm wondering if there's any way to tweak settings to produce more accurate results? I've tried running a smaller sample-ratio (0.1) which just produced 

Mutation testing report:
Number of mutations:    24
Killed:                 0 / 24

Not sure if I'm doing something wrong here? If so, please feel free to close this issue. For reference, here's the command & options I passed:

vertigo run --network vertigo_test_network_1 --network vertigo_test_network_2 --truffle-location /Users/overlord/DAPPS/unlock/smart-contracts/node_modules/.bin/truffle --output ./vertigo_output.txt --sample-ratio 0.1
JoranHonig commented 4 years ago

Hi @nfurfaro, thanks for opening an issue!

I think there might be an issue in Vertigo here, I'll triage!

nfurfaro commented 4 years ago

No problem. One thing that may be of help to you: In our project we use truffle to compile and test. Often, after making changes to a contract and running truffle test, truffle will fail to pick up the changes made to contracts, and run the tests against the old artifacts instead of re-compiling. The workaround for us is to delete the build directory and recompile.

With vertigo, it may be the case that simply forcing recompilation between test runs could solve our issue. Anyways, happy to help if I can. Please reach out if you need more info.

JoranHonig commented 4 years ago

@nfurfaro Thanks for the tip.

There we're actually two small issues I found.

  1. Truffle doesn't like the pre-existing build directory (like you mentioned, thanks!)
  2. With the feature addition of trivial compiler equivalence I accidentally put a statement in an else branch that I shouldn't have.

Anyway, it should work now! Version 1.1.3 has the fix. I hope you can now use it on your project, tell me how that goes!

nfurfaro commented 4 years ago

@JoranHonig Thanks for working on this so quickly! I'm unable to get it working atm, but it's a different issue. 

 Artifacts written to /var/folders/3_/f3d20vzs5z71mqh32lgkkz2m0000gq/T/test-2020120-92945-1b8b0jo.mjd1
> Compiled successfully using:
   - solc: 0.5.16+commit.9c3226ce.Emscripten.clang

[-] Encountered an error while running the framework's test command:
Error: Contract Unlock not found in path /private/var/folders/3_/f3d20vzs5z71mqh32lgkkz2m0000gq/T/tmpbot0c63s/build/contracts/Unlock.json

I want to triple-check and make sure I'm following the right steps to get it up and running. I'm using this as my reference: https://medium.com/swlh/mutation-testing-for-smart-contracts-a-step-by-step-guide-68c838ca2094

Update: So I've looked at this a bit more. Here's what I see happening in /private/var/folders/3_/f3d20vzs5z71mqh32lgkkz2m0000gq/T/:

moose-code commented 3 years ago

I think I am getting the exact same issue.

`Error: Contract Staker not found in path /private/var/folders/c4/vx4lqxtd5v11mxtr_md_6qdh0000gn/T/tmp4upmx947/build/contracts/Staker.json

Truffle v5.1.54 (core: 5.1.54) Node v14.15.1`

When simply running vertigo run --network development

Have tried deleting the build folder and running again