Using it on a Rails app

[radanskoric]

Is anyone using this on a Rails app?

I'm sure it's possible but I can not find an example or documentation for the mutant command line parameters to figure out what is the correct way to use it on a rails app.

Any help or pointers appreciated.

[mbj]

@radanskoric Can you tell me what parts of your rails application you plan to mutate, and how this parts are tested? This will help me to give you a more concrete answer.

I expect you are trying to mutate stuff under app/models/**/*.rb? You are using rspec? (mutant currently only supports rspec2)

Do the specs you are plan to use have db or any other type of non local interaction? Beware, mutant currently forks once per mutation kill to isolate mutation effects, this will have effects on connection setup. I'd not encourage to use mutant on any other than plain old in memory unit specs.

[radanskoric]

Hi, yes, I am using rspec2. Some of the tests are fully stubbed and some go to the database.

Well, ideally, I would like to be able to mutate the whole app folder with --rspec-full as well as specific model/controller. For the whole app folder mutation I wouldn't mind waiting for a long time, even for example leaving it running while I go to lunch. I am just interested in getting it's report on coverage every few days of development.

Could I somehow tell it to forgo forking and boot a fresh environment for each mutation kill? (Again, in that case I would be fine letting it run in the background and check the results a few hours later).

[mockdeep]

I'm curious what sorts of side effects it might have. Will it leave data in the database? Maybe something like database_cleaner can help manage it. Also very interested in using it on Rails apps.

[mbj]

Current state: In summary you can use mutant on rails, but with some development the experience will be much better.

Problems:

• The typical rails way does not create an application namespace. For this reason you have various classes in the toplevel, so mutant cannot easily run over a full library/app like you can see on the virtus asciicast. You have to expicitly select stuff like controllers models.
• In case you have specs that touch the database these needs to be cleaned between mutations for clean results. Currently isolation is done doing a fork(), mutant currently does not have a hook for additional isolation.
• Strategies like --rspec-fulll that are running the equivalent of rspec spec are not a good idea to kill mutations. Also the currenty building strategies are not a good fit for a typical rails app.

You can target specific subjects this way, but it will not be fun to work with:

mutant -r ./config/environment.rb ::YourModeControllerOrSomething --rspec-full

But do not expect such great results. But please forward all results, positive an negative to my person. I'd really like to support rails as a first class citizen. This would definitely end up on the roadmap if such a thing would exist ;)

[dkubb]

@mbj

The typical rails way does not create an application namespace. For this reason you have various classes in the toplevel, so mutant cannot easily run over a full library/app like you can see on the virtus asciicast. You have to expicitly select stuff like controllers models.

If it were possible to specify 1 to n "root" namespaces that might help with the first point; it's probably a good idea not to assume a single root anyway. It would take some configuration, but I think it's doable, especially given the benefits.

In case you have specs that touch the database these needs to be cleaned between mutations for clean results.

I think this is pretty much best practice anyway, isn't it? I mean, if your specs don't clean up after themselves then you can't run with the --random switch, and there could be a ton of coupling between the tests. If people aren't doing this, then they should be.

Strategies like --rspec-fulll that are running the equivalent of rspec spec are not a good idea to kill mutations. Also the currenty building strategies are not a good fit for a typical rails app.

If we had the per-class strategy I think it would be a decent alternative. The most common way to organize specs is per-class by far. It's baked into rails and rspec as a convention and all the generators create files using this organization.

It's not as fast as per-method, but I don't think we'll see many people doing that except maybe in DM2 libs or from DM2 developers in their own projects. I've been using this kind of organization the longest, and in my work project I still do per-class, since it's the easiest "sell" to the teams I work with.

[mbj]

@dkubb Mutant already supports many roots:

mutant ::RootA ::RootB Will create a Matcher::Chain instance that chains multiple roots.

Yeah it is best practice to clean database between spec runs that touch it. I see problems where the database_cleaner might not clean because the modification happend in another process. Also the amount of database cleans will be very high as mutant easily generates 10-15 mutations per subject.

The per class strategy should be 90% of the solution for rails. We'll need is for most of the non dm2 gems also. I'll add it in the next releases.

I plan to support multiple styles within the same project, so transitions are also easy to "sell".

[mockdeep]

I'm not at all familiar with the underpinnings of this, but heckle might be a good reference. It tied in with rspec 1 and ran without a hitch on rails 2 projects. I still use it on some older projects I'm maintaining. If you have heckle installed you can run rspec 1 with --heckle.

[dkubb]

@mockdeep When you used heckle with your rails project did you specify specific classes and methods to mutate, eg:

spec spec/**/*_spec.rb --heckle User#full_name

Or did you have some way of automating this and use it across your entire project?

With mutant you can specify an exact method to mutate in the same way. However you can also specify that it mutate all the methods in one or more namespaces, like say User, Order, Account etc. It works even better when everything is namespaced inside your app like Cart::User and Cart::Order for example.

What @mbj and I were discussing above was a way that you could configure all the namespaces so that mutant could just walk through all of them unattended and then report at the end which methods are uncovered.

[mockdeep]

Ah, okay. I would generally specify the class and method to test and the spec to test it with. Something like:

spec --heckle User#full_name spec/models/user_spec.rb

Occasionally I'd run it on the whole class, or run the whole test suite, but that would take forever to complete. It would be great to be able to automatically test relevant classes against their specs, and maybe to fail fast, since it's nice to be able to focus on one thing to test at a time. A mutation server that feeds up one thing at a time would be an amazing tool.

[mbj]

@mockdeep I'll also add a mode where you can explicitly specify the specs used to kill the mutations. The example you added could also be captured via the --rspec-class option we plan to add.

Mutant was initially planned to only support datamapper 2 but I'm very happy it is used outside this project also. I'll support the non dm2 use cases also.

[mockdeep]

Thanks! I loved working with heckle and was really disappointed to see that they didn't intend to update it to support ruby 1.9. It was a bit over my head when I last looked at it, but I may take another look to see if I can contribute to either gem in some way.

[mbj]

@mockdeep Mutant should be a way better than heckle. It was able to find valid mutations and bugs in veritas where hecke already reported 100% mutation coverage. Also the internal code organization of mutant will allow to fix bugs or add support for other use cases quite easy. I'm currently a busy, but I plan to push --rspec-class soon.

[mbj]

@mockdeep played around with a rails project I had access to. The following should work till I release a version where you can configure the behavior better.

Mutant with --rspec-unit`` expects specs underspec/unit. So if you'd like to use mutant now, just create a symbolic link fromspec/modeltospec/unit```.

Than you can run:

mutant -r ./config/environments.rb --rspec-unit ::YourModel

Mutant will waste a lot of time as it runs all specs, but you can try it now. (Make sure you setup and teardown clean db before each test).

[mockdeep]

Hmm, it runs but it doesn't seem to be actually testing anything. I did ln -s spec/models spec/unit and then ran:

mutant -r ./config/environment.rb --rspec-unit ::User

The output:

Mutant configuration:
Matcher:   #<Mutant::Matcher::ObjectSpace scope_name_pattern=/\AUser(\z|::)/>
Filter:    Mutant::Mutation::Filter::ALL
Strategy:  #<Mutant::Strategy::Rspec::Unit>

subjects:   0
mutations:  0
noop_fails: 0
kills:      0
alive:      0
mtime:      0.00s
rtime:      0.30s

The User class has one class method and a handful of associations, but I would've expected at least some mutation.

[mbj]

It did not identified any subject to mutate, hence the subject counter is zero.

Can you add -r ./app/models/user.rb maybe we see have a lazy loading effect, where the class is not fully loaded? Or can you disable lazy loading for a while?

I tested on an app where cashe_classes is true.

[mockdeep]

Now I'm getting an error:

cannot load such file -- /home/fletch/Dropbox/projects/track/spec/unit (LoadError)

[mbj]

Can you post the stacktrace?

[mockdeep]

/home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/activesupport-3.2.11/lib/active_support/dependencies.rb:245:in `load': cannot load such file -- /home/fletch/Dropbox/projects/track/spec/unit (LoadError)
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/activesupport-3.2.11/lib/active_support/dependencies.rb:245:in `block in load'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/activesupport-3.2.11/lib/active_support/dependencies.rb:236:in `load_dependency'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/activesupport-3.2.11/lib/active_support/dependencies.rb:245:in `load'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/rspec-core-2.12.2/lib/rspec/core/configuration.rb:789:in `block in load_spec_files'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/rspec-core-2.12.2/lib/rspec/core/configuration.rb:789:in `each'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/rspec-core-2.12.2/lib/rspec/core/configuration.rb:789:in `load_spec_files'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/rspec-core-2.12.2/lib/rspec/core/command_line.rb:22:in `run'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/rspec-core-2.12.2/lib/rspec/core/runner.rb:77:in `rescue in run'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/rspec-core-2.12.2/lib/rspec/core/runner.rb:73:in `run'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/killer/rspec.rb:32:in `run'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/adamantium-0.0.5/lib/adamantium/module_methods.rb:84:in `call'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/adamantium-0.0.5/lib/adamantium/module_methods.rb:84:in `block (2 levels) in define_memoize_method'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/adamantium-0.0.5/lib/adamantium/module_methods.rb:83:in `fetch'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/adamantium-0.0.5/lib/adamantium/module_methods.rb:83:in `block in define_memoize_method'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/killer.rb:119:in `run_with_benchmark'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/killer.rb:108:in `initialize'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/killer/rspec.rb:17:in `initialize'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/abstract_type-0.0.2/lib/abstract_type.rb:37:in `new'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/abstract_type-0.0.2/lib/abstract_type.rb:37:in `new'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/adamantium-0.0.5/lib/adamantium/class_methods.rb:15:in `new'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/killer/forking.rb:45:in `block in run'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/killer/forking.rb:43:in `fork'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/killer/forking.rb:43:in `run'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/killer.rb:119:in `run_with_benchmark'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/killer.rb:108:in `initialize'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/killer/forking.rb:17:in `initialize'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/abstract_type-0.0.2/lib/abstract_type.rb:37:in `new'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/abstract_type-0.0.2/lib/abstract_type.rb:37:in `new'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/adamantium-0.0.5/lib/adamantium/class_methods.rb:15:in `new'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/killer/forking.rb:92:in `new'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/strategy.rb:56:in `kill'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/runner.rb:147:in `killer'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/runner.rb:132:in `kill'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/runner.rb:92:in `block in run_subject'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/subject.rb:43:in `block in each'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/mutator.rb:181:in `call'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/mutator.rb:181:in `emit!'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/mutator.rb:137:in `emit'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/mutator/node.rb:109:in `block in emit_attribute_mutations'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/mutator.rb:181:in `call'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/mutator.rb:181:in `emit!'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/mutator.rb:137:in `emit'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/mutator/node.rb:109:in `block in emit_attribute_mutations'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/mutator.rb:181:in `call'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/mutator.rb:181:in `emit!'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/mutator.rb:137:in `emit'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/mutator/node.rb:109:in `block in emit_attribute_mutations'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/mutator.rb:181:in `call'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/mutator.rb:181:in `emit!'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/mutator.rb:137:in `emit'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/mutator/util/array.rb:65:in `dispatch'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/mutator.rb:70:in `initialize'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/abstract_type-0.0.2/lib/abstract_type.rb:37:in `new'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/abstract_type-0.0.2/lib/abstract_type.rb:37:in `new'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/adamantium-0.0.5/lib/adamantium/class_methods.rb:15:in `new'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/mutator.rb:17:in `each'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/mutator/node.rb:103:in `emit_attribute_mutations'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/mutator/node/formal_arguments_19.rb:33:in `emit_required_mutations'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/mutator/node/formal_arguments_19.rb:22:in `dispatch'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/mutator.rb:70:in `initialize'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/abstract_type-0.0.2/lib/abstract_type.rb:37:in `new'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/abstract_type-0.0.2/lib/abstract_type.rb:37:in `new'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/adamantium-0.0.5/lib/adamantium/class_methods.rb:15:in `new'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/mutator.rb:17:in `each'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/mutator/node.rb:103:in `emit_attribute_mutations'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/mutator/node/define.rb:19:in `dispatch'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/mutator.rb:70:in `initialize'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/abstract_type-0.0.2/lib/abstract_type.rb:37:in `new'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/abstract_type-0.0.2/lib/abstract_type.rb:37:in `new'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/adamantium-0.0.5/lib/adamantium/class_methods.rb:15:in `new'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/mutator.rb:17:in `each'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/mutator/node.rb:103:in `emit_attribute_mutations'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/mutator/node/define.rb:36:in `dispatch'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/mutator.rb:70:in `initialize'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/abstract_type-0.0.2/lib/abstract_type.rb:37:in `new'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/abstract_type-0.0.2/lib/abstract_type.rb:37:in `new'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/adamantium-0.0.5/lib/adamantium/class_methods.rb:15:in `new'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/mutator.rb:17:in `each'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/subject.rb:42:in `each'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/runner.rb:89:in `run_subject'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/runner.rb:75:in `block in run'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/matcher/scope_methods.rb:96:in `block in emit_matches'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/matcher/method.rb:39:in `block in each'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/matcher/method.rb:38:in `tap'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/matcher/method.rb:38:in `each'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/matcher/scope_methods.rb:95:in `emit_matches'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/matcher/scope_methods.rb:29:in `block in each'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/matcher/scope_methods.rb:28:in `each'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/matcher/scope_methods.rb:28:in `each'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/matcher/object_space.rb:81:in `block in emit_scope_matches'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/matcher/object_space.rb:80:in `each'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/matcher/object_space.rb:80:in `emit_scope_matches'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/matcher/object_space.rb:42:in `block in each'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/matcher/object_space.rb:109:in `emit_scope'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/matcher/object_space.rb:95:in `block in scopes'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/matcher/object_space.rb:94:in `each_object'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/matcher/object_space.rb:94:in `scopes'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/matcher/object_space.rb:41:in `each'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/matcher/object_space.rb:41:in `each'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/runner.rb:73:in `run'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/runner.rb:52:in `initialize'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/adamantium-0.0.5/lib/adamantium/class_methods.rb:15:in `new'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/adamantium-0.0.5/lib/adamantium/class_methods.rb:15:in `new'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/support/method_object.rb:28:in `run'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/lib/mutant/cli.rb:22:in `run'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/gems/mutant-0.2.16/bin/mutant:14:in `<top (required)>'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/bin/mutant:19:in `load'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/bin/mutant:19:in `<main>'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/bin/ruby_noexec_wrapper:14:in `eval'
    from /home/fletch/.rvm/gems/ruby-1.9.3-p362/bin/ruby_noexec_wrapper:14:in `<main>'

[mbj]

@mockdeep Can you verify your symbolic link works correctly? ls spec/unit. I could not reproduce this issue.

EDIT: Fix nick.

[mockdeep]

Good call. Turns out linking directories is a bit funny. Doing ln -s spec/models spec/unit did some weird stuff. What worked for me is first cding into the spec directory and doing ln -s models unit.

Now I've been running variations on the line:

mutant -r ./config/environment.rb -r ./app/models/user.rb --rspec-unit ::User

But I keep getting errors. It looks like it's handing the --rspec-unit option off to rspec:

invalid option: --rspec-unit

Please use --help for a listing of valid options

The mutations all run and register as all still being alive. Same result with --rspec-full. I tried it with the --rspec-dm2 option and it seemed to be working, but of course not finding the expected specs, so again all mutations survived. The dm2 style looks pretty interesting, so I may start porting over some of my projects to that, but happy to help debug getting it to work on typical Rails/RSpec apps.

[mockdeep]

BTW, thought I linked it before, but here's the rails project I'm toying around with if you want something to look at: https://github.com/mockdeep/track

[mbj]

@mockdeep You are using the wrong mutant. Mine does not offer --help or advertise to do so ;). This is a flaw I know.

Most likely you have another mutant (the old one by txus witch is not related to mine) installed. And mine might only come via bundler (in your env). Try prefixing with bundle exec.

[dkubb]

@mockdeep maybe check to see what version of mutant you have installed, eg:

$ gem list mutant --local

*** LOCAL GEMS ***

mutant (0.2.16)
mutant-melbourne (2.0.3)

[mockdeep]

Yep, that's what I've got. The error, I think is coming from rspec. You get the same error if you run rspec --rspec-unit. Here's a bit more of the output, from a couple of mutations:

Mutant configuration:
Matcher:   #<Mutant::Matcher::ObjectSpace scope_name_pattern=/\AUser(\z|::)/>
Filter:    Mutant::Mutation::Filter::ALL
Strategy:  #<Mutant::Strategy::Rspec::Full>
Subject: User.authenticate:/home/fletch/Dropbox/projects/track/app/models/user.rb:11
invalid option: --rspec-full

Please use --help for a listing of valid options
Coverage report generated for RSpec to /home/fletch/Dropbox/projects/track/coverage. 116 / 120 LOC (96.67%) covered.
Alive: rspec:noop:User.authenticate:/home/fletch/Dropbox/projects/track/app/models/user.rb:11:36d69 (24.62s)
invalid option: --rspec-full

Please use --help for a listing of valid options
Coverage report generated for RSpec to /home/fletch/Dropbox/projects/track/coverage. 88 / 120 LOC (73.33%) covered.
Alive: rspec:User.authenticate:/home/fletch/Dropbox/projects/track/app/models/user.rb:11:dedac (11.15s)
@@ -1,4 +1,4 @@
 def self.authenticate(email, password)
-  find_by_email(email).try(:authenticate, password)
+  find_by_email(email)
 end
invalid option: --rspec-full

Please use --help for a listing of valid options
Coverage report generated for RSpec to /home/fletch/Dropbox/projects/track/coverage. 88 / 120 LOC (73.33%) covered.

[mbj]

@mockdeep You have rcov installed? And require rspec/autorun in your spec helper? Remove/disable this for a while? We had the same error with devtools.

[mbj]

rspec/autorun adds an at_exit helper and leaks the mutant options to the rspec cli parser.

[dkubb]

@mbj I don't like fiddling with $LOAD_PATH, but what if we unshifted something into the front of it that made it so require 'rspec/autorun' requires a file that either warns, raises or is a noop?

[mockdeep]

Yeah, turns out it was require 'rspec/autorun. Now running great. It doesn't have a problem with Simplecov, though I suppose that isn't very useful when using mutant, too. I also have Spork plugged in, and no problem with that in the spec_helper, though it doesn't work if the spork server is running.

[dkubb]

@mockdeep fwiw the dm2-style organization is really nice. Finding the spec that relates to a method is much easier than digging through a few hundred lines of specs for a class. It's faster for mutation testing since you can isolate the spec to the method and run only the one file for each mutation; I used it with heckle before mutant was ready. The isolation also helps to make sure the specs for that method completely cover the method and that you don't have "incidental coverage" from other methods in your class' specs.

[mbj]

@dkubb I hate to fix other libraries obtrusive behavior with more obstrusive behavior ;). Also we can never be sure the $LOAD_PATH isnt mutated later again. Maybe we should add a big fat and ugly warning. Or an FAQ entry. Maybe we can also detect installed at_exit hooks and raise error!

[mockdeep]

@dkubb yeah, I guess I always took the view that incidental coverage was coverage, but like you said, it's faster, and not only that, it's more intentional about what you're testing. I'll play around with it some. Do you think there's a way to make it work with the Rails/RSpec typical folder structure, though? I'd love to keep models and controllers separate. Something like spec/models/user/class_methods/authenticate_spec.rb and spec/controllers/users_controller/instance_methods/create_spec.rb.

[mbj]

@dkubb FWIW, I'm planning an big improvement of the rspec strategy. I'll not select the specs to kill a mutation via files anymore. Only via the description should be used for test selection! This should bring lots of improvements to the more common "spec per class" style. (When the description includes the method name). Also we could preload all stuff in spec/unit/*_/__spec.rb. 80 - 85% of a mutation kill time is the rspec setup/startup.

[mbj]

@mockdeep I think I'll come up with a rails specific subclass of Mutant::Strategy::Rspec that would infer the spec folder to resolve specs within. That code is ultra simple. Maybe you can come up with a PR?

[mockdeep]

Looking at it. Is it basically a copy of example_lookup with the glob_expression altered?

[mbj]

I'd not copy, I'd compose ;) I'd create some code that detects the type {controller,method,foo} and inject that type into an added parameter of the example lookup. (It currently hardcodes spec/unit).

Now the sad part:

I did lots of improvements in the development branch. So I'd like to ask you to base your work on it. BUT I did not had much time for mutant lately, so the development branch does not build. So you cannot easily test your changes.

So please base your work on the master branch, I'll take care the feature is also integrated with the changes I did on development.

Sadly I left the route of incremental changes, and the refactoring I started resulted in the red development branch. A project management anipattern.

[mockdeep]

Sounds good. I'll see what I can do. I'm not terribly familiar with gem development, but I'm happy to help if I can.

[mbj]

Where do you expect a difference between library and app development? IMHO it is totally the same.

[mockdeep]

Well, I've done a lot of Rails development, but I'm not too familiar with best practices when it comes to subclassing, modules, folder structure, testing, documentation and so on when it comes to gems. I can trace through the code just fine, but figuring out how to add to it feels a little more tricky. I guess it's like being able to understand a language spoken to you, but not being quite able to speak it. No better time to learn, though, I suppose.

[mbj]

It makes me sad I find many people seeing to be stuck in rails practices. This is the major problem I see with rails. Many people just press their domain in AR models, without having a chance to get out and see "the light". As the whole stack expects AR infected stuff.

Feel free to push any PR, our community is really open and I bet you only get positive critic.

[mbj]

If you are interested in our styleguide take a look at dkubb/styleguide. Also I'd say veritas is our reference project when it comes to style ;).

BTW mutant for itself does not have perfect spec coverage, mutant on mutant is becoming more stable with each release of to_source, so mutant will soon also have 100% mutation coverge. This is just a disclaimer on why you should NOT use this project as reference. Who mutation specs the mutation tester is the reason this one is not perfect ;)

[dkubb]

@mockdeep feel free to ask about any of that in #datamapper on IRC via freenode.net. I think we've all gone through a ton of experimentation and iterations on those things that we can give you a pretty good rundown of the pros and cons of different approaches. It's not just about DataMapper ;) .. Plus this is especially relevant because mutant is one of our main tools and your questions are related.

[mockdeep]

Thanks guys. I'll give it a try. Probably won't get to it until tomorrow or Friday, though.

[mockdeep]

Jumping back into this. I've got some hairy code I want to test thoroughly before refactoring. Trying to run this on a rails project again. I linked spec/controllers to spec/unit and ran the following command:

mutant -I lib -r ./config/environment.rb -r ./app/controllers/field_sets_controller.rb --rspec-dm2 ::FieldSetsController

It appears to find the proper spec files, but it doesn't seem to run them. I printed out the spec files being run for each mutation, and they are as expected. But it shows every mutation as surviving, even though I've tested a couple manually and the specs fail when I make the change. Here's some of the output:

Mutant configuration:
Matcher:   #<Mutant::Matcher::ObjectSpace scope_name_pattern=/\AFieldSetsController(\z|::)/>
Filter:    Mutant::Mutation::Filter::ALL
Strategy:  #<Mutant::Strategy::Rspec::DM2>
Subject: FieldSetsController#create:/home/fletch/Dropbox/projects/synchroform/app/controllers/field_sets_controller.rb:13
["spec/unit/field_sets_controller/create_spec.rb"]
Alive: rspec:noop:FieldSetsController#create:/home/fletch/Dropbox/projects/synchroform/app/controllers/field_sets_controller.rb:13:9c13f (0.83s)
["spec/unit/field_sets_controller/create_spec.rb"]
Alive: rspec:FieldSetsController#create:/home/fletch/Dropbox/projects/synchroform/app/controllers/field_sets_controller.rb:13:c2f57 (0.87s)
@@ -1,5 +1,5 @@
 def create
-  if params[:matrix]
+  if params
     create_from_matrix
   else
     create_from_csv
["spec/unit/field_sets_controller/create_spec.rb"]

Any ideas?

[mockdeep]

Interesting, tracking down through the code, rspec is running with the --drb option. I tried some different things and got a different result, though not necessarily any better. If I comment it out in the rspec code, remove it from my .rspec file, or just boot up spork, I get the same result, a bunch of noop fails:

subjects:   7
mutations:  0
noop_fails: 7
kills:      0
alive:      0
mtime:      4.17s
rtime:      4.77s

[dkubb]

@mockdeep Interesting. Does your spec helper require "rspec/autorun" or something similar? I believe that causes issues with mutant and is often included in examples.

[mockdeep]

Yeah, I commented that out.

[mbj]

@mockdeep Can you run mutant with the debug flag? -d and post the output?

[mockdeep]

Interesting. I had guard running on another app and when I cd'ed into this app it tried to find the spec I was specifying in the other project. i.e: I was in the directory /home/fletch/Dropbox/projects/track, ran the command from above and got the error:

Exception encountered: #<LoadError: cannot load such file -- /home/fletch/Dropbox/projects/quickies/spec/unit/items_controller/update_spec.rb>

Anyway, so what's probably more relevant, is that when I shut down the guard server and try it again, I get this error:

Subject: ItemsController#update:/home/fletch/Dropbox/projects/track/app/controllers/items_controller.rb:23
{:color=>true, :formatters=>[["documentation"]], :drb=>true, :fail_fast=>true, :files_or_directories_to_run=>["spec/unit/items_controller/update_spec.rb"]}
No DRb server is running. Running in local process instead ...
Run options: include {:focus=>true}

All examples were filtered out; ignoring {:focus=>true}

ItemsController
  #update

An error occurred in an after hook
  PG::Error: result has been cleared
  occurred at /home/fletch/.rvm/gems/ruby-1.9.3-p385/gems/activerecord-3.2.13/lib/active_record/connection_adapters/postgresql_adapter.rb:1147:in `error_field'

    updates the item (FAILED - 1)

Failures:

  1) ItemsController#update updates the item
     Failure/Error: Unable to find matching line from backtrace
     PG::Error:
       result has been cleared
     # /home/fletch/.rvm/gems/ruby-1.9.3-p385/gems/activerecord-3.2.13/lib/active_record/connection_adapters/postgresql_adapter.rb:1147:in `error_field'
     # /home/fletch/.rvm/gems/ruby-1.9.3-p385/gems/activerecord-3.2.13/lib/active_record/connection_adapters/postgresql_adapter.rb:1147:in `translate_exception'
     # /home/fletch/.rvm/gems/ruby-1.9.3-p385/gems/activerecord-3.2.13/lib/active_record/connection_adapters/abstract_adapter.rb:284:in `rescue in log'
     # /home/fletch/.rvm/gems/ruby-1.9.3-p385/gems/activerecord-3.2.13/lib/active_record/connection_adapters/abstract_adapter.rb:275:in `log'
     # /home/fletch/.rvm/gems/ruby-1.9.3-p385/gems/activerecord-3.2.13/lib/active_record/connection_adapters/postgresql_adapter.rb:649:in `execute'
     # /home/fletch/.rvm/gems/ruby-1.9.3-p385/gems/activerecord-3.2.13/lib/active_record/connection_adapters/postgresql_adapter.rb:699:in `begin_db_transaction'
     # /home/fletch/.rvm/gems/ruby-1.9.3-p385/gems/activerecord-3.2.13/lib/active_record/fixtures.rb:856:in `block in setup_fixtures'
     # /home/fletch/.rvm/gems/ruby-1.9.3-p385/gems/activerecord-3.2.13/lib/active_record/fixtures.rb:853:in `each'
     # /home/fletch/.rvm/gems/ruby-1.9.3-p385/gems/activerecord-3.2.13/lib/active_record/fixtures.rb:853:in `setup_fixtures'
     # /home/fletch/.rvm/gems/ruby-1.9.3-p385/gems/rspec-rails-2.13.0/lib/rspec/rails/adapters.rb:50:in `block (2 levels) in setup'

Let me know if you want more of the stack trace.

[mockdeep]

Those are with the -d option.

[mockdeep]

Still poking around with this. I set it up to use database_cleaner instead and I'm getting a little further. No more errors. But the problem I'm seeing now is:

{:color=>true, :formatters=>[["documentation"]], :drb=>true, :fail_fast=>true, :files_or_directories_to_run=>["spec/unit/items_controller/index_spec.rb"]}
No DRb server is running. Running in local process instead ...
Run options: include {:focus=>true}

All examples were filtered out; ignoring {:focus=>true}

Finished in 0.00487 seconds
0 examples, 0 failures

Randomized with seed 2271

0 examples. The spec files are there. I added a puts that displays the filename...