sourcefrog / cargo-mutants

:zombie: Inject bugs and see if your tests catch them!
https://mutants.rs/
MIT License
588 stars 27 forks source link

Support for proc-macro crates #406

Closed chesedo closed 1 month ago

chesedo commented 1 month ago

I tried running cargo-mutants on a proc-macro crate and was surprised to see it found no mutants. When I turned log level to tracing I saw this in the logs:

DEBUG package{name=despatma-dependency-container}: skipping target "despatma_dependency_container" of kinds ["proc-macro"]

Is there a reason proc-macros are skipped that I might be missing?

sourcefrog commented 1 month ago

I don't think there's any positive reason, it's just not something that we recognize yet.

It looks like this is from https://github.com/chesedo/despatma

Looking at the code, it does seem to have tests that cargo-mutants could run, and code that it would understand. So this might be as simple as just changing the code in cargo.rs that decides which cargo targets will be mutated, where this trace line comes from.

sourcefrog commented 1 month ago

@chesedo it looks like the tests fail at head in despatma:

test tests/fail/missing_types.rs ... mismatch

EXPECTED:
┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈
error[E0405]: cannot find trait `Circle` in this scope
 --> tests/fail/missing_types.rs:6:14
  |
6 | visitor!(dyn Circle, Rectangle, dyn Sphere, Arc, dyn Cube);
  |              ^^^^^^ not found in this scope
  |
help: consider importing this trait
  |
4 + use crate::library::shapes::Circle;
  |

error[E0412]: cannot find type `Rectangle` in this scope
 --> tests/fail/missing_types.rs:6:22
  |
6 | visitor!(dyn Circle, Rectangle, dyn Sphere, Arc, dyn Cube);
  |                      ^^^^^^^^^ not found in this scope
  |
help: consider importing this struct
  |
4 + use crate::library::shapes::Rectangle;
  |

error[E0405]: cannot find trait `Sphere` in this scope
 --> tests/fail/missing_types.rs:6:37
  |
6 | visitor!(dyn Circle, Rectangle, dyn Sphere, Arc, dyn Cube);
  |                                     ^^^^^^ not found in this scope
  |
help: consider importing this trait
  |
4 + use crate::library::shapes::Sphere;
  |

error[E0412]: cannot find type `Arc` in this scope
 --> tests/fail/missing_types.rs:6:45
  |
6 | visitor!(dyn Circle, Rectangle, dyn Sphere, Arc, dyn Cube);
  |                                             ^^^ not found in this scope
  |
help: consider importing one of these items
  |
4 + use crate::library::shapes::Arc;
  |
4 + use std::sync::Arc;
  |

error[E0405]: cannot find trait `Cube` in this scope
 --> tests/fail/missing_types.rs:6:54
  |
6 | visitor!(dyn Circle, Rectangle, dyn Sphere, Arc, dyn Cube);
  |                                                      ^^^^ not found in this scope
  |
help: consider importing this trait
  |
4 + use crate::library::shapes::Cube;
  |
┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈

ACTUAL OUTPUT:
┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈
error[E0405]: cannot find trait `Circle` in this scope
 --> tests/fail/missing_types.rs:6:14
  |
6 | visitor!(dyn Circle, Rectangle, dyn Sphere, Arc, dyn Cube);
  |              ^^^^^^ not found in this scope
  |
help: consider importing this trait
  |
4 + use crate::library::shapes::Circle;
  |

error[E0412]: cannot find type `Rectangle` in this scope
 --> tests/fail/missing_types.rs:6:22
  |
6 | visitor!(dyn Circle, Rectangle, dyn Sphere, Arc, dyn Cube);
  |                      ^^^^^^^^^ not found in this scope
  |
help: consider importing this struct
  |
4 + use crate::library::shapes::Rectangle;
  |

error[E0405]: cannot find trait `Sphere` in this scope
 --> tests/fail/missing_types.rs:6:37
  |
6 | visitor!(dyn Circle, Rectangle, dyn Sphere, Arc, dyn Cube);
  |                                     ^^^^^^ not found in this scope
  |
help: consider importing this trait
  |
4 + use crate::library::shapes::Sphere;
  |

error[E0412]: cannot find type `Arc` in this scope
 --> tests/fail/missing_types.rs:6:45
  |
6 | visitor!(dyn Circle, Rectangle, dyn Sphere, Arc, dyn Cube);
  |                                             ^^^ not found in this scope
  |
help: consider importing one of these structs
  |
4 + use crate::library::shapes::Arc;
  |
4 + use std::sync::Arc;
  |

error[E0405]: cannot find trait `Cube` in this scope
 --> tests/fail/missing_types.rs:6:54
  |
6 | visitor!(dyn Circle, Rectangle, dyn Sphere, Arc, dyn Cube);
  |                                                      ^^^^ not found in this scope
  |
help: consider importing this trait
  |
4 + use crate::library::shapes::Cube;
  |
┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈
note: If the actual output is the correct output you can bless it by rerunning
      your test with the environment variable TRYBUILD=overwrite

test fail ... FAILED
tests/expand/compound_name.rs - ok
tests/expand/helper_tmpl.rs - ok
tests/expand/multiple.rs - ok
tests/expand/no_default.rs - ok
tests/expand/simple.rs - ok
tests/expand/visitor_mut.rs - ok
test expand ... ok

failures:

---- fail stdout ----
thread 'fail' panicked at /Users/mbp/.cargo/registry/src/index.crates.io-6f17d22bba15001f/trybuild-1.0.99/src/run.rs:101:13:
1 of 4 tests failed
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
chesedo commented 1 month ago

@chesedo it looks like the tests fail at head in despatma:

A newer Rust version recently changed some of the error messages. Can't quite remember if I updated the repo yet.

I assume you are trying to test if cargo-mutants can run against a proc-macro crate?

chesedo commented 1 month ago

I don't think there's any positive reason

If there is no solid, known reason then I can make a fork to test locally. And can make a PR if it works

sourcefrog commented 1 month ago

The linked branch here has a change to enable it, it just needs some manual tests, and a test added to the tree!

On Thu, Sep 12, 2024, 15:51 Pieter @.***> wrote:

I don't think there's any positive reason

If there is no solid, known reason then I can make a fork to test locally. And can make a PR if it works

— Reply to this email directly, view it on GitHub https://github.com/sourcefrog/cargo-mutants/issues/406#issuecomment-2347115720, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACUR4YHZ3ABAONINXSGDDLZWHWCTAVCNFSM6AAAAABOCPMJDCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNBXGEYTKNZSGA . You are receiving this because you commented.Message ID: @.***>

chesedo commented 1 month ago

The linked branch here has a change to enable it, it just needs some manual tests, and a test added to the tree!

Okay, I was able to manually test the linked branch and the mutants went from 16 to 49. I also fixed the repo if you also wanted to perform a manual test.

sourcefrog commented 1 month ago

OK yep this seems to work well on your crate:

; c r mutants -d ../despatma/
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.14s
     Running `target/debug/cargo-mutants mutants -d ../despatma/`
Found 72 mutants to test
ok       Unmutated baseline in 7.7s build + 26.7s test
 INFO Auto-set test timeout to 2m 14s
MISSED   despatma-lib/src/key_value.rs:51:31: replace && with || in <impl PartialEq for KeyValue>::eq in 0.4s build + 0.2s test
MISSED   despatma-lib/src/key_value.rs:51:9: replace <impl PartialEq for KeyValue>::eq -> bool with true in 0.3s build + 0.2s test
MISSED   despatma-dependency-container/src/processing/visitor/extract_async.rs:20:33: replace |= with ^= in <impl VisitorMut for ExtractAsync>::visit_dependency_mut in 0.9s build + 8.7s test
MISSED   despatma-dependency-container/src/processing/mod.rs:54:9: replace <impl PartialEq for Lifetime>::eq -> bool with true in 0.8s build + 9.4s test
72 mutants tested in 3m 20s: 4 missed, 55 caught, 13 unviable
chesedo commented 1 month ago

Awesome! :partying_face: Thanks for the update!