Introduce mutation testing

[cljoly]

For instance with https://github.com/sourcefrog/cargo-mutants

Also, display test coverage prominently?

[czocher]

I understand that we'd like to have two things here: a code coverage report (with a badge and integration with GitHub, so that there's a certain coverage requirement for new PRs) plus mutation testing, which should fail on uncaught cases?

I found a nice tutorial for tarpaulin for the first case: https://morioh.com/p/bfe6c22e43bf, the second one should not be a big problem.

First step would probably be fixing the current issues found with cargo-mutants:

Found 40 mutants to test
Unmutated baseline ... ok in 10.8s build + 1.5s test
Auto-set test timeout to 20.0s
rusqlite_migration/src/lib.rs:134: replace <impl PartialEq for Box<dyn MigrationHook>>::eq -> bool with false ... NOT CAUGHT in 0.5s build + 0.8s test
rusqlite_migration/src/errors.rs:44: replace <impl Display for Error>::fmt -> fmt::Result with Ok(Default::default()) ... NOT CAUGHT in 0.5s build + 0.8s test
rusqlite_migration/src/lib.rs:142: replace <impl Debug for Box<dyn MigrationHook>>::fmt -> fmt::Result with Ok(Default::default()) ... NOT CAUGHT in 0.5s build + 0.8s test
rusqlite_migration/src/errors.rs:51: replace <impl Error for Error>::source -> Option<&(dyn std::error::Error + 'static)> with Default::default() ... NOT CAUGHT in 0.6s build + 0.8s test
rusqlite_migration/src/asynch.rs:57: replace AsyncMigrations::current_version -> Result<SchemaVersion> with Ok(Default::default()) ... NOT CAUGHT in 0.5s build + 0.8s test
rusqlite_migration/src/lib.rs:134: replace <impl PartialEq for Box<dyn MigrationHook>>::eq -> bool with true ... NOT CAUGHT in 0.5s build + 0.8s test
rusqlite_migration/src/errors.rs:114: replace <impl Display for MigrationDefinitionError>::fmt -> fmt::Result with Ok(Default::default()) ... NOT CAUGHT in 0.5s build + 0.8s test
rusqlite_migration/src/errors.rs:150: replace <impl Display for ForeignKeyCheckError>::fmt -> fmt::Result with Ok(Default::default()) ... NOT CAUGHT in 0.5s build + 0.8s test
rusqlite_migration/src/asynch.rs:120: replace AsyncMigrations::to_version -> Result<()> with Ok(Default::default()) ... NOT CAUGHT in 0.6s build + 0.8s test
rusqlite_migration/src/lib.rs:329: replace <impl PartialOrd for SchemaVersion>::partial_cmp -> Option<Ordering> with Default::default() ... NOT CAUGHT in 0.6s build + 0.8s test
rusqlite_migration/src/asynch.rs:144: replace AsyncMigrations::validate -> Result<()> with Ok(Default::default()) ... NOT CAUGHT in 0.5s build + 0.8s test
rusqlite_migration/src/lib.rs:319: replace <impl Display for SchemaVersion>::fmt -> fmt::Result with Ok(Default::default()) ... NOT CAUGHT in 0.6s build + 0.8s test
rusqlite_migration/src/errors.rs:86: replace <impl Display for SchemaVersionError>::fmt -> fmt::Result with Ok(Default::default()) ... NOT CAUGHT in 0.6s build + 0.9s test
rusqlite_migration/src/asynch.rs:85: replace AsyncMigrations::to_latest -> Result<()> with Ok(Default::default()) ... NOT CAUGHT in 0.5s build + 0.8s test
40 mutants tested in 0:43: 14 missed, 11 caught, 15 unviable

[cljoly]

You are right, we would like 1. mutation testing and 2. to measure test coverage.

For the later, I had in mind https://lib.rs/crates/cargo-llvm-cov, as it might be a bit faster than tarpaulin. The readme also offers some guidance. But I guess either cargo-llvm-cov or tarpaulin would be fine.

[cljoly]

Thanks for tackling this!

[czocher]

@cljoly can you register to one of the coverage report sites (like the one suggested by cargo-llvm-cov - Codecov) and add a PR with the following new job in the CI pipeline:

  coverage:
    runs-on: ubuntu-latest
    env:
      CARGO_TERM_COLOR: always
    steps:
      - uses: actions/checkout@v3
      - name: Install Rust
        run: rustup update stable
      - name: Install cargo-llvm-cov
        uses: taiki-e/install-action@cargo-llvm-cov
      - name: Generate code coverage
        run: cargo llvm-cov --all-features --workspace --lcov --output-path lcov.info
      - name: Upload coverage to Codecov
        uses: codecov/codecov-action@v3
        with:
          token: ${{ secrets.CODECOV_TOKEN }}
          files: lcov.info
          fail_ci_if_error: true

You will need to create the CODECOV_TOKEN secret, which will be provided upon registering the repository in Codecov.

[cljoly]

Yeah, I’ll do that over the weekend.

[cljoly]

Done, I’m adding the badge and tweaking the settings.

[czocher]

@cljoly can be closed in my opinion.