Add option to reveal unused expectations files

[xymaxim]

This adds the --pm-reveal-unused-files option to run tests in the collect-only mode and print out the unused expectations files. I find this functionality quite useful to keep a base directory clean after renaming or removing tests. Does it deserve a new plugin option?

The standard TerminalReporter is replaced with a custom reporter to print only revealed unused files separated with newlines—without any headers, summary, etc. The output can be piped to commands like mv or rm. Some plugins write reports to files while keeping the default output, but I think in our case it’ll be redundant.

Only files with .out and .err extensions are checked. This allows to ignore files like README, backup files, etc.

A small question: are the words ‘patterns’ and ‘expectations’ interchangeable? Or ‘patterns’ is a narrow term and refers to expectations containing regexp patterns? Should I rename the option to --pm-reveal-unused-expectations?

[zaufi]

@xymaxim Thank you very much for the great improvement!

I've rebased your branch against the latest master, where I've added GH workflows to run linters and tests. Please look at the type-checking issues found.

Please note you need the following:

$ git fetch
$ git reset --hard FETCH_HEAD

[zaufi]

Or ‘patterns’ is a narrow term and refers to expectations containing regexp patterns?

Strictly speaking, it's correct. However, I'm not sure how these terms are really mentioned in code/docs nowadays ;) So, I think nowadays they are quite interchangeable ;)

Should I rename the option to --pm-reveal-unused-expectations?

As for me, the current option name is also fine... Up to you...

[zaufi]

Adding GH actions I was thinking that it'll be nice to have a CI check for the orphaned files... That means the process should exit w/a non-zero exit code. However, it'll ruin your original scenarios like pytest --pm-reveal-unused-files | xargs git rm. So, probably it should be another option for use in GH actions (or pre-commit) --pm-check-unused-files that will exit w/ a non-zero exit code to fail a build if any files have been found.

Thoughts?

[xymaxim]

@xymaxim Thank you very much for the great improvement!

I've rebased your branch against the latest master, where I've added GH workflows to run linters and tests. Please look at the type-checking issues found.

Please note you need the following:

$ git fetch
$ git reset --hard FETCH_HEAD

Thanks, done. Fewer worries with linters and tests now.

[xymaxim]

Please look at the type-checking issues found.

Well, yes, the Item objects are actually Function class instances.

I’ve silenced them, as in the pytest codebase (see, e.g. this file).

[xymaxim]

Adding GH actions I was thinking that it'll be nice to have a CI check for the orphaned files... That means the process should exit w/a non-zero exit code. However, it'll ruin your original scenarios like pytest --pm-reveal-unused-files | xargs git rm. So, probably it should be another option for use in GH actions (or pre-commit) --pm-check-unused-files that will exit w/ a non-zero exit code to fail a build if any files have been found.

Thoughts?

Actually, pytest doesn’t fail a pipe—it’s possible to pipe failed tests. So, it wouldn't be a problem to return a non-zero code.

However, is it conventional to treat a successful output as an error? I think that it’s all about naming. For example, the mentioned --pm-check-unused-files option implies 0 (no unused files found) or 1 (found unused files). The proposed --pm-reveal-unused-files option kind of allows files not to be found.

So, probably it should be another option for use in GH actions (or pre-commit) --pm-check-unused-files that will exit w/ a non-zero exit code to fail a build if any files have been found.

Another option would be to add the environment variable for CI runs: PYTEST_MATCHER_RETURN_CODES=yes pytest tests --pm-reveal-unused-files.

What do you prefer more?

[zaufi]

PYTEST_MATCHER_RETURN_CODES=yes pytest tests --pm-reveal-unused-files

This variant LGTM.

[zaufi]

@xymaxim Thank you very much for the great job you've done!