search

jakemcc / test-refresh

Refreshes and reruns clojure.tests in your project.
393 stars 28 forks source link

Displaying the difference in the number of tests, asserts, passes and fails, compared to the previous run #56

Closed not-raspberry closed 8 years ago

not-raspberry commented 8 years ago

Currently the summary says (e.g):

Ran 9 tests containing 60 assertions.
0 failures, 0 errors.

That would change the summary to:

Ran 9 tests (+1) containing 60 assertions (+3).
0 (+0) failures, 0 (+0) errors.

"+0" may be omitted.

I can try to hack it in if you say OK to this feature.

jakemcc commented 8 years ago

I think implementing this and having it interact well with some of the other test-refresh features might be challenging. I'm not sure its worth figuring out how everything should interact.

What is the motivation?

not-raspberry commented 8 years ago

User feedback - that it really saw and ran your new tests and that you put them in the right place (see: beginner errors with misused lazy seqs) and saved the file. And it's fancy to have a difference between 2 runs shown.

We only need a map of previous test results for that. What would it conflict with?

jakemcc commented 8 years ago

Its a bit unclear to me how it would interact with the two features below.

  • Runs previously failing tests first, giving you feedback even quicker.
  • Optionally only automatically runs tests in changed namespaces.

In the most basic implementation I think the two features above would cause the +n or -n to vary wildly between test runs. I'm not sure if that would be helpful or if, when a smaller subset of namespaces are tested, seeing a large negative number would be jarring. lein-test-refresh is already sort of a mess with the variety of options it supports, I'm hoping to not have it keep expanding and make the interaction between features more complicated.

I do see a benefit to the output being helpful for catching some portion of user mistakes. I don't even look at the test output unless there is an error so I might be having a hard time judging the usefulness. Going to think about it some and bounce the idea off some other users.

jakemcc commented 8 years ago

I've bounced the idea around a few other lein-test-refresh contributors/users and going to opt to not add this feature at this time. Not sure how to make it work cleanly with the current set of features.

Thanks for the suggestion.