ExCheck usage in an umbrella app causes end-of-run spec counts to be reported incorrectly.

[myronmarston]

We have an umbrella app and we use ExCheck in the test suite of one of the apps. When we run mix test in our project root (which recursively runs mix test in each app), the end-of-run test counts reported for each app are inflated with an extra 100 tests. For example, we get output like this:

# ...
==> s3
Excluding tags: [:manual, :smoke]

Delorean.S3Test

Delorean.S3AcceptanceTest
  * test both clients calculate `presigned_get_url` in the same way (and the URL works) (195.1ms)
  * test `Delorean.S3.Test.FakeS3` can upload objects to S3 and later list and fetch them (13.2ms)
  * test `Delorean.S3` can upload objects to S3 and later list and fetch them (330.6ms)

Finished in 0.7 seconds
103 tests, 0 failures

Randomized with seed 54640
# ...

3 tests but it is reported as 103. I originally reported this as an elixir issue but eventually realized ExCheck is causing this.

[luc-tielen]

The IO server and formatter.ex are the cause for this most likely. For each test with triq/excheck it adds the amount of generated tests. Maybe we could add flag/config to switch this behaviour on and off. Is the code online so I can test this?

[myronmarston]

Is the code online so I can test this?

Sorry, it's a proprietary code base I can't share.

[luc-tielen]

Or can you recreate a minimal failing example? Would be much appreciated.

[myronmarston]

Or can you recreate a minimal failing example? Would be much appreciated.

Sure:

https://github.com/myronmarston/excheck_issue_33

Clone it, run mix deps.get and run mix test. You'll see this output:

$ mix test
==> bar
......................................................................................................

Finished in 0.07 seconds
102 tests, 0 failures

Randomized with seed 893339
==> foo
.

Finished in 0.03 seconds
101 tests, 0 failures

Randomized with seed 997063

The bar app has a property test with 100 iterations (plus one normal exunit test) and it reports the total as 102 tests (although...shouldn't that be 101 tests? 1 normal test + 100 property test iterations?). The foo has only one ex unit test and does not use excheck at all, but it reports 101 tests.

[luc-tielen]

Thanks, the example helped a lot with debugging. :smile:

[luc-tielen]

Fix got merged in, should work now.