Unexpected change in test reporting in tasty 1.5

andreasabel commented 9 months ago

THIS IS STILL A DRAFT ISSUE! Investigation is still ongoing, so no need to answer/address this now.

I noticed a change in the test reporting for Agda. I now prints:

all
  Succeed
    Issue6931:                                                    OK (0.36s)
  Succeed
    Issue6931-Three:                                              OK (0.15s)
  Succeed
    Issue6931-Two:                                                OK (0.12s)
...

I.e., it repeats the group Succeed for each single test. It used to print this:

all
  Succeed
    Issue6931:                                                    OK (0.36s)
    Issue6931-Three:                                              OK (0.15s)
    Issue6931-Two:                                                OK (0.12s)
...

This follows the structure of the test tree directly and feels like the natural thing to do.

I pin this observation to the switch from tasty-1.4.3 to tasty-1.5. I have not tried yet to establish hard evidence for this though. However, no change happened on our side or in our tasty-client, tasty-silver.

Looking through the changes between 1.4.3 and 1.5 I noticed a strange commit: https://github.com/UnkindPartition/tasty/commit/8043c0cddc59e6ae389e2dfb52b20d0543b76a28 It is labelled "clean code" but it does change

when nonempty $ do printHeading :: IO (); getTraversal printBody

to

when nonempty $ printHeading
getTraversal printBody

A priori, this is not just a cleaning, but it is a semantic change, invoking getTraversal printBody now even when not nonempty. (I am getting heartbleed vibes here.)

Questions:

Could this change be connected to the behavioral change I am observing?
Was this change intended? It looks unintended to me, like a late-night coding blunder.
Also, the changelog for 1.5 does not mention the change in behavior.
A fundamental question: Don't we have any golden value test for the test reporter? Those would have shown the change in behavior.

andreasabel commented 9 months ago

I could reproduce this now in tasty-silver's own testsuite.

$ cabal get tasty-silver-3.3.1.2
$ cd tasty-silver-3.3.1.2

With tasty-1.5:

$ cabal test --enable-tests -c 'tasty==1.5'
...
tests
  findByExtension: OK (0.00s)
tests
  Filter.checkRF
    empty1a:       OK (0.00s)
  Filter.checkRF
    empty1b:       OK (0.00s)
  Filter.checkRF
    empty2a:       OK (0.00s)
  Filter.checkRF
    empty2b:       OK (0.00s)

With tasty-1.4.3:

$ cabal test --enable-tests -c 'tasty==1.4.3'
...
  findByExtension: OK (0.00s)
  Filter.checkRF
    empty1a:       OK (0.00s)
    empty1b:       OK (0.00s)
    empty2a:       OK (0.00s)
    empty2b:       OK (0.00s)

So it could be that the change manifests itself because tasty-silver sets up test-trees in a peculiar way. Need to investigate this more.

andreasabel commented 9 months ago

This was actually botched update of tasty-silver to tasty-1.5:

https://github.com/phile314/tasty-silver/issues/41

UnkindPartition / tasty

Unexpected change in test reporting in tasty 1.5 #394