advancedtelematic / quickcheck-state-machine

Test monadic programs using state machine based models
Other
204 stars 25 forks source link

Labelling improvements #339

Closed kderme closed 4 years ago

kderme commented 4 years ago

This pr adds an addition, simple option for labelling sequential tests and also makes all option available for the parallel case (and also the n-parallel case). I replaced the existing tags with the new option, for the test which run on ci. Opinions on this? I think it reduces the verbosity. We now have this:

    prop_circularBuffer:                                           OK (0.06s)
      +++ OK, passed 100 tests.

      Commands (333 in total):
      28.5% New
      25.2% Len
      24.9% Put
      21.3% Get

instead of


    prop_circularBuffer:                                           OK (0.07s)
      +++ OK, passed 100 tests:
      74% coverage

       7% []
       3% [("New",1)]
       2% [("Get",1),("Len",4),("New",1),("Put",1)]
       2% [("New",1),("Put",1)]
       1% [("Get",1),("Len",12),("New",2),("Put",2)]
       1% [("Get",1),("Len",12),("New",2),("Put",7)]
       1% [("Get",1),("Len",2),("New",1),("Put",1)]
       1% [("Get",1),("Len",2),("New",1),("Put",2)]
       1% [("Get",1),("Len",2),("New",2),("Put",4)]
       1% [("Get",1),("Len",4),("New",2),("Put",4)]
       1% [("Get",1),("Len",4),("New",2),("Put",5)]
       1% [("Get",1),("Len",5),("New",5),("Put",6)]
       1% [("Get",1),("Len",6),("New",1),("Put",6)]
       1% [("Get",1),("Len",6),("New",3),("Put",6)]
       1% [("Get",1),("Len",6),("New",4),("Put",8)]
       1% [("Get",1),("New",1),("Put",1)]
       1% [("Get",1),("New",2),("Put",1)]
       1% [("Get",1),("New",2),("Put",2)]
       1% [("Get",1),("New",2),("Put",3)]
       1% [("Get",10),("Len",19),("New",5),("Put",15)]
       1% [("Get",10),("Len",28),("New",8),("Put",25)]
       1% [("Get",11),("Len",11),("New",4),("Put",12)]
       1% [("Get",11),("Len",13),("New",7),("Put",21)]
       1% [("Get",11),("Len",30),("New",6),("Put",30)]
       1% [("Get",12),("Len",17),("New",6),("Put",18)]
       1% [("Get",13),("Len",12),("New",2),("Put",16)]
       1% [("Get",13),("Len",19),("New",4),("Put",22)]
       1% [("Get",13),("Len",33),("New",13),("Put",29)]
       1% [("Get",14),("Len",15),("New",3),("Put",24)]
       1% [("Get",14),("Len",18),("New",5),("Put",27)]
       1% [("Get",14),("Len",25),("New",10),("Put",21)]
       1% [("Get",15),("Len",21),("New",9),("Put",31)]
       1% [("Get",16),("Len",19),("New",5),("Put",20)]
       1% [("Get",17),("Len",27),("New",3),("Put",32)]
       1% [("Get",2),("Len",1),("New",1),("Put",3)]
       1% [("Get",2),("Len",1),("New",2),("Put",3)]
       1% [("Get",2),("Len",10),("New",2),("Put",6)]
       1% [("Get",2),("Len",3),("New",1),("Put",6)]
       1% [("Get",2),("Len",3),("New",3),("Put",5)]
       1% [("Get",2),("Len",4),("New",2),("Put",3)]
       1% [("Get",2),("Len",4),("New",2),("Put",4)]
       1% [("Get",2),("Len",5),("New",1),("Put",7)]
       1% [("Get",2),("Len",6),("New",1),("Put",4)]
       1% [("Get",2),("Len",9),("New",2),("Put",7)]
       1% [("Get",3),("Len",13),("New",5),("Put",4)]
       1% [("Get",3),("Len",17),("New",3),("Put",6)]
       1% [("Get",3),("Len",3),("New",1),("Put",4)]
       1% [("Get",3),("Len",5),("New",2),("Put",6)]
       1% [("Get",3),("Len",7),("New",3),("Put",12)]
       1% [("Get",3),("Len",9),("New",3),("Put",12)]
       1% [("Get",4),("Len",12),("New",3),("Put",7)]
       1% [("Get",4),("Len",13),("New",3),("Put",9)]
       1% [("Get",4),("Len",13),("New",6),("Put",11)]
       1% [("Get",4),("Len",18),("New",5),("Put",7)]
       1% [("Get",4),("Len",2),("New",1),("Put",4)]
       1% [("Get",4),("Len",4),("New",2),("Put",6)]
       1% [("Get",4),("Len",5),("New",2),("Put",7)]
       1% [("Get",4),("Len",6),("New",4),("Put",7)]
       1% [("Get",4),("Len",7),("New",1),("Put",5)]
       1% [("Get",4),("Len",7),("New",3),("Put",6)]
       1% [("Get",5),("Len",10),("New",3),("Put",6)]
       1% [("Get",5),("Len",12),("New",7),("Put",8)]
       1% [("Get",5),("Len",19),("New",5),("Put",14)]
       1% [("Get",5),("Len",7),("New",5),("Put",9)]
       1% [("Get",6),("Len",14),("New",4),("Put",7)]
       1% [("Get",6),("Len",17),("New",9),("Put",14)]
       1% [("Get",6),("Len",4),("New",4),("Put",14)]
       1% [("Get",6),("Len",6),("New",4),("Put",7)]
       1% [("Get",6),("Len",7),("New",7),("Put",13)]
       1% [("Get",7),("Len",10),("New",1),("Put",8)]
       1% [("Get",7),("Len",10),("New",4),("Put",14)]
       1% [("Get",7),("Len",19),("New",5),("Put",9)]
       1% [("Get",7),("Len",7),("New",1),("Put",7)]
       1% [("Get",8),("Len",13),("New",2),("Put",11)]
       1% [("Get",8),("Len",13),("New",3),("Put",10)]
       1% [("Get",8),("Len",15),("New",7),("Put",14)]
       1% [("Get",8),("Len",5),("New",1),("Put",10)]
       1% [("Get",9),("Len",13),("New",3),("Put",10)]
       1% [("Get",9),("Len",22),("New",5),("Put",14)]
       1% [("Get",9),("Len",7),("New",3),("Put",12)]
       1% [("Len",1),("New",1),("Put",1)]
       1% [("Len",15),("New",6),("Put",9)]
       1% [("Len",2),("New",1),("Put",1)]
       1% [("Len",2),("New",2),("Put",3)]
       1% [("Len",3),("New",1),("Put",1)]
       1% [("Len",3),("New",3)]
       1% [("Len",4),("New",1),("Put",2)]
       1% [("Len",4),("New",3)]
       1% [("Len",6),("New",2),("Put",4)]
       1% [("New",1),("Put",2)]
kderme commented 4 years ago

Nice!

I think this improves the output and perhaps we should remove the old checkCommandNames completely? I think I introduced it before QuickCheck had tabulate.

Not sure how important the coverage checking that checkCommandNames does is? (I got the idea from Quviq's Erlang eqc API).

Maybe we can leave checkCommandNames for backwards compatibility, but remove the cover part. I also didn't include the cover part in countCommandNames, because the output is very small and the user can easily check coverage.

stevana commented 4 years ago

Maybe we can leave checkCommandNames for backwards compatibility

Sorry, I meant to remove checkCommandNames and rename your countCommandNames to checkCommandNames and hence keep backwards compatibility.

I also didn't include the cover part in countCommandNames, because the output is very small and the user can easily coverage.

It's small until you have 25+ commands, then checking if all commands are there gets annoying :-)

Perhaps coverage should be a separate combinator though, like QuickCheck's cover?

stevana commented 4 years ago

(Travis found a type error.)

kderme commented 4 years ago

(Travis found a type error.)

Oops yes, this was a last moment "improvement" which didn't work.