Surprises wrt --state option vs. file arguments

[andk]

What --state does when combined with file arguments is underdocumented and sometimes surprising. The following in the Test-Harness distribution directory:

Options + Arguments	what is does	ok/surprise
rm .prove
--state=slow,save	refuses to test	surprise[0]
rm .prove
--state=save	runs all tests	ok
--state=save,fast t/parse.t	runs all tests	surprise[1]
rm .prove
--state=save,fast t/parse.t	runs parse.t	surprise[2]
--state=save,fast t/base.t	runs parse.t and base.t
--state=save,all,fast t/base.t	runs parse.t and base.t	surprise[3]

[0] because there are tests on disk and they would be run with state=save alone; this is one of the cases where it would be helpful to just run all tests from disk.

[1] what happens is that the argument t/parse.t is merged with all files in .prove. Apparently there is no way to run a subset of the known tests with state options without editing the .prove file.

[2] Given that surprise [0] above refused to work, it's now surprising that this incantation does the right thing

[3] keyword "all" apparently does not mean all tests on disk. Not mentioned in the docs.

I believe the currently undocumented rules are:

(1) as long as no statefile exists, any combination of state arguments turn prove into a noop unless there are explicit file or directory arguments on the commandline. E.g.

prove --state=save,all         # noop
prove --state=save,all t/1.t   # runs 1.t

(2) if a statefile exists, the set of tests to be considered to be run is a union of the test files in the state file and files or directories on the command line. From this union set all tests on the command line are run unconditionally while files from the statefile are run according to the rule. E.g.

rm -f .prove
prove --state=save        t/1.t t/2.t # runs 1.t and 2.t
prove --state=save,failed t/1.t       # runs 1.t and whatever failed above

(3) the keyword "all" refers to the files in the statefile, not to the files on disk

I'm unsure whether one would prefer the rules to change or get these rules documented. I find the rules hard to understand and to document and also disputable.

[KES777]

Faced into problem [1] too.

[1] what happens is that the argument t/parse.t is merged with all files in .prove. Apparently there is no way to run a subset of the known tests with state options without editing the .prove file.

To my mind that was bad idea to allow --state multiple times.

There is already ability --state=hot,save

There is not clear how these two will be merged: --state=hot,save --state=slow,failed

[KES777]

also [2]

prove --state=save t/OpenAPI3

prove --state=failed t/OpenAPI3

Here I expect that only failed tests from t/OpenAPI3 will be running

To my mind provided list of tests must be filtered by --state

[KES777]

Also this belongs to --state option:

Tests that have never failed will not be selected. To run all tests with the most recently failed first use $ prove -b --state=hot,all,save

kes@work ~/t $ prove --state=hot,all,save
t/OpenAPI3/pointer.t ................... ok                             
t/OpenAPI3/types.t ..................... ok                             
t/OpenAPI3/keywords.t .................. ok                             
t/OpenAPI3/empty.t ..................... ok                             
===(      32;1  6/?  0/?  0/?  6/6  0/?  0/? )==========================
#   Failed test '404 Not Found'
#   at t/OpenAPI3/exception.t line 29.
#          got: '501'
#     expected: '404'

#   Failed test 'exact match for JSON Pointer "/code"'
#   at t/OpenAPI3/exception.t line 29.
#          got: undef

As you can see failed test are not run first.

Next also has not effect:

$prove --state=hot,slow,save

Here I expect to run most recently failed first, then slowest, then save test result

This occur when ~/.proverc file exists and there is already have --state option

Even if the content of .proverc will be just:

--state=hot,all,save
--jobs=9
-l

and I do:

$ prove
t/OpenAPI3/types.t ..................... ok                             
t/OpenAPI3/keywords.t .................. ok                             
t/OpenAPI3/pointer.t ................... ok                             
===(       4;0  1/?  0/?  0/?  0/?  0/?  0/?  0/? )=====================
#   Failed test '404 Not Found'
#   at t/OpenAPI3/exception.t line 29.
#          got: '501'
#     expected: '404'

as you can see testing do not run hot tests first