Make test discovery from file fully lazy

[nattofriends]

Takes #326 to its conclusion (there were still some parts that were not lazy). This allows test methods to be lazily read from a test discovery file.

[nattofriends]

The reason I am making this change is not to make Testify itself faster. I would like to reduce the time associated with importing the code under test (~25s?) when repeatedly running groups of tests which has been, up to now, passed in via --rerun-test-file. I plan on accomplishing this by passing a fifo instead of an ordinary file, through which I would write a list of tests. These changes to Testify make it so that the file is only read line by line, making it possible to use a fifo to feed in more tests after one group of tests is finished.

[asottile]

ok that makes sense, but there won't be any code reloading so why use a fifo?

[nattofriends]

Normally testify would exit after each group of tests. I'd like to keep the fifo open and write more tests into the fifo after the first group of tests is done running.

[asottile]

But why? presumably the usual test workflow is:

• run some tests
• change some code
• run some more tests

with your proposal, the changes won't be propagated so it might as well just be

• run some tests
• run some other tests

which might as well just be

• run both sets of tests

[nattofriends]

A human would not use this feature; this is for running all tests in parallel.

[asottile]

curious what this gains over doing something like

split -n5 tests testsplit
ls testsplit* | xargs -n1 -P5 testify ...

fifos seem like an overly-complicated solution to splitting a file and parallelizing test running

[asottile]

Note that testify used to have a mode similar to what you're describing but was scrapped due to the sheer complexity: https://github.com/Yelp/Testify/pull/317

[nattofriends]

Right, so there is enough value to solving this problem that another stab at the problem had even been attempted before. Here, something else is doing all the parallelization, so this change is less complicated from the testify point of view.