[test_runner] executes runner once per file in files

[blairconrad]

For each file specified in files, the exe command is run on that file, with whatever parameters are given. This is handy for test runners that can only deal with one file at a time, but can cause problems with others.

For example if multiple files are passed to a runner that's supposed to execute an xUnit command line runner with the --xml (or other reporting) option, the test runs will overwrite the same output file, so the reports from some of the test assemblies will be lost.

I discovered this while using the very old-style Albacore XUnitTestRunner, but inspection of the new TestRunner seems to reveal the same behaviour.

I think that changing the behaviour would be a breaking change, so perhaps the best way forward (if there's interest at all) would be to add an additional flag to indicate that the files should be passed to exe en masse rather than sequentially.

[haf]

+1 sure, PR + tests that.

[blairconrad]

Sorry for the delay. I think that meant, "Sounds okay to me. Send a PR that includes tests." I'm working on it (slowly, between other things). rspec is completely new to me. Also ruby, but I'm a talented mimic, so should be able to come up with something. It may not be idiomatic, but I'm happy to learn. In the meantime, if you feel like assigning the work to someone else, I will not be offended.

[haf]

That's ok, I can wait. Just ask if you have any questions.

[blairconrad]

Hi, @haf. I thought I was getting somewhere, but during spec-writing/code-implementing, I noticed that two other switches will interfere with the new one:

• is_ms_test because it only supports one test file, and
• copy_local because it copies recursively from the directory the test file's directory

For is_ms_test, I think it's reasonable to raise an error if both it and the new switch (call it batch_execution for convenience) are specified. For copy_local, I'd had the same thought. But I suppose an alternative would be to merge all the directories. Although that seems complicated.

How do you feel about forbidding either of those switches to be used in conjunction with batch_execution?

[haf]

I think that sounds like a reasonable trade-off as long as you also write that in the wiki. :)

[blairconrad]

Oh, sure. I'm all about the docs.

[blairconrad]

Funny story. I'm not the first to complain about this issue. In fact, I've read a comment about it, and then forgot about it for 16 months or so.

https://github.com/FakeItEasy/FakeItEasy/issues/407#issuecomment-65199802