Recursive traversal of testcase directory

[jgough]

In the beta 1 release in daemon mode, If I have my tests in the following format:

/usr/share/logstash/tests/suite1/test.yml
/usr/share/logstash/tests/suite2/test.yml
/usr/share/logstash/tests/suite3/test.yml

Then I cannot seem to see any way of running all these tests at once with one command.

The following options run no tests at all:

--testcase-dir /usr/share/logstash/tests
--testcase-dir /usr/share/logstash/tests/

The following run only the tests in the suite1 directory

--testcase-dir /usr/share/logstash/tests/*
--testcase-dir /usr/share/logstash/tests/*/*

This seems a bug that it runs tests in only one of the subdirectories.

[breml]

The logic to read test case directories and files did not change since LFV v1. Until now, it was never posible to run test cases from subdirectories. Therefore, I consider this not a bug bug a feature request.

[jgough]

I see. I could argue that the behaviour of only running tests in one subdirectory is confusing, but equally one could argue that it is undocumented and unsupported so the behaviour is not defined. Happy for this to be a feature request if need be.

[breml]

Why have you decided to put each test suite into its own directory? Do you have multiple files containing testcases in each suite? If not, you could have all the test suite files in the same directory. With this it would be possible to either run all test suites or only one be providing the exact filename.

My main focus at the moment is to stabilize the current state of the daemon mode (V2), so this feature request is not of highest priority for me. But maybe @magnusbaeck can have a look at this.

[jgough]

@breml Thanks for your comments. We've organised this way as a hangover from v1 where we basically did unit testing of each pipeline individually, one pipeline at a time. We do have loads of pipelines and having subdirectories just made it easier to organise and reason with - especially when some do not feature pipeline-to-pipeline communication. With moving to v2 we'll probably be doing a lot less unit testing but still being able to organise this way is handy when working with so many pipelines.

At the end of the day we can always still do this and run once per directory if we wish to keep them separate.

I think this issue is just that the behaviour when selecting a directory is a bit weird, but understand that that is separate from the great work you've been doing on v2!