c1505
commented
4 years ago Adding some more context to this issue that we should consider what tests can be run in an environment together without having to clean that environment or create a new one as we discussed for issue 168.
Test cases need to be able to be grouped together by function. A simple way of doing this is simply by directory, i.e. a directory is considered a "bucket" and any test cases inside that directory are test cases in that bucket. The directories should be nestable, so for example, a high level directory might be called "pipelines" with a child directory that indicates a specific sub-function of pipelines, and then individual test cases can be placed into that child directory to test that function. The test aggregator should be able to determine the buckets by scanning the directory structure.
NOTE: The above system for grouping test cases into buckets is the simplest one I could think of and should be considered just one possible way to do this. If there are better ways of creating and grouping tests into buckets, such as with metadata, we should be open to that.