includeTestsWithTags doesn't respect includeTestsFromTemplates

[TripleG]

OpenTest version 1.3.8

Steps to reproduce:

1. Create a base template base_template.yaml:

   
   sessionLabel: Dummy session
   maxIterations: 2
   tests:
   - name: dummy_test_pass
   path: unit-tests/dummy
   
   - name: dummy_test_pass2
   path: unit-tests/dummy

2. Add some tag to `dummy_test_pass2`:
`tags: [someTag]`

3. Create a new template and inherit the base template `child_template.yaml`. Note the negation of the tag "!someTag":

sessionLabel: Dummy session maxIterations: 2 includeTestsWithTags: (!someTag)

includeTestsFromTemplates:

• name: base_test_template path: .

  
  4. Run the new template => Result is that the `includeTestsWithTags: (!someTag)` is applied to ALL existing tests in repo instead of the ones defined in the parent template, i.e. ALL tests in repo are executed except the one containing the `someTag` tag

[adrianth]

It sounds like the behavior you are describing is the one that should be expected. The logic for computing the list of tests for the parent template goes something like this:

1. Start with the tests listed in the tests property (if any).
2. Compute the list(s) of tests that are matching the expression in the includeTestsWithTags property (if any).
3. Compute the list(s) of tests for each one of the templates listed in includeTestsFromTemplates (if any).

Compute the union of all tests from points 1, 2 and 3 and that will give you the final list of tests to run.

So based on the above logic it results that whatever tests are matched by the includeTestsWithTags: (!someTag) expression, they will surely appear in the final list of tests for this template, regardless of anything else that might be happening in other parts of the template. The only way the other parts of the template could influence the result is by including some of the tests that don't match the expression !someTag. I hope that makes sense.

[TripleG]

Hi @adrianth ,

I am very confused. What I am expecting is that the includeTestsFromTemplates + tests should be the pool of tests from which we will filter the final list according to includeTestsWithTags.

Did I understand you correctly that it is expected for includeTestsWithTags to add tests from whole repo even though my child_template includes only the tests defined in base_template?

How can I achieve the desired behavior then? I want to be able to include tests only from specific parent templates and apply filtering on them only.

Looks like includeTestsFromTemplates is useless when we are using includeTestsWithTags at the same time, as includeTestsWithTags has the ultimate scope(all tests)

[adrianth]

What I am expecting is that the includeTestsFromTemplates + tests should be the pool of tests from which we will filter the final list according to includeTestsWithTags.

That is not accurate. The final list of tests is going to be the union (not intersection - which is what you are describing) of the three lists of tests identified by tests, includeTestsWithTags and includeTestsFromTemplates. Typically, parent templates that use includeTestsFromTemplates don't need to populate anything in their tests and includeTestsWithTags properties and they will rely exclusively on the child templates to bring in the lists of tests. This is actually kind of a best practice which ensures that parent templates only act as "aggregators" of other templates. I can see how this can become confusing.

[TripleG]

I still don't understand - what is the point of parent templates aggregation, if when I use includeTestsWithTags in the child it will not respect the parents scope of tests?

It is just simpler to create templates based on includeTestsWithTags only and never use includeTestsFromTemplates, isn't it?

[adrianth]

It is just simpler to create templates based on includeTestsWithTags only and never use includeTestsFromTemplates, isn't it?

I agree. The tag-based approach works best in the vast majority of scenarios. You could probably be able to only ever use includeTestsWithTags in your templates. The includeTestsFromTemplates comes in handy when you need to combine multiple test suites. For example, you could have something like this:

# Child template 1
sessionLabel: Run feature1 tests
includeTestsWithTags: feature1

# Child template 2
sessionLabel: Run feature2 tests
includeTestsWithTags: feature2

# Parent template
sessionLabel: Test all features
includeTestsFromTemplates:
    - name: feature1
      path: .
    - name: feature2
      path: .

The parent template could also be built based on tags by using includeTestsWithTags: feature1 || feature2. However, using includeTestsFromTemplates is probably the better choice because it delegates the test selection to those child templates, so if the child templates were to change, the parent template would automatically reflect those changes. This is useful when - for example - each child template is maintained by a different team.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.