How to exclude tests in templates for specific environment?

[TripleG]

Hi,

Is there a way to exclude tests for an environment - for example I want specific tests to NOT be executed on Staging, only on Dev?

[adrianth]

You can, if you tag your tests appropriately. For example, you can have two tags: "dev" and "staging" and use them in your tests like this:

description: Test 1
tags: [ dev ]
...

description: Test 2
tags: [ staging ]
...

description: Test 3
tags: [ dev, staging ]
...

Then, you can create session templates based on those tags:

description: Run all dev tests (will run "Test 1" and "Test 3")
includeTestsWithTags: dev

description: Run all staging tests (will run "Test 2" and "Test 3")
includeTestsWithTags: staging

You can even do more advanced stuff, like running all the tests that are tagged "dev" but exclude the ones that are also tagged "staging":

description: Run all dev tests but exclude the ones tagged "staging" (will just run "Test 1")
includeTestsWithTags: dev && !staging

[TripleG]

Actually, I don't think this will work, because the tags are hard coded and they don't know on what environment they are run against.

Let's say I have 2 environments in data-env dir:

dev
stg

When I select any of them it doesn't matter for the tags.

We need the environment to be evaluated in the template at runtime. Therefore I have 2 suggestions:

1. Allow script evaluation in includeTestsWithTags - so, we would be able to use for example $session.environment or other complex evaluations
2. Add new key excludeTestsWithTags - so we won't be forced to add tags on ALL tests(what if I have 1k tests?) when we want to exclude some by using the includeTestsWithTags.

What do you think?

[adrianth]

You are right - the method I described doesn't allow you to exclude tests dynamically based on the environment that was selected, but it allows you to create test session templates that will execute the right tests in the right environment. You can create one template for the "dev" environment and use includeTestsWithTags: dev and another similar template for the "stg" environment. Supporting JavaScript code in template files along with an API to allow the script access to the necessary information is not very straightforward and it can also potentially introduce confusion and security concerns.

[TripleG]

Okay, what do you think about adding new key excludeTestsWithTags?

[adrianth]

You can already achieve this with includeTestsWithTags. You can use an arbitrarily complex boolean expression built from your test tags. For example:

includeTestsWithTags: (dev && !staging) && stable

[TripleG]

Yes, it is achievable, but when there is only "include" prop, I must go and update ALL tests, except those to be excluded. What happens when after time I need to exclude by another tag? Again I am forced to go through ALL tests.

My point is that achieving "exclude" through "include" and vice versa is not convenient. Do you agree?

[adrianth]

I'm not sure I understand. Let me describe a possible scenario and let me know if it covers your concern.

Let's say I initially have a test session template that includes all tests tagged with "dev":

includeTestsWithTags: dev

Later on, I decide I want to exclude the tests that are also tagged with "staging" (maybe those require a more powerful test environment):

includeTestsWithTags: dev && !staging

Then, my dev team starts complaining that a lot of the tests are flaky, meaning that they often fail even when there is no problem with the application itself. So I now have to come up with the "flaky" tag that allows me to exclude the flaky tests. Of course, I will also have to go and add the "flaky" tag to the tests my dev team identifies as being flaky, as they are being discovered (of course there's no way around the requirement to properly tag the tests). I can then exclude the flaky tests like this:

includeTestsWithTags: dev && !staging && !flaky

Please let me know the scenario that you have in mind that cannot be achieved by using a proper boolean expression in includeTestsWithTags.

[TripleG]

Sorry for me not explaining well. Your scenario is kind of fine in its context, but you would need to have two templates(1 for dev and 1 for staging). Yes, one template can inherit another, but it is still unnecessary file duplication.

As I thought again about it, I think the real problem I face is related to the fact that we cannot access session data from the template dynamically. This would remove the problem of having multiple templates depending on the tags with want to include/exclude.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.