Delay Between Tests

[zaicurity]

Description First of all, I'm unsure if this is a bug or intentional. If it is intentional please consider this a suggestion for enhancement.

The Runtime Automation option "Delay Between Tests (In Seconds)" adds a delay between individual commands within test cases and between test cases. Take the following Test Case for example. If we configure a delay of 10 seconds for the runtime the test will wait for 10 seconds after each of the included commands. In this example the test case would take over a minute to run. If we wanted to create a delay of multiple minutes between test cases they would take very long to execute.

The naming of the available option suggests to me that it should only create a delay between test cases and possibly also between execution and cleanup, which is why I think it might be a bug.

To Reproduce Steps to reproduce the behavior:

1. Create an assessment and add test cases that contain multiple commands (e.g. "ART - System Owner/User Discovery")
2. Click on "Campaign Actions" -> "Build Automation Runtime"
3. Select the test case(s)
4. Set a value of 10 seconds for "Delay Between Tests (In Seconds)"
5. Click Build, download the archive, extract and run

Why can this be a problem When running bigger campaigns we might want to have a longer delay between test cases to make it easier to correlate tests with produced alerts. However, if we configure a long delay through the currently available option it also affects the runtime of each test case.

Expected Behavior/Suggestion I think an improved behavior would be to only have a delay between test cases and not between individual commands within test cases. A delay between test case and cleanup is also recommended as sometimes cleanup won't work when executed immediately after a test. Alternatively it might also be worth considering two different delay options. One delay between individual commands and one delay between test cases.

[thebleucheese]

this will be addressed in the upcoming release with the workaround suggested (delay will apply to between tests vs between commands)

[thebleucheese]

this will be addressed in the upcoming release with the workaround suggested (delay will apply to between tests vs between commands)

Behavior changed in VECTR 8.8.0

[zaicurity]

Hi, I finally got around to testing this and it seems the delay is still getting applied between individual commands. Here is an example test case for demo purposes which simply displays the system time three times in a row:

Building a runtime with 60 seconds delay:

When running this it can be seen that the delay is applied between each individual command within the test case:

[doodleincode]

I'm not able to replicate what you're seeing on the latest VECTR release. Did you use the latest compose file included in the release? The release included updates to the RTA services as well:

• rta_webserver:1.5.5
• rta_builder:1.5.5

[zaicurity]

I'm not able to replicate what you're seeing on the latest VECTR release. Did you use the latest compose file included in the release? The release included updates to the RTA services as well:

• rta_webserver:1.5.5
• rta_builder:1.5.5

Thank you for the reply! You are correct. We had apparently not updated that component. Now the delay no longer occurs between individual commands, as expected.

One additional suggestion would be to add a short delay between test execution and cleanup as for some test cases the cleanup will not work if executed immediately.

[doodleincode]

Awesome!

That's a good suggestion. Could you create a new issue for tracking purposes? We're going through a pretty extensive architectural update right now so it will be a while until we're able to get to this.