search

SecurityRiskAdvisors / VECTR

VECTR is a tool that facilitates tracking of your red and blue team testing activities to measure detection and prevention capabilities across different attack scenarios
1.35k stars 159 forks source link

Variables in test cases for Automation Runtime are inherited across test cases #215

Closed chryzsh closed 1 year ago

chryzsh commented 1 year ago

Describe the bug When running multiple test cases at once with the Automation Runtime on Windows, any variable name that are used across multiple test cases are applied to all test cases in the runtime. Thus, the variable of individual test cases are not respected. This is a problem because a lot of the automation for test cases include commonly used variable names like "path", "url" and similar. From what I can tell this happens individually of executor, so I can confirm the bug is present on both Powershell, CMD and bash. I have tested it on Windows, Linux and Mac.

To Reproduce Steps to reproduce the behavior: Preparation: Go to Assessments, select a Campaign and create two Test Cases. For each of them: Click on Configure under Automation & Logging Verify that the test case has an automation configuration, or create one. Use the same variable name for both cases, but different values. Save In Campaign Actions click Build Automation Runtime. This will prep and download a ZIP-file containg the binary to be executed Extract the zip file to a folder. Execution: Open a Console window and navigate to the folder. Execute the automation runtime binary. Inspect the STDOUT in the console window and/or JSON file which is written to test that the first environment variable is inherited to future test cases when they are executed.

Expected behavior I expected the variables set on each test case to only be used on that test case, and not inherited to future test cases in the same runtime binary.

Screenshots To provide a minimal and clear example, I've here created two test cases : Test 1 and Test 2. Both are identical, but in the automation I have configured Test 1 to use the variable url set to http://test1.xyz and Test 2 the same variable name but value http://test2.xyz.

image image image

Here, both test cases are executed, and the variable from the first executed test case, here "Test Case 1" and its url variable is applied to Test Case 2 because the variable name is the same. So Test Case 2 does not get executed with the correct url set in the automation configuration.

image

I replicated this test by changing the executor to Bash and testing the binary on Linux and Mac. The issue is the same there:

image

Desktop (please complete the following information): OS: Windows 10 Browser: Chrome Version: ce-8.7.1

Additional context A possible workaround is to generate individual test cases. Another workaround is to use individual variable names for all test cases. Neither are very scalable solutions.

SRAPSpencer commented 1 year ago

Thanks for reporting, appreciate the detailed reporting.

We've confirmed this is an issue. No ETA on a fix right now.

thebleucheese commented 1 year ago

Fixed in VECTR 8.8.0