Test subset concept, parallel test execution

[tkonieczny]

Context The current testkube Tests and TestSuites are fine for organizing simple, low-level, or just fast running tests. No matter if the tests use Curl, Postman, SoapUI, K6 or Artillery executors, they will usually execute relatively fast. However, the high-level, E2E tests are slow by design.

The typical E2E test set may execute several dozen times longer than other test types. Usually the main way of making E2E tests faster is running them in parallel. Let's assume there is a set of 30 Cypress tests, and execution of each of them takes about 20 seconds, so it takes about 10 minutes to execute this whole set (without parallel execution). Considering this Cypress test set as just a single Testkube Test is ineffective. It's not possible to split the Test into smaller parts, and to run the tests (cypress) in parallel.

Describe the solution you'd like

• Adding new, additional abstraction Test subset - something smaller than the current Test, which will allow to split it
• Test Parametrization - splitting Test into different Test subsets based on files/tags, etc.
• Parallel execution of Test subsets

Additional context It's not limited to just Cypress - the same abstraction/feature will be useful for other high-level testing frameworks like Selenium, Playwright, etc.

[vsukhin]

Technically, we support now running multiple tests in parallel, if you run them using selector parameter. Then all matched tests by lables will be started in parallel with a concurency parameter value

[vsukhin]

So, it's about groupping internal tests in Testkube Test

[tkonieczny]

Update after discussion: Cypress (with official Dashboard), or sorry-cypress automatically split tests by files in case of parallel mode. Doing it automatically may be a great option in the future, but for now I think we should start with simpler alternative, which won't even require external "coordinator".

Let's assume it's cypress project with the following structure of test files.

.
.
└── e2e
    ├── some
    │   └── folder
    │       ├── spec4.cy.js
    │       └── spec5.cy.js
    ├── spec1.cy.js
    ├── spec2.cy.js
    └── spec3.cy.js

Cypress allows to run a single spec with --spec param: for example cypress run --spec "cypress/e2e/spec2.cy.js". It's also possible to run specs from specific directory: cypress run --spec "cypress/e2e/some/folder/**". Both spec files, and directories can be combined cypress run --spec "cypress/e2e/spec2.cy.js,cypress/e2e/some/folder/**", so it's possible to choose any set of specs, or directories. (more details here: https://docs.cypress.io/guides/guides/command-line#cypress-run-spec-lt-spec-gt)

So, my idea is to allow splitting Tests into sub-sets/sub-sets by files/folders:

apiVersion: tests.testkube.io/v3
kind: Test
metadata:
    name: some_name
    namespace: testkube
spec:
    type: cypress/project
    content:
        type: git-dir
        repository:
            type: git-dir
    sub-sets:
        split-by:
            files:
            - cypress/e2e/spec1.cy.js
            - cypress/e2e/spec2.cy.js
            - cypress/e2e/spec3.cy.js
            - cypress/e2e/some/folder/** # this one will execute both 4 and 5

So, the Test will be split to 4 different subsets based on spec files/folders - each of subset Tests will just run cypress with specific spec param from split-by.

I wrote this examples based on cypress, but the whole abstraction is reusable to any other framework/tool/runner that supports running specific files (they just need to be passed to the runner). Other runners (like Cucumber) may also support selecting tests by tags, so in the future it can be extended with split-by tags.

[olensmar]

would it be better to do this at the TestSuite level - allowing the possibility to run a subset of Tests in parallel? This would allow parallelisation of different types of tests - for example a load test and functional tests in parallel (which is very useful to see if load affects functionality)

For the above scenario one would have to either define separate cypress Tests for each folder - or be able to parameterise the Cypress executor with which folder to run - and specify that as a parameter in the parallelisation configuration...!?

[tkonieczny]

@olensmar https://github.com/kubeshop/testkube/issues/2356 I think we need both parallel execution on the Testsuite and Test level. Anything that can be parallelized should be run in parallel - that's the only option to get complete quality feedback fast. Theoretically it's possible to create multiple smaller Tests instead of single one, but it's better not to do it. This issue will allow to have the tests organized on the Cypress (or any other framework) side and still run the subsets in parallel.

[vsukhin]

Hey team! Please add your planning poker estimate with Zenhub @nicufk @vLia