Incomplete steps/report returned by `/executions` endpoint in case of multiple test files (junit)

[tkonieczny]

Based on user report - execution steps returned by /executions endpoint are incomplete if multiple test files are run. Example: 3 cypress test files were ran, the steps/report only contains one of them.

[TheBrunoLopes]

• Junit response object does not contain all steps. @tkonieczny to provide more info.

[tkonieczny]

Reproduced: https://demo.testkube.io/tests/cypress-multi-files-test/executions/64f70c1f17807111c6a9879b Example: 2 cypress test files (smoke-without-envs.cy.js, and smoke-without-envs-2.cy.js) are executed. But, /executions/64f70c1f17807111c6a9879b executionResult.steps only contans the test from the first file (smoke-without-envs.cy.js) - the test from the second one is missing.

It may be related to junit TestSuite not being parsed.

[vsukhin]

@tkonieczny this is running command ./node_modules/cypress/bin/cypress run --reporter junit --reporter-options mochaFile=/data/repo/test/cypress/executor-tests/cypress-without-envs/results/junit.xml,toConsole=false --env

junit report file. is it the same if we execute cypress test locally?

<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="Mocha Tests" time="0.0550" tests="1" failures="0">
  <testsuite name="Root Suite" timestamp="2023-09-11T12:57:48" tests="0" file="cypress/e2e/smoke-without-envs.cy.js" time="0.0000" failures="0">
  </testsuite>
  <testsuite name="Smoke test 1" timestamp="2023-09-11T12:57:48" tests="1" time="0.0550" failures="0">
    <testcase name="Smoke test 1 expect 1=1" time="0.0550" classname="expect 1=1">
    </testcase>
  </testsuite>
</testsuites>

[vsukhin]

So, junit was parsed properly, the question is, why does it contain only one script

[TheBrunoLopes]

@tkonieczny do you think this is an issue with our cypress executor ?

[tkonieczny]

Partially - looks like it's Cypress issue "feature" that report file is overwritten: https://github.com/cypress-io/cypress/issues/1824

So, based on Cypress docs (https://docs.cypress.io/guides/tooling/reporters#Merging-reports-across-spec-files) the command would need to be changed to: ./node_modules/cypress/bin/cypress run --reporter junit --reporter-options "mochaFile=/data/repo/test/cypress/executor-tests/cypress-without-envs/results/junit-[hash].xml,toConsole=false --env" (this [hash] need to be added)

I checked it locally, and multiple (two in this case) junit files are then generated:

So, the cypress executor would need to have this command adjusted, and additionally we would probably need to support multiple junit files for single Test

[vsukhin]

ok, thank you, in this case, we have a plan:

1. change command line
2. support multiple junit reports

[tkonieczny]

Looks like it's still missing: https://demo.testkube.xyz/tests/cypress-12-executor-smoke-electron/executions/65255d9e51476cbbdca44e6b (this specific execution need to be checked because of run overwrites)

$ kubectl describe executor cypress-executor -ntestkube-integration|grep Image
  Image:  kubeshop/testkube-cypress-executor:fb932a6