Could not get Newman result

[guiyomh]

Describe the bug

I provision a postman test and overload the execution arguments like this:

apiVersion: tests.testkube.io/v3
kind: Test
metadata:
  name: my-smoke-test
  namespace: my-namespace
  labels:
    type: ms
    version: feat-testkube-newman-5eb99672
spec:
  type: postman/collection
  content:
    repository:
      branch: feat/testkube-newman
      path: tests/newman/smoke-test.postman_collection.json
  source: ms-test
  executionRequest:
    variables:
      BASE_URL:
        name: BASE_URL
        value: "https://blabla.com"
        type: basic
      NODE_ENV:
        name: NODE_ENV
        value: "production"
        type: basic
    args:
      - "--reporters"
      - "cli,json,junit"
      - "--reporter-json-export"
      - "/data/artifacts/report.json"
      - "--reporter-junit-export"
      - "/data/artifacts/junit.xml"
    artifactRequest:
      storageClassName: ebs-gp3
      volumeMountPath: /share/
      dirs:
      - /data/artifacts/
    executePostRunScriptBeforeScraping: false

When I run the test, I can see that the executor generates this command line for me: newman run /data/repo/tests/newman/smoke-test.postman_collection.json -e /tmp/testkube-tmp4270038816 --reporters cli,json --reporter-json-export /tmp/testkube-tmp1625347806.json --reporters cli,json,junit --reporter-json-export /data/artifacts/report.json --reporter-junit-export /data/artifacts/junit.xml

It's not a problem for newman, it takes into account the last arguments and generates the reports at the desired location.

Tests run successfully, but the runner tries to read another result file, which has not been generated.

I think it's because the arguments are duplicated in the command line. There are the default ones and the ones I added thinking I was overloading them.

[vsukhin]

hey, @guiyomh Can you please add argsMode: override to executionRequest and run the test? in cli --args-mode string usage mode for argumnets. one of append|override (default "append")

[guiyomh]

Ok, I've added the option. It overrides the command line arguments. But it also removes :

• the file with environment variables -e /tmp/testkube-tmp4270038816
• the sub command run
• and the postman collection file

- newman run /data/repo/tests/newman/smoke-test.postman_collection.json -e /tmp/testkube-tmp4270038816 --reporters cli,json --reporter-json-export /tmp/testkube-tmp1625347806.json --reporters cli,json,junit --reporter-json-export /data/artifacts/report.json --reporter-junit-export /data/artifacts/junit.xml
+ newman --reporters cli,json,junit --reporter-json-export /data/artifacts/report.json --reporter-junit-export /data/artifacts/junit.xml

I think this is where the problem comes in: https://github.com/kubeshop/testkube/blob/develop/contrib/executor/postman/pkg/runner/newman/newman.go#L117

I don't know if this is a good solution, but perhaps we could reassign the tmpName variable with the value read when it's not <reportFile>.

[vsukhin]

this is a full arg list: "run", "", "-e", "", "--reporters", "cli,json", "--reporter-json-export", ""

leave what you need and combine with your args, like:

• "run"
• ""
• "-e"
• ""
• "cli,json,junit"
• "--reporter-json-export"
• "/data/artifacts/report.json"
• "--reporter-junit-export"

[guiyomh]

Nice it work Now I have the right commande line:

newman run /data/repo/tests/newman/smoke-test.postman_collection.json -e /tmp/testkube-tmp53552217 --reporters cli,json,junit --reporter-json-export /data/artifacts/report.json --reporter-junit-export /data/artifacts/junit.xml

Here the final config:

apiVersion: tests.testkube.io/v3
kind: Test
metadata:
  name: my-smoke-test
  namespace: my-namespace
  labels:
    type: ms
    version: feat-testkube-newman-5eb99672
spec:
  type: postman/collection
  content:
    repository:
      branch: feat/testkube-newman
      path: tests/newman/smoke-test.postman_collection.json
  source: ms-test
  executionRequest:
    variables:
      BASE_URL:
        name: BASE_URL
        value: "https://blabla.com"
        type: basic
      NODE_ENV:
        name: NODE_ENV
        value: "production"
        type: basic
    args:
+     - "run"
+     - "<runPath>"
+     - "-e"
+     - "<envFile>"
      - "--reporters"
      - "cli,json,junit"
      - "--reporter-json-export"
      - "/data/artifacts/report.json"
      - "--reporter-junit-export"
      - "/data/artifacts/junit.xml"
+   argsMode: override
    artifactRequest:
      storageClassName: ebs-gp3
      volumeMountPath: /share/
      dirs:
      - /data/artifacts/
    executePostRunScriptBeforeScraping: false

[vsukhin]

congrats!