webdriverio-community / wdio-cucumberjs-json-reporter

A WDIO reporter that creates CucumberJS JSON files for WebdriverIO
https://webdriver.io/docs/wdio-cucumberjs-json-reporter
MIT License
22 stars 30 forks source link

Json out put failing when parsed against Cucumber schema on import to Jira via Zephyr scale plugin #101

Closed Nemesischild closed 1 year ago

Nemesischild commented 1 year ago

I am running a large number of tests that are all working quite nicely and I get a full set of JsOn File which generate a nice HTML report.

However when I I then try to run an update to Jira / Zephyr scale cloud via the Jenkins plugin I am getting parse failures as the 'line' element in the json output is added to the json as Null (see Json in attached zip).

[ANDROID-dev-BAT-08-10-2022-13_55_58 (1).zip](https://github.com/webdriverio-community/wdio-cucumberjs-json-reporter/files/9301385/ANDROID-dev-BAT-08-10-2022-13_55_58.1.zip)

Parse Error:
`[Zephyr Scale] [INFO] Publishing test results...
Finished: Uploading to mobile-build-reports/ANDROID/dev/BAT/ANDROID-dev-BAT-08-10-2022-13:55:58.zip
[Zephyr Scale] [ERROR] Error: 

{
  "errorCode": 400,
  "message": "Error parsing the following file(s): /var/lib/jenkins/jobs/Janus/jobs/QA/jobs/Core/jobs/BS-Scale-Test/builds/107/target/cucumber_tmp/jp-1289---accounts---logging-into-william-hill-from-betslip_16601393916108364.json - reason: Invalid value '{\"keyword\"=>\"Scenario\", \"type\"=>\"scenario\", \"description\"=>\"\", \"name\"=>\"Selecting the back button on the Log In page\", \"tags\"=>[{\"name\"=>\"@INT\"}, {\"name\"=>\"@INT_GREEN\"}, {\"name\"=>\"@JP-1289\"}, {\"name\"=>\"@SMOKE\"}, {\"name\"=>\"@BAT\"}], \"id\"=>\"jp-1289---accounts---logging-into-william-hill-from-betslip;selecting-the-back-button-on-the-log-in-page\", \"steps\"=>[{\"keyword\"=>\"Before\", \"name\"=>\"\", \"result\"=>{\"status\"=>\"passed\", \"duration\"=>3000000}, \"line\"=>nil}, {\"keyword\"=>\"Before\", \"name\"=>\"\", \"result\"=>{\"status\"=>\"passed\", \"duration\"=>398000000}, \"line\"=>nil}, {\"keyword\"=>\"Before\", \"name\"=>\"\", \"result\"=>{\"status\"=>\"passed\", \"duration\"=>2849000000}, \"line\"=>nil}, {\"keyword\"=>\"Given\", \"name\"=>\"I am on \\\"betslip\\\" screen\", \"result\"=>{\"status\"=>\"passed\", \"duration\"=>2205000000}, \"line\"=>nil}, {\"keyword\"=>\"And\", \"name\"=>\"I select William Hill bookmaker\", \"result\"=>{\"status\"=>\"passed\", \"duration\"=>6768000000}, \"line\"=>nil}, {\"keyword\"=>\"And\", \"name\"=>\"I select the login CTA\", \"result\"=>{\"status\"=>\"passed\", \"duration\"=>1621000000}, \"line\"=>nil}, {\"keyword\"=>\"When\", \"name\"=>\"I select the back button\", \"result\"=>{\"status\"=>\"passed\", \"duration\"=>3276000000}, \"line\"=>nil}, {\"keyword\"=>\"Then\", \"name\"=>\"I will be returned to the Compare Odds screen\", \"result\"=>{\"status\"=>\"passed\", \"duration\"=>2831000000}, \"line\"=>nil}, {\"keyword\"=>\"After\", \"name\"=>\"\", \"result\"=>{\"status\"=>\"passed\", \"duration\"=>1672000000}, \"line\"=>nil}, {\"keyword\"=>\"After\", \"name\"=>\"\", \"result\"=>{\"status\"=>\"passed\", \"duration\"=>1000000}, \"line\"=>nil}, {\"keyword\"=>\"After\", \"name\"=>\"\", \"result\"=>{\"status\"=>\"passed\", \"duration\"=>1000000}, \"line\"=>nil}]}' at path /0/elements/0. Validation schema used: {\"type\"=>\"object\", \"properties\"=>{\"keyword\"=>{\"type\"=>\"string\"}, \"type\"=>{\"type\"=>\"string\"}, \"id\"=>{\"type\"=>\"string\"}, \"line\"=>{\"type\"=>\"integer\"}, \"name\"=>{\"type\"=>\"string\"}, \"tags\"=>{\"type\"=>\"array\", \"items\"=>{\"$ref\"=>\"#/definitions/tag\"}}, \"steps\"=>{\"type\"=>\"array\", \"items\"=>{\"$ref\"=>\"#/definitions/step\"}}}, \"required\"=>[\"type\", \"line\", \"name\"]}. Details: {\"missing_keys\"=>[\"line\"]}\\nInvalid value '' at path /0/elements/0/steps/0/line. Validation schema used: {\"type\"=>\"integer\"}.\\nInvalid value '' at path /0/elements/0/steps/1/line. Validation schema used: {\"type\"=>\"integer\"}.\\nInvalid value '' at path /0/elements/0/steps/2/line. Validation schema used: {\"type\"=>\"integer\"}.\\nInvalid value '' at path /0/elements/0/steps/3/line. Validation schema used: {\"type\"=>\"integer\"}.\\nInvalid value '' at path /0/elements/0/steps/4/line. Validation schema used: {\"type\"=>\"integer\"}.\\nInvalid value '' at path /0/elements/0/steps/5/line. Validation schema used: {\"type\"=>\"integer\"}.\\nInvalid value '' at path /0/elements/0/steps/6/line. Validation schema used: {\"type\"=>\"integer\"}.\\nInvalid value '' at path /0/elements/0/steps/7/line. Validation schema used: {\"type\"=>\"integer\"}.\\nInvalid value '' at path /0/elements/0/steps/8/line. Validation schema used: {\"type\"=>\"integer\"}.\\nInvalid value '' at path /0/elements/0/steps/9/line. Validation schema used: {\"type\"=>\"integer\"}.\\nInvalid value '' at path /0/elements/0/steps/10/line. Validation schema used: {\"type\"=>\"integer\"}., /var/lib/jenkins/jobs/Janus/jobs/QA/jobs/Core/jobs/BS-Scale-Test/builds/107/target/cucumber_tmp/jp-1293---betslip---display-multiple-bets-on-betslip_16601393123740454.json - reason: Invalid value '{\"keyword\"=>\"Scenario\", \"type\"=>\"scenario\", \"description\"=>\"\", \"name\"=>\"Display View selections - double\", \"tags\"=>[{\"name\"=>\"@INT\"}, {\"name\"=>\"@INT_GREEN\"}, {\"name\"=>\"@SMOKE\"}, {\"name\"=>\"@BAT\"}], \"id\"=>\"jp-1293---betslip---display-multiple-bets-on-betslip;display-view-selections---double\", \"steps\"=>[{\"keyword\"=>\"Before\", \"name\"=>\"\", \"result\"=>{\"status\"=>\"passed\", \"duration\"=>3000000}, \"line\"=>nil}, {\"keyword\"=>\"Before\", \"name\"=>\"\", \"result\"=>{\"status\"=>\"passed\", \"duration\"=>609000000}, \"line\"=>nil}, {\"keyword\"=>\"Before\", \"name\"=>\"\", \"result\"=>{\"status\"=>\"passed\", \"duration\"=>2949000000}, \"line\"=>nil}, {\"keyword\"=>\"Given\", \"name\"=>\"I logout of all the bookmakers\", \"result\"=>{\"status\"=>\"passed\", \"duration\"=>3000000}, \"line\"=>nil}, {\"keyword\"=>\"And\", \"name\"=>\"I logged in to \\\"William Hill\\\" Bookmaker via Betslip\", \"result\"=>{\"status\"=>\"passed\", \"duration\"=>25014000000}, \"line\"=>nil}, {\"keyword\"=>\"And\", \"name\"=>\"I click on the \\\"Racing\\\" Navigation CTA\", \"result\"=>{\"status\"=>\"passed\", \"duration\"=>3439000000}, \"line\"=>nil}, {\"keyword\"=>\"And\", \"name\"=>\"I select the Sort by Time CTA\", \"result\"=>{\"status\"=>\"passed\", \"duration\"=>3319000000}, \"line\"=>nil}, {\"keyword\"=>\"Given\", \"name\"=>\"I select 2 selection from different races on Index screen\", \"result\"=>{\"status\"=>\"passed\", \"duration\"=>24921000000}, \"line\"=>nil}, {\"keyword\"=>\"When\", \"name\"=>\"I select Betslip CTA\", \"result\"=>{\"status\"=>\"passed\", \"duration\"=>4407000000}, \"line\"=>nil}, {\"keyword\"=>\"Then\", \"name\"=>\"the bet type double is displayed\", \"result\"=>{\"status\"=>\"failed\", \"duration\"=>23147000000, \"error_message\"=>\"AssertionError: expect double multibet to be found: expected false to be true\\n    + expected - actual\\n\\n    -false\\n    +true\\n\\n    at World.<anonymous> (/home/jenkins/workspace/workspace/Janus/QA/Core/BS-Scale-Test/tests/e2e/steps/betslip/betslip.Then.js:847:5)\"}, \"line\"=>nil}, {\"keyword\"=>\"And\", \"name\"=>\"the View Selections icon and text will be displayed\", \"result\"=>{\"status\"=>\"pending\", \"duration\"=>0}, \"line\"=>nil}, {\"keyword\"=>\"After\", \"name\"=>\"\", \"result\"=>{\"status\"=>\"passed\", \"duration\"=>1624000000}, \"line\"=>nil}, {\"keyword\"=>\"After\", \"name\"=>\"\", \"result\"=>{\"status\"=>\"passed\", \"duration\"=>1000000}, \"line\"=>nil}, {\"keyword\"=>\"After\", \"name\"=>\"\", \"result\"=>{\"status\"=>\"passed\", \"duration\"=>1000000}, \"line\"=>nil}]}' at path /0/elements/0. Validation schema used: {\"type\"=>\"object\", \"properties\"=>{\"keyword\"=>{\"type\"=>\"string\"}, \"type\"=>{\"type\"=>\"string\"}, \"id\"=>{\"type\"=>\"string\"}, \"line\"=>{\"type\"=>\"integer\"}, \"name\"=>{\"type\"=>\"string\"}, \"tags\"=>{\"type\"=>\"array\", \"items\"=>{\"$ref\"=>\"#/definitions/tag\"}}, \"steps\"=>{\"type\"=>\"array\", \"items\"=>{\"$ref\"=>\"#/definitions/step\"}}}, \"required\"=>[\"type\", \"line\", \"name\"]}. Details: {\"missing_keys\"=>[\"line\"]}\\nInvalid value '' at path /0/elements/0/steps/0/line. Validation schema used: {\"type\"=>\"integer\"}.\\nInvalid value '' at path /0/elements/0/steps/1/line. Validation schema used: {\"type\"=>\"integer\"}.\\nInvalid value '' at path /0/elements/0/steps/2/line. Validation schema used: {\"type\"=>\"integer\"}.\\nInvalid value '' at path /0/elements/0/steps/3/line. Validation schema used: {\"type\"=>\"integer\"}.\\nInvalid value '' at path /0/elements/0/steps/4/line. Validation schema used: {\"type\"=>\"integer\"}.\\nInvalid value '' at path /0/elements/0/steps/5/line. Validation schema used: {\"type\"=>\"integer\"}.\\nInvalid value '' at path /0/elements/0/steps/6/line. Validation schema used: {\"type\"=>\"integer\"}.\\nInvalid value '' at path /0/elements/0/steps/7/line. Validation schema used: {\"type\"=>\"integer\"}.\\nInvalid value '' at path /0/elements/0/steps/8/line. Validation schema used: {\"type\"=>\"integer\"}.\\nInvalid value '' at path /0/elements/0/steps/9/line. Validation schema used: {\"type\"=>\"integer\"}.\\nInvalid value '' at path /0/elements/0/steps/10/line. Validation schema used: {\"type\"=>\"integer\"}.\\nInvalid value '' at path /0/elements/0/steps/11/line. Validation schema used: {\"type\"=>\"integer\"}.\\nInvalid value '' at path /0/elements/0/steps/12/line. Validation schema used: {\"type\"=>\"integer\"}.\\nInvalid value '' at path /0/elements/0/steps/13/line. Validation schema used: {\"type\"=>\"integer\"}."
}

[Zephyr Scale] [ERROR] Test Cycle was not created `

wdio Config file wdio.shared.conf.js.txt

Jenkins POST Step `post{ always {

                      publishTestResults serverAddress: "https://racingpost.atlassian.net/",
                                  projectKey: "JTEST",
                                  format: "Cucumber",
                                  filePath: "test-report/json/*.json",
                                  autoCreateTestCases: false,
                                      customTestCycle: [
                                          name: "Jenkins Test Build",
                                          description: "Results from Jenkins Build",
                                          customFields: "{\"single-line\":\"Sprint 66\"}"
                                        ]
                   }
               }`

Now I obviously understand that the error is being returned by a separate system that is trying to parse the output from this package but I have been back and forth with the Smartbear support team (piublisher of Zephyr scale) about this and they are adament that they are parsing the json against the correct schema which states that the 'line' element is an integer and cannot there fore be 'null'

From the SmartBear Support team I received the following

'I received a feedback from our development team.

We are using the official schema here:

https://github.com/cucumber/cucumber-json-schema/blob/main/schema.json
On it, “line” is type integer and required. Hence it can’t be null.

Please update it and try again, it will work after that.'

What I am trying to figure out and get help with here is

  1. am I somehow generating invalid JSON
  2. I messing something else in my code/config to ensure that the line field is never null or if the output coming out of this package is invalid in itself and needs fixing.
  3. are they doing something wrong when parsing and what support can I get to prove that they need to fix something their side

Any/All help very much appreciated at this point

Nemesischild commented 1 year ago

Looks like these PR's highlights the same issue but may not be included in the latest package yet??

88 / #89

rp-eduardoantunez commented 1 year ago

Hi @Nemesischild, any good news on this? I see that the issue is not fixed in 4.4.3.