search

saadichouaib / codeceptjs-xray-cloud-helper

"codeceptjs-xray-cloud-helper" is a plugin that automatically retrieves the result of CodeceptJS tests and sends them to XRAY/JIRA.
MIT License
13 stars 1 forks source link

Test result excluded from Xray report when test does not use a browser or produce a corrupt screenshot file #8

Open jmychasiw opened 1 year ago

jmychasiw commented 1 year ago

Context

We use CodeceptJS for API testing as well as E2E. In these cases we will not instantiate a browser unless needed. Found that test will be excluded if no screenshot is present.

Actual Behavior

Observed the error in console:

TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string or an instance of Buffer or URL. Received undefined

Traced traced the error close to: https://github.com/saadichouaib/codeceptjs-xray-cloud-helper/blob/58765837ec6f2a7a02cf158ea2a6b19ea57748b8/lib/common.js#L46 where the screenshot is undefined.

See

image

Steps to Reproduce the Problem

Configure project with no browser.
See attached project: xray-plugin-debug.zip

jmychasiw commented 1 year ago

The setting testExecutionSendEvidenceOnFail: false will prevent the problem. However I think for improved user/developer experience this error should throw an informative validation message indicating that it was expecting a screen shot when none was created

saadichouaib commented 1 year ago

Should add a validation to https://github.com/saadichouaib/codeceptjs-xray-cloud-helper/blob/develop/helpers/conf_schema_checker.js

jmychasiw commented 1 year ago

Some new information on this issue. I ran across a random case where the screen shot was blank or not properly generated. The path error was thrown and the test was silently excluded from the import. TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string or an instance of Buffer or URL. Received undefined.

I agree that the suggested validation should be added but the actual file handling code should handle the case where a file is bad and fail more gracefully.

The result of the this error was the the pipeline failed but the generated xray test execution just showed all tests pass but with a lower test count (which was not initially detected by the team)