search

Xray-App / xray-junit-extensions

Enhanced JUnit 5 integration with Xray Test Management for Jira
Eclipse Public License 2.0
16 stars 8 forks source link

Assertion result not shown in Xray when uploading evidence #9

Closed ncariss closed 2 years ago

ncariss commented 2 years ago

I am currently doing some JUnit tests that compare very long Maps and Lists using custom Hamcrest matchers. The matchers describe the mismatch in detail once an assertion fails. If I use the Xray-JUnit-Extension to upload the results to Jira, we can see the description of the AssertionError in the TestRun of our Testcases (test type "generic").

If, however, I also add some files as TestRun evidence, all I see in the TestRun in Jira is the link to one of the files. Is this something that could potentially be fixed/changed in the extension or is this due to behaviour of Xray itself?

bitcoder commented 2 years ago

Hi @ncariss , to be able to help you out,

  1. can you please provide more details, including a code snippet where you attach the files as testrun evidence?
  2. can you provide a screenshot of the test run details you see in Xray?
  3. can you provide the junit.xml file that was generated by this extension?
  4. are you using Xray server/datacenter (i.e., on Jira server/datacenter) or Xray cloud (i.e., on Jira cloud)?
ncariss commented 2 years ago

Hi @bitcoder ! Thanks for the quick reply!

We are using Xray Server+DC.

I created a minimum working example here. (Interestingly, mvn test does not find any tests, but running the tests using IntelliJ works. Apparently just including the xray-junit-extensions as a Maven dependency blocks Maven Surefire from finding any tests.)

The XML report and screenshots can also be found in the repo: TEST-junit-jupiter.xml Without evidence uploaded we can see the AssertionError (which is what we want): noEvidenceUploaded

When uploading evidence, the AssertionError is unfortunately replaced by the evidence filename/link: withEvidenceUploaded

Cheers Nick

bitcoder commented 2 years ago

Hi @ncariss , so it's a bug on Xray server/DC side. I've used your report and upload to a Xray cloud instance and it shows fine as you can see. Please reach out to Xray support team and report it (please provide the junit xml file you generate). You can say that the same file seems to work fine in Xray cloud as I've checked it. I'll close this issue here. Thanks for the feedback :)

Note: concerning mvn test, not sure what is happening. I can't replicate it. Besides the workflows are running fine,

image
ncariss commented 2 years ago

Hi Sergio! Thank you for the analysis! I will address this with the Xray support then. Regarding the Maven issues I am facing, I will ask some colleagues to try my repo and report back in another ticket, if I can pinpoint it to the extension.