Problem in failure analysis

[mzdih]

If I encounter a failure in a class that contains many test cases, as a test automation engineer, analyzing the exact location of the problem can be challenging. This difficulty is illustrated in the screenshot below, where there's a failure in CreateDeliveryPointTest.java, making it hard to determine the specific test that's causing the issue.

[OliverHoenig]

Hello mzdih,

thank you for creating the ticket. To avoid misunderstandings I would ask you to answer the following questions:

• "there's a failure in CreateDeliveryPointTest.java" What kind of error was it? Did a test case fail or is it a potential bug in QTAF that caused unwanted behaviour?
• What behaviour would you have expected from QTAF in the ideal case?
• What would be an error handling that would have helped you in this specific case? And from your point of view, is there general potential for improvement here? If so, what would this improvement look like?

Thanks for your help.

[csvtuda]

After a brief discussion with @mzdih:

What kind of error was it? Did a test case fail or is it a potential bug in QTAF that caused unwanted behaviour?

This is pretty much the root issue, it is very hard to identify. In both situations, the output is usually the above one. But it should be a lot clearer/easier to read if a test case simply failed.

What would be an error handling that would have helped you in this specific case? And from your point of view, is there general potential for improvement here? If so, what would this improvement look like?

• the steps should not be visible at all, only the messages with [Test]
• if provided in the annotated test method, issue IDs should appear in the [Test] messages (DEV-123, ...)
• the [Test] message should include the concrete method name of the test that failed
  • if the test is data-driven, the message should contain more detailed output hinting at the concrete parameters used

Proposed suggestions:

• a logging level flag might solve a few of these issues

[OliverHoenig]

Thank you very much. The issue is now being worked on.

[OliverHoenig]

There was also a request to log the XrayTestID if a test method is annotated with the Xray annotation.

[csvtuda]

Yes, this would be very helpful. I meant this when talking about issue IDs, but probably should have clarified a little bit.

[OliverHoenig]

Everything except the request to log the XrayID has been released in version v0.2.12.