All the failed tests are reported in a successful state in the HTML report with Appium

[zbrea]

Expected behavior

In the tests which cannot find elements (id) on the screen of the app, must have passed in error in the logs and in the html reporting.

Actual behavior

all tests are successfully passed., even if they cannot find elements (id) on the screen of the app, since I started using gauge-java, for Appium for a native application.

Steps to reproduce

When I run my tests which must fail, I have the impression that the html-report extension does not manage the drivers on Appium. You will find the details of the logs here. I deleted the personal info in the logs)

Gauge version

❯ gauge -v Gauge version: 1.3.2 Commit Hash: 063444f

Plugins

flash (0.0.2) html-report (4.1.0) java (0.7.15) json-report (0.3.6) screenshot (0.1.0) spectacle (0.1.4) xml-report (0.2.3)

Log : (I submitted a short log to facilitate readability)

[sriv]

This is not a problem with html-report. Rather it seems that your tests are not getting marked as failed when the exceptions occur.

if you notice your console log, you'll see gauge sees all scenarios as passed.

Can you share what a sample project? or at least what is the implementation of the step where the error occurs?

[zbrea]

Thank you for your quick response ! I put the same code in a new project created via gauge init, and suddenly the code works fine, it shows the error in the reporting. Yet it is the same code, I do not understand the problem. I am analyzing the structure of my old project and checking it with the pom file to find the cause of the problem.

Log in the new project for the same code:

[sriv]

I suspect that the error in your project is getting handled somewhere and thus the project is not propogating an exception to gauge (which is something gauge needs to identify a failure, ex. assert will throw an exception on failure which gauge will analyse and report_

[zbrea]

Thank you very much for your tip, you are the best. Eeffectively, I just deleted the try / catch, suddenly it works fine with a nice screenshot from my smartphone app. I'm very happy :) It's been over a 2 weeks since I have been on this problem, I never thought that if I handle the error by myself, it will be ignored by . I think you have to remember to take into account the errors handled by try/catch or throw. The first thing a developer will think about is to handle errors with try / catch. You think that it is possible of an evolution to manage it differently ?

Here is an example of a method handled by a try catch :

Even when I use the Custom messages in reports, it displays the error message in a step that is green (color) successful, which is paradoxical, as following :

I want to display short messages with the Custom messages, but in failed steps like in "screenshot 0" in red color. screenshot 0:

I prefer to display a short readable error from "screenshot 1" on my reporting html, than to have a long log like the one in the "screenshot 2" of the reporting .

The reporting must be understandable by a non-technical person (not a developer) !

screenshot 1 (Exception handled by try / catch) 👍 :

screenshot 2 (Exception handled by default by language, which is not understandable by a non-technical person)

[sriv]

Hi, some notes below:

The first thing a developer will think about is to handle errors with try / catch. You think that it is possible of an evolution to manage it differently ?

The norm AFAIK is that if you want the error to propagate, then you rethrow the exception in the catch block. Most testing libraries rely on exceptions to ascertain failure conditions. Take for example the junit assert. If you were to wrap assert in a try/catch, you'd notice that the tests are reported as passed even when the assert fails.

Even when I use the Custom messages in reports, it displays the error message in a step that is green (color) successful, which is paradoxical...

Gauge.writeMessage is a function that takes a string and puts it into the report. It has no sematic knowledge of what the string represents. The green colour is indicative of the fact that the scenario passed (i.e. no exceptions were passed on).

Gauge, by design, is framework agnostic and it relies on users to feed it the right inputs. I can understand this being a learning curve, but unless we tune Gauge to certain use cases, it will be tough to get this intelligence IMO. If you have ideas, please raise issues! It will be most interesting and helpful.

The reporting must be understandable by a non-technical person

I agree with this principle, but it is upto the individual writing the tests to bubble up the right contextual, domain specific message to the report.