Currently, when a test fails due to an unexpected error on the UI, the test will fail in a couple of ways:
NoSuchElementException
WebDriverException
Instead of having these errors at the top level by default, we should first check to see if there are any errors displayed on the UI first and report those prior to reporting an element couldn't be found or if there's a problem with the WebDriver.
An example failure:
org.openqa.selenium.NoSuchElementException: no such element: Unable to locate element: {"method":"css selector","selector":"[data-qa='some-link']"}
(Session info: chrome=71.0.3578.80)
(Driver info: chromedriver=2.44.609538 (b655c5a60b0b544917107a59d4153d4bf78e1b90),platform=Windows NT 6.1.7601 SP1 x86_64) (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 0 milliseconds
For documentation on this error, please visit: http://seleniumhq.org/exceptions/no_such_element.html
...
This error appeared during the Navigation class so we can assume that there is a problem with navigation; but, we don't know the exact reason why there was a problem with navigation. We must first look at the sauce labs dashboard to view the video from the test run. In this example above, we found that there was a problem with the UI and it displayed a generic error message; therefore, the element we were looking for could not be found.
Not being able to find an element on a page, as it's doing now, infers that the automation did not encounter any issues with the UI and that it simply wasn't able to find the element; which is not entirely the most truthful case.
~As a bonus to this, if we could also report any errors in the console, this would help out tremendously for debugging purposes.~
Risk/other
I'm not sure there's a one sized fits all solution for this. There are a lot of different potential cases for where we would like to see this kind of behavior.
One potential thing to consider is figuring out a way that we can generalize this to apply to multiple element types. Thankfully, the way our strongly typed elements are set up right now, they all inherit off of a parent class.
I think, in general, it is possible to just check the dom for any instances of an attribute on an element of "is-error". ~Maybe we can check the console?~
A/C:
Scaffold should be able to report UI level errors as a first level form of error reporting
If there's no problems with the UI, Scaffold should report the next level down - e.g. NoSuchElementException or WebDriverException
Summary:
Currently, when a test fails due to an unexpected error on the UI, the test will fail in a couple of ways:
Instead of having these errors at the top level by default, we should first check to see if there are any errors displayed on the UI first and report those prior to reporting an element couldn't be found or if there's a problem with the WebDriver.
An example failure:
This error appeared during the Navigation class so we can assume that there is a problem with navigation; but, we don't know the exact reason why there was a problem with navigation. We must first look at the sauce labs dashboard to view the video from the test run. In this example above, we found that there was a problem with the UI and it displayed a generic error message; therefore, the element we were looking for could not be found.
Not being able to find an element on a page, as it's doing now, infers that the automation did not encounter any issues with the UI and that it simply wasn't able to find the element; which is not entirely the most truthful case.
~As a bonus to this, if we could also report any errors in the console, this would help out tremendously for debugging purposes.~
Risk/other
I'm not sure there's a one sized fits all solution for this. There are a lot of different potential cases for where we would like to see this kind of behavior.
One potential thing to consider is figuring out a way that we can generalize this to apply to multiple element types. Thankfully, the way our strongly typed elements are set up right now, they all inherit off of a parent class.
I think, in general, it is possible to just check the dom for any instances of an attribute on an element of "is-error". ~Maybe we can check the console?~
A/C: