microsoft / WinAppDriver

Windows Application Driver
MIT License
3.69k stars 1.4k forks source link

Exception Handling Best Practices in Test Solutions #348

Open AmirHNK opened 6 years ago

AmirHNK commented 6 years ago

This is just a general question.

Given that exceptions are thrown in many situations (such as when creating a new test session, or finding an element by name), what is the best way to handle them?

I know that if an exception is thrown, the scenario containing the exception is halted and marked as failed while the rest of the scenarios are executed regardless. So is it recommended to let the exceptions bubble all the way up to the test runner? Or should I be catching them but calling Assert.Fail methods in higher layers of code to issue a fail? FYI, the example projects included on this website do not include try/catch blocks around statements that may throw exceptions.

PandaMagnus commented 6 years ago

Personally, I dislike anytime I have to put a try/catch in a test itself, so I avoid it unless there's really no other way to handle a scenario. Most of my exception handling occurs at the page/window definition level, and any extensions and underlying framework I write. So for a web based app, I tend to use my own WebElement implementation that has a bunch of try/catches and retries due to the slowness of the sites I'm working on which minimizes the need for them in most other cases. For desktop apps using TestStack White, WAP, etc., I tend to use them in the window definitions and my base test class more often since a lot of these tools already are wrapping other libraries that all may have their own ideas of retries (and wrapping retries with more retries ends up resulting in some very unexpected behavior.)

So I guess, tl;dr version for my general rule of thumbs: I put try/catches as low level as I can without wrapping other try/catches if possible, and in most cases the catch results in a retry for a certain period of time or certain number of attempts before throwing.

I also very rarely use Assert.Fail, so YMMV. Generally speaking I let the throw be the failure, which works for MSTest, NUnit, and XUnit. I understand this is a departure from typical programming, but the places I work always force us to really strike a balance between designing classes to avoid exceptions, and dealing with the fact that UI testing has some associated fragility that must be dealt with so we're not failing anytime a page or window takes 2 seconds longer to load.