lefthandedgoat / canopy

f# web automation and testing library, built on top of Selenium (friendly to c# also)
http://lefthandedgoat.github.io/canopy/
MIT License
507 stars 115 forks source link

Make tests fail instead of crash when exception is raised in before or after #487

Closed Pwntheon closed 5 years ago

Pwntheon commented 5 years ago

We're changing a few things in our application and because of this, a few of the steps needed to setup a couple of our tests in before functions fail.

This unfortunately raises an exception instead of failing the test - i suspect because the test hasn't "started" yet.

Since this rewrite is taking some time we'd still like for our tests to run, but when we're running a production build of the canopy project on our jenkins server, the exception causes the whole application to crash, thus preventing us from testing.

We could wrap the whole shebang in a try-catch, but ideally i'd like for the tests affected to fail and the rest of the test suites to continue as usual. Is there any way to achieve this?

Also, we're writing quite a lot of tests in many contexts so any solution that prevents a lot of additional code having added to each before and after function would be a big plus as this increases boilerplate.

lefthandedgoat commented 5 years ago

You can make a new module and create your own try-catch wrapped version of before, then make sure in your test the last line of your open statements is for your new module. In F# the most recently defined version of a function is the one that is used in your code.

Something like this: (untested)

module canopyExtensions

open canopy.runner.classic

let before f =
  try
    before f
  with ex -> //do whatever

Looking at the runner code the Before and the actual Test are run at the same time : https://github.com/lefthandedgoat/canopy/blob/cfe3fa17526c290c29dfb8f2b8873577bf7d1792/src/canopy/runner.fs#L162

In a try catch: https://github.com/lefthandedgoat/canopy/blob/cfe3fa17526c290c29dfb8f2b8873577bf7d1792/src/canopy/runner.fs#L134-L137

If you can create a sample test on a public site that demonstrates escaping this I can fix it.

Pwntheon commented 5 years ago

Tried to recreate the problem in isolation, and it appears you are correct.

An error in once however just got swallowed, not sure if this is intended. Anyway this will just lead to tests failing because of wrong preconditions so i guess it's fine either way.

If we experience the problem again i'll open a new ticket after doing some more thorough research. Thanks a lot for your help.