Fail fast option - Githubissues

wttech / aet

AET - a system that detects visual changes on web sites and performs basic page health checks

https://wttech.blog/open-source/

Apache License 2.0

143 stars 48 forks source link

Fail fast option #80

Open jwadolowski opened 7 years ago

jwadolowski commented 7 years ago

Hey,

after a while with AET I came to a conclusion that it'd be great to have a fail fast option.

Our test suite lasts for around 10 minutes (2 Karaf workers) and the goal is to keep it green all the time. Right now we have to wait until the end to find out that 3rd out of 60 pages failed.

Is it possible to stop processing upon first error? This should significantly shorten feedback loop.

Regards, Kuba

tommix1987 commented 7 years ago

@jwadolowski to be completely honest due to various rendering issues keeping AET green all the time might be pretty hard as in big project we see false positivites pretty often (too often to base a continuous delivery process automation on result from AET - a looot of builds would be failed ;) )

jwadolowski commented 7 years ago

Well, no one said that's an easy process, but it's definitely achievable :)

Last year I participated in a project (quite a big one) that heavily relied on AET as a base for release and it worked quite well. We run AET in CI pipeline and simply paid a lot of attention to its stability.

@tommix1987, you actually touched upon a quite interesting topic - stability. If your tests randomly fail - what's the value they bring to the table? Is there any reason to keep such tests in place if you can't rely on them?

Getting back to that fail fast feature - in my opinion that'd be a great addition to AET, as there's no point to wait for the entire test to complete if you know something's not right since the very beginning.

malaskowski commented 7 years ago

Idea for a fast fail is very interesting. However it is not so easy to define... Please remember that AET tool does not detect errors - it detects changes. As I understand you wold like a mode that enables failing whole suite if more than X changes were detected? Or rather when there is some level of errors (e.g. 5 tested pages returned 404)?

jwadolowski commented 7 years ago

A change is an error in my case :)

I'd like to fail the whole build as soon as any change (number of changes greater than 0) was detected by AET to shorten feedback loop.

The second case (level of errors) you brought up is quite interesting too, but up to some extent can be covered by xUnit plugin in Jenkins (leaving aside the need to wait for build end).

tMaxx commented 3 years ago