search

Financial-Times / n-test

A CLI tool and module for lightweight testing of web applications in browsers, designed for FT.com
4 stars 2 forks source link

Remove or improve the "error threshold" #83

Open adgad opened 6 years ago

adgad commented 6 years ago

The threshold was introduced to accomodate for "random errors" (e.g. browserstack queue full, or random puppeteer timeouts), so as not to get noisy tests.

However it's not a great solution. In cases like https://circleci.com/workflow-run/fa1e2bc6-450b-4c3c-b648-40f51f0580b1, there's only one smoke tests, and it "errored" for a genuine problem - however the tests passed because it was within the threshold.

A better solution would catch and handle specific errors (like the browserstack queue one) and handle them specifically. For example, we could retry tests if it's a queue timeout. Or have specific types of errors that we do allow to fail. And fail for all other errors.

taktran commented 5 years ago

Same thing happens here: https://circleci.com/gh/Financial-Times/next-graphics-page/2150