search

storybookjs / test-runner

🚕 Turn stories into executable tests
https://storybook.js.org/docs/writing-tests/interaction-testing
MIT License
234 stars 72 forks source link

Question: How do I ignore an error #306

Open MTRNord opened 1 year ago

MTRNord commented 1 year ago

Hi,

I have some stories in which the page will work but since it is based on webapis set by user interaction there will be an error claiming the user is not logged in. This currently fails the test even though this is expected and fine when running in the storybook. Is there any way to catch this error and tell test-runner that it is safe to resume?

sw-tracker commented 1 year ago

We are encountering the same problem:

  1. Create a component that throws an exception
  2. Wrap that component in an error boundary (i.e. https://www.npmjs.com/package/react-error-boundary)
  3. Exception is caught by the error boundary and a different INFO component should be rendered

Current Behaviour:

david-morris commented 1 year ago

This is important for me too.

Error-boundary doesn't stop the propagation of an error, only change what gets rendered when it's thrown.

That's a blocker when trying to test while you have components with an error boundary in them.

I think it would be nice to add an error handler to the CSF to deal with this, since apparently Decorators can't be used for this. The normal way to do this is to expect an error to be thrown from the render().

Skipping a smoke-test would be another good option.

david-morris commented 1 year ago

I'm concerned about how slow this seems to be moving. We use chromatic for visual regression testing, including data fetching errors (which happen on load).

Storybook's behavior means that it catches errors when built statically (which is how we do our CI, both test-runner and Chromatic), but it errors when building in dev-mode. So that means that first of all, the smoke-test is disabled when running against a built storybook, which is a bug of its own.

I suspect that has to do with differences in how Storybook behaves in "dev" and "prod" modes.

That means that stories that render errors are incompatible with TDD via test-runner. I'm looking for a workaround but it seems very difficult to change individual stories in a way that would allow testing components with error stories in dev mode.

Another thing I'm interested in is just disabling smoke-tests, since they don't even run in my CI.

sw-tracker commented 1 year ago

This is also stopping us from using Storybook for error usecases. We have fat web-components that make API calls. We dont want to mock the API calls, because we want to test the integration with the BFF (backend-for-frontend). As of now, if the backend responds with an error, Storybook swallows the exception and the catch part of the API call is never called.

yannbf commented 1 year ago

Hey everyone! We just released a test filtering feature in the test-runner v0.15.0, which you can read here.

I know this doesn't really fix the issue, but if it's a blocker because these tests fail, you could use the filtering to skip these specific stories for now. We'll get to this at some point! Thanks for your patience <3