search

reg-viz / storycap

A Storybook Addon, Save the screenshot image of your stories :camera: via puppeteer.
https://www.npmjs.com/package/storycap
MIT License
712 stars 89 forks source link

Provide documentation how Storycap it is different to Storyshots-Puppeteer #199

Open raDiesle opened 4 years ago

raDiesle commented 4 years ago

Hello, how does Storycap stands next to Storyshots-Puppeteer and what are the differences?

It would be nice to add some information to the README.md of the project. I would be especially interested in, if storycap can provide the same results but improved performance by parallelization of executing the image tests

yhatt commented 4 years ago

+1. Our team is comparing between storyshots-puppeteer and storycap but I cannot recognize the difference in essential features from an official storyshots-puppeteer addon. IMO if these aim the same concept, I prefer the addon provided from Storybook officially.

raDiesle commented 4 years ago

I can tell you out of my detailed comparison and checks: dont go with the official one, its 50 times slower and has much less features than reg-viz

yhatt commented 4 years ago

Then do you think storycap can run E2E tests using Puppeteer? As far as I know, storyshots-puppeteer can do and works well, but storycap has not supported. Our team's comparison was focusing to exactly that, and it's also a primary reason why we have taken the official addon.

storycap is definitely designed for reg-viz, visual regression test. So I think to mention this fact in README will help people who is blinded by the speediness of storycap in spite of purposing E2E test.

raDiesle commented 4 years ago

In our project, we decided not to mix assertions with snapshot testing. That is why I do believe to keep separation of concerns is where storycap shines over storyshots capabilities. storycap approach is technology independant and its more clear for developers.

Such as for a reason: when a process is correct from ui screens, we can expect the backend wrote the correct data to database. ( For full e2e tests, we setup other test suites with other technologies in other projects to cover that. Maybe you should consider to do the same. - If in your setup you want to cover full e2e verification and it cannot be verified by ui only, I see three possible solutions for you:

  1. keep storyshots solution with mixed assertions for backend verification in same test definitions
  2. split/reproduce backend assertions from ui tests ( and storycap) and run those with another testrunner than the ones from storycap for ui verification.
  3. create a ui page, which dummy prints the backend state and let storybook define it as a page, which will be available as snapshot test, and no need for assertion

In general, In the .stories definition, the storybook pages can connect to mocked data or real data - here you are flexible. In current project we have growing number of 140 screenshots, which takes in total 7 minutes to test with storyshots, where storycap will take around 30 seconds.

yhatt commented 4 years ago

I agree with your think in the long view. BTAIM testing architecture would be different depending on projects. My working project was starting from small, and have empowered Jest + Storybook ecosystem gradually along with growing the project. Its result is storyshots-puppeteer.

Simply I'm just not yet as dissatisfied to take screenshots in storyshots-puppeteer + jest-image-snapshot. I'm figuring out the timing when I really need Storycap. I think it's similar to the think of "When should I use Redux?".

Well, let's back to the subject of this issue. I think the most point of difference to storyshots-puppeteer is "its purpose" and have a worth to mention in the documentation to both:

[Off-topic] Why our team need story-based assertions: My project have to provide a bunch of components for general purpose that have tested interactions well. In addition, (sadly) we still have to support for IE. storyshots-puppeteer provides a basic idea to integrate with anything with Storybook + Jest. So we can use StoryShots as the platform of cross-browser test by developing plugin based on that. We made it on us own, but there is an open-source solution in https://github.com/andrcuns/addon-storyshots-wdio.
raDiesle commented 4 years ago

I would add to highlight, that storycap focuses to resolve specific problem most efficient, with a ready to go setup, e.g. meant for unit testing components in isolation with fast speed, that is why it only supports chromium headless driver browser.

Quramy commented 4 years ago

@yhatt Thanks . I agree your comparison summary .

We, reg-viz, intend the following separation of concern.

So ,for example, users can choice Storycap and jest-image-snapshot if they want. (I have not tried this)

petermikitsh commented 3 years ago

One bright spot of @storybook/addon-storyshots-puppeteer is that it takes screenshots and it does the diffing. After reading this issue, and https://github.com/reg-viz/storycap/issues/279, it's clear you're supposed to pair storycap with reg-suit. I may have missed it, but I haven't seen an example of how this is done. A migration guide example from addon-storyshots-puppeteer to storycap + reg-suit could help drive adoption for a complete alternative.

As a user, I'm not too discerning about which tool does the diffing. An opinionated approach is completely fine to me; the core issue driving me to abandon addon-storyshots-puppeteer is its slow performance. If these un-opinionated tools were wrapped up into a higher abstraction, migration could be simplified.