If a user has a Storybook configured in their development environment, specifically for the latest major releases (e.g., 7.6 and 8.0), they automatically have the option to enable the Visual testing addon to run visual tests on their components locally. However, if they are working with Chromatic E2E integration (i.e., Playwright, Cypress), they can enable the Visual tests addon to run visual tests on their components when that should not be the case, whether intentionally or not. This is something that should be avoided as much as possible, as this can lead to unnecessary visual tests being run on components that are not intended to be tested, polluting the build screen with unnecessary builds, and depleting the allowed snapshots for the user's account.
It would be in our best interest to either:
Filter the projects during the onboarding workflow to prevent this issue
If the above is impossible, prevent the build from going through, as this is not the correct type, and showcase a message in the addon's panel to reflect this fact
If none of the above is possible, I'm open to discussing this further as a potential solution.
For context, I've opened an internal ticket in our internal system to track this so that both teams are aware of the issue and can work on a solution to prevent it from happening.
cc @ghengeveld , @thafryer
To reproduce
Create a project with the framework of choice
Enable Storybook
Add an E2E solution to the project and the Chromatic counterpart
Run tests
Run Storybook and enable Storybook's Visual testing addon
Link it and run a build
Environment
Tests were performed with Storybook 7.6 but can be replicated in the latest stable Storybook version.
Describe the bug
If a user has a Storybook configured in their development environment, specifically for the latest major releases (e.g., 7.6 and 8.0), they automatically have the option to enable the Visual testing addon to run visual tests on their components locally. However, if they are working with Chromatic E2E integration (i.e., Playwright, Cypress), they can enable the Visual tests addon to run visual tests on their components when that should not be the case, whether intentionally or not. This is something that should be avoided as much as possible, as this can lead to unnecessary visual tests being run on components that are not intended to be tested, polluting the build screen with unnecessary builds, and depleting the allowed snapshots for the user's account.
It would be in our best interest to either:
If none of the above is possible, I'm open to discussing this further as a potential solution.
For context, I've opened an internal ticket in our internal system to track this so that both teams are aware of the issue and can work on a solution to prevent it from happening.
cc @ghengeveld , @thafryer
To reproduce
Environment
Tests were performed with Storybook 7.6 but can be replicated in the latest stable Storybook version.