woocommerce / woocommerce

A customizable, open-source ecommerce platform built on WordPress. Build any commerce solution you can imagine.
https://woocommerce.com
9.38k stars 10.76k forks source link

Playwright E2E: Add fixture data for tests #42303

Open imanish003 opened 1 year ago

imanish003 commented 1 year ago

This issue is result of following GH discussion: https://github.com/woocommerce/woocommerce-blocks/discussions/10099

As we strive to streamline our E2E testing setup, we propose to establish fixture-data.js as the central place for test data in our Playwright E2E tests.

The main objectives of this enhancement include:

  1. Transitioning Responsibilities from test-env-setup.sh to fixture-data.js: The test-env-setup.sh script currently performs several tasks including emptying the site, adding attributes, importing sample products, regenerating product lookup tables, creating pages and posts, and setting up payment methods. We aim to shift these responsibilities to fixture-data.js.
  2. Supplementing fixture-data.js with Additional Test Data: Alongside handling the tasks currently executed by test-env-setup.sh, fixture-data.js would also store the extra data needed for Playwright E2E tests.
  3. Establishing fixture-data.js as the Single Source of Truth for Test Setup: Upon migration, fixture-data.js would be used to set up the test environment and subsequently export a woocommerce.sql file. This .sql file would then be imported into the E2E test environment. This method will help reduce the test setup time significantly. The new .sql file would only need to be generated when our fixture data changes.

For more info: https://github.com/woocommerce/woocommerce-blocks/discussions/10099

### Tasks
dinhtungdu commented 1 year ago

Great write up @imanish003!. I agree that using JS to create test data will benefit us in the long run.

The new .sql file would only need to be generated when our fixture data changes.

I think there are more cases we may need to update the .sql file:

Those screen can break out tests, and we shouldn't check for their existence. Instead, we should ensure they won't show up (by regenerate the sql file in this case).

WunderBart commented 3 months ago

Upon migration, fixture-data.js would be used to set up the test environment and subsequently export a woocommerce.sql file. This .sql file would then be imported into the E2E test environment. This method will help reduce the test setup time significantly. The new .sql file would only need to be generated when our fixture data changes.

As a follow-up to https://github.com/woocommerce/woocommerce/pull/46125, we've briefly discussed using a pre-existing db dump instead of generating the environment each time. Since the import logic is already there, it should be straightforward to implement. Indeed it would make things much faster! 🙌

Regarding the test data transition to fixture-data.js, I think it's a good idea that would help write more deterministic tests, especially in situations where we need to know the exact number of products to assert against. This could be implemented in a Playwright Fixture convention, making the data available on-demand for each test without explicitly importing it.

IMO, the two issues above can also be dealt with separately as they have no direct dependency. What do you think, @imanish003?

cc: @gigitux

imanish003 commented 3 months ago

Yeah, I agree with you, @WunderBart. We can handle the two issues separately 💯