search

Automattic / woocommerce-payments

Accept payments via credit card. Manage transactions within WordPress.
https://wordpress.org/plugins/woocommerce-payments/
Other
174 stars 69 forks source link

Introduce dedicated smoke tests suite #5643

Open achyuthajoy opened 1 year ago

achyuthajoy commented 1 year ago

Description

With https://github.com/Automattic/woocommerce-payments/pull/5586, we have introduced building a plugin zip file and running E2E - Pull Request tests against that package when release, pre-release or build-zip workflows are triggered. However, instead of the existing E2E tests, we should add a dedicated smoke testing suite.

Acceptance criteria

The smoke tests suite should run when:

haszari commented 1 year ago

@achyuthajoy What does "smoke tests" mean in this case?

To my mind, smoke tests are manual tests of very high level functions – e.g. checkout page renders payment form, payment can be completed.

If we are proposing automated tests to assist with release, I think we should use a different term, so we're really clear about what's different about this suite of tests. E.g. if they are automated, why do we not run all tests? Or all tests on critical flows? etc.

haszari commented 6 months ago

However, instead of the existing E2E tests, we should add a dedicated smoke testing suite.

@achyuthajoy What does "smoke tests" mean in this case?

To my mind, smoke tests are manual tests of very high level functions – e.g. checkout page renders payment form, payment can be completed.

If we are proposing automated tests to assist with release, I think we should use a different term, so we're really clear about what's different about this suite of tests. E.g. if they are automated, why do we not run all tests? Or all tests on critical flows? etc.

Totally agree with @vbelolapotkov here. We need to define the why of this different test suite.

I have voted 👍 based on the assumption that we want:

Perhaps we can rename this issue, one of the following: