woocommerce / qit-cli

A Testing Platform for WordPress Plugins and Themes
https://qit.woo.com
15 stars 2 forks source link

Self-tests for Custom Tests #162

Open Luc45 opened 4 months ago

Luc45 commented 4 months ago

This issue is to keep track of what the self-tests of the Custom Tests covers, and to keep track of what scenarios we still need to cover.

Current Test coverage

Environment

Behavior Covered by Tests
Env up with default settings :heavy_check_mark:
Env up with PHP and WP versions passed as parameters :heavy_check_mark:
Env up with PHP and WP versions in config file :heavy_check_mark:
Env up with PHP and WP versions in config file overridden by parameters :heavy_check_mark:
Env up with PHP and WP versions in config file overridden by config file override :x:
Env up with Object Cache :heavy_check_mark:
Env up with Additional Volumes :heavy_check_mark:
Env up with WordPress stable version :heavy_check_mark:
Env up with WordPress rc version :x:
Env up with WordPress WPOrg tag version :heavy_check_mark:
Env up with WooCommerce stable version :heavy_check_mark:
Env up with WooCommerce rc version :heavy_check_mark:
Env up with WooCommerce git tag version :x:
Env up with additional PHP Extensions :heavy_check_mark:
Env up with additional theme(s) :heavy_check_mark:

Tagging

Behavior Covered by Tests
Upload test tag :heavy_check_mark:
List test tags :heavy_check_mark:
List test tags of a specific extension :heavy_check_mark:
Delete test tag :heavy_check_mark:
Upload, delete and list test tags of extension you don't own :x:

E2E Test

Behavior Covered by Tests
Scaffold a basic E2E test :heavy_check_mark:
Run a scaffolded E2E test locally :heavy_check_mark:
Tag a scaffolded test and run it from the tag :heavy_check_mark:
Run a test and assert it fails if SUT failed to activate :heavy_check_mark:
Run an E2E test with multiple tags :heavy_check_mark:
Run an E2E test with theme as SUT :heavy_check_mark:
Run an E2E test with plugin as SUT and theme with test or bootstrap actions :x:
Run an E2E test with snapshot test for visual regression :heavy_check_mark:

Compatibility Test

Behavior Covered by Tests
Run an E2E test and activate additional plugin sourced from slug :heavy_check_mark:
Run an E2E test with bootstrap additional plugin sourced from slug :heavy_check_mark:
Run an E2E test with testing additional plugin sourced from slug :heavy_check_mark:
Run an E2E test with multiple tags and additional plugin :heavy_check_mark:
Run an E2E test with same additional plugin having multiple tags :heavy_check_mark:
Run a scaffolded E2E test with additional plugin sourced from local directory/zip file/URL (activate/bootstrap/test actions), both from runtime parameters, config files and config file override :x:
Run a test with multiple plugins, some of them with tests, some with local directory and others from tags :x: