Custom Tests

[Luc45]

Custom Tests

This PR allows Partner developers of the Woo marketplace to submit their own E2E tests to QIT, and to run the E2E tests that we make available to them.

Testing instructions

• Spin up the Manager on the custom tests branch
• Go to _tests/custom_tests and run composer install
• Copy _tests/custom_tests/.env.sample to _tests/custom_tests/.env
• Fill out the .env with the necessary values
• Run ./vendor/bin/paratest, it should take around 5 minutes and pass
• This provides the coverage outlined here

Manual testing instructions

Explore the new commands added, which are:

• qit scaffold:e2e
• qit run:e2e
• qit tag:upload
• qit tag:delete
• qit tag:list

Scaffold a test, upload it, list it and run it:

• qit scaffold:e2e /tmp/foo
• qit tag:upload automatewoo /tmp/foo
• qit tag:list
• qit run:e2e automatewoo --plugin woocommerce:activate

Publish another test tag

• Now scaffold another test and publish it for automatewoo-birthdays
• Run it with qit run:e2e automatewoo-birthdays --plugin woocommerce:activate --plugin automatewoo:test
• You can also do this using a qit-env.yml or qit-env.json files

There are many options, try out with --php_version 8.3, different WP and Woo versions, object cache, etc.

Also try out --codegen and --ui.

The self-tests is pretty self-explanatory too, if you're interested. Check this out for instance:

    public function test_runs_scaffolded_e2e() {
        $output = qit( [
                'run:e2e',
                'automatewoo',
                $this->scaffold_test(),
                '--plugin',
                'woocommerce:activate',
            ]
        );

        $output = $this->normalize_scaffolded_test_run_output( $output );

        $this->assertMatchesNormalizedSnapshot( $output );
    }

Documentation:

The documentation contains example commands that we can use to try more scenarios: https://qit.woo.com/docs/custom-tests/introduction

[Luc45]

Current Test coverage

Behavior	Covered by Tests
Env up with default settings	:heavy_check_mark:
Env up with PHP and WP versions passed as parameters	:heavy_check_mark:
Env up with PHP and WP versions in config file	:heavy_check_mark:
Env up with PHP and WP versions in config file overridden by parameters	:heavy_check_mark:
Env up with PHP and WP versions in config file overridden by config file override	:x:
Env up with Object Cache	:heavy_check_mark:
Env up with Additional Volumes	:x:
Env up with WordPress version (stable/rc/tag)	:x:
Env up with additional PHP Extensions	:x:
Env up with additional theme(s)	:x:
Upload test tag	:heavy_check_mark:
List test tags	:heavy_check_mark:
List test tags of a specific extension	:heavy_check_mark:
Delete test tag	:heavy_check_mark:
Upload, delete and list test tags of extension you don't own	:x:
Scaffold a basic E2E test	:heavy_check_mark:
Run a scaffolded E2E test locally	:heavy_check_mark:
Tag a scaffolded test and run it from the tag	:heavy_check_mark:
Run a test and assert it fails if SUT failed to activate	:heavy_check_mark:
Run a scaffolded E2E test with additional plugin sourced from slug (activate action)	:heavy_check_mark:
Run a scaffolded E2E test with additional plugin sourced from local directory/zip file/URL (activate/bootstrap/test actions), both from runtime parameters, config files and config file override	:x:
Run a test with multiple plugins, some of them with tests, some with local directory and others from tags	:x:

[zhongruige]

Test coverage April 26th

Nice!!

[Luc45]

I've moved the coverage table to a dedicated, long-lived issue

[rcstr]

Hi @Luc45 - I have tested both PRs and noticed some issues while running the different commands:

./vendor/bin/paratest 🔴

``` ParaTest v7.4.3 upon PHPUnit 11.1.3 by Sebastian Bergmann and contributors. Processes: 12 Runtime: PHP 8.3.0 Configuration: /Users/rommel/Projects/qit-cli/_tests/custom_tests/phpunit.xml.dist ..F..FFFFFFFFFF............S.. 30 / 30 (100%) Time: 06:09.393, Memory: 10.00 MB There were 2 PHPUnit test runner warnings: 1) Exception in third-party event subscriber: Failed to remove directory "/Users/rommel/Projects/qit-cli/_tests/custom_tests/tmp/tmp_qit_config-qit_custom_tests_66393a99f0ab5": rmdir(/Users/rommel/Projects/qit-cli/_tests/custom_tests/tmp/._0ED): Directory not empty #0 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/symfony/filesystem/Filesystem.php(153): Symfony\Component\Filesystem\Filesystem::doRemove(Array, false) #1 /Users/rommel/Projects/qit-cli/_tests/custom_tests/QITTestExtension.php(274): Symfony\Component\Filesystem\Filesystem->remove(Array) #2 /Users/rommel/Projects/qit-cli/_tests/custom_tests/QITTestExtension.php(237): QITTestFinish::delete_temp_environment() #3 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/phpunit/phpunit/src/Event/Dispatcher/DirectDispatcher.php(102): QITTestFinish->notify(Object(PHPUnit\Event\TestRunner\ExecutionFinished)) #4 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/phpunit/phpunit/src/Event/Dispatcher/DeferringDispatcher.php(45): PHPUnit\Event\DirectDispatcher->dispatch(Object(PHPUnit\Event\TestRunner\ExecutionFinished)) #5 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/phpunit/phpunit/src/Event/Emitter/DispatchingEmitter.php(1143): PHPUnit\Event\DeferringDispatcher->dispatch(Object(PHPUnit\Event\TestRunner\ExecutionFinished)) #6 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/brianium/paratest/src/WrapperRunner/ApplicationForWrapperWorker.php(224): PHPUnit\Event\DispatchingEmitter->testRunnerExecutionFinished() #7 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/brianium/paratest/bin/phpunit-wrapper.php(70): ParaTest\WrapperRunner\ApplicationForWrapperWorker->end() #8 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/brianium/paratest/bin/phpunit-wrapper.php(80): {closure}() #9 {main} 2) Exception in third-party event subscriber: Failed to remove directory "/Users/rommel/Projects/qit-cli/_tests/custom_tests/tmp/tmp_qit_config-qit_custom_tests_66393a99f183f": rmdir(/Users/rommel/Projects/qit-cli/_tests/custom_tests/tmp/._YUT): Directory not empty #0 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/symfony/filesystem/Filesystem.php(153): Symfony\Component\Filesystem\Filesystem::doRemove(Array, false) #1 /Users/rommel/Projects/qit-cli/_tests/custom_tests/QITTestExtension.php(274): Symfony\Component\Filesystem\Filesystem->remove(Array) #2 /Users/rommel/Projects/qit-cli/_tests/custom_tests/QITTestExtension.php(237): QITTestFinish::delete_temp_environment() #3 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/phpunit/phpunit/src/Event/Dispatcher/DirectDispatcher.php(102): QITTestFinish->notify(Object(PHPUnit\Event\TestRunner\ExecutionFinished)) #4 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/phpunit/phpunit/src/Event/Dispatcher/DeferringDispatcher.php(45): PHPUnit\Event\DirectDispatcher->dispatch(Object(PHPUnit\Event\TestRunner\ExecutionFinished)) #5 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/phpunit/phpunit/src/Event/Emitter/DispatchingEmitter.php(1143): PHPUnit\Event\DeferringDispatcher->dispatch(Object(PHPUnit\Event\TestRunner\ExecutionFinished)) #6 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/brianium/paratest/src/WrapperRunner/ApplicationForWrapperWorker.php(224): PHPUnit\Event\DispatchingEmitter->testRunnerExecutionFinished() #7 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/brianium/paratest/bin/phpunit-wrapper.php(70): ParaTest\WrapperRunner\ApplicationForWrapperWorker->end() #8 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/brianium/paratest/bin/phpunit-wrapper.php(80): {closure}() #9 {main} -- There were 11 failures: 1) CompatibilityTest::test_sut_and_activate_additional Failed asserting that two strings are equal. Snapshots can be updated by passing `-d --update-snapshots` through PHPUnit's CLI arguments. --- Expected +++ Actual @@ @@ Bootstrapping automatewoo /qit/tests/e2e/automatewoo/local/bootstrap/bootstrap.php\n Bootstrapping automatewoo /qit/tests/e2e/automatewoo/local/bootstrap/bootstrap.sh\n Running E2E Tests\n +\n +What's Next?\n +View summary of image vulnerabilities and recommendations → docker scout quickview automattic/qit-runner-playwright:1.42.1\n Running 1 test using 1 worker\n [1/1] [automatewoo-local] › automatewoo/local/example.spec.js:9:5 › I can see my plugin menu\n 1 passed (TIME)\n /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/Drivers/TextDriver.php:27 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/Snapshot.php:55 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:199 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:57 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:104 /Users/rommel/Projects/qit-cli/_tests/custom_tests/tests/Traits/SnapshotHelpers.php:44 /Users/rommel/Projects/qit-cli/_tests/custom_tests/tests/CompatibilityTest.php:22 2) CompatibilityTest::test_sut_and_bootstrap_additional Failed asserting that two strings are equal. Snapshots can be updated by passing `-d --update-snapshots` through PHPUnit's CLI arguments. --- Expected +++ Actual @@ @@ Bootstrapping automatewoo /qit/tests/e2e/automatewoo/local/bootstrap/bootstrap.php\n Bootstrapping automatewoo /qit/tests/e2e/automatewoo/local/bootstrap/bootstrap.sh\n Running E2E Tests\n +\n +What's Next?\n +View summary of image vulnerabilities and recommendations → docker scout quickview automattic/qit-runner-playwright:1.42.1\n Running 1 test using 1 worker\n [1/1] [automatewoo-local] › automatewoo/local/example.spec.js:9:5 › I can see my plugin menu\n 1 passed (TIME)\n /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/Drivers/TextDriver.php:27 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/Snapshot.php:55 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:199 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:57 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:104 /Users/rommel/Projects/qit-cli/_tests/custom_tests/tests/Traits/SnapshotHelpers.php:44 /Users/rommel/Projects/qit-cli/_tests/custom_tests/tests/CompatibilityTest.php:49 3) CompatibilityTest::test_sut_and_test_additional Failed asserting that two strings are equal. Snapshots can be updated by passing `-d --update-snapshots` through PHPUnit's CLI arguments. --- Expected +++ Actual @@ @@ Bootstrapping automatewoo /qit/tests/e2e/automatewoo/local/bootstrap/bootstrap.sh\n Exporting DB\n Running E2E Tests\n +\n +What's Next?\n +View summary of image vulnerabilities and recommendations → docker scout quickview automattic/qit-runner-playwright:1.42.1\n Running 3 tests using 1 worker\n [1/3] [woocommerce-self-test-sut-and-test-additional] › woocommerce/self-test-sut-and-test-additional/example.spec.js:9:5 › I can see my plugin menu\n [2/3] [db-import] › db-import.js:5:5 › Database Import\n /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/Drivers/TextDriver.php:27 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/Snapshot.php:55 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:199 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:57 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:104 /Users/rommel/Projects/qit-cli/_tests/custom_tests/tests/Traits/SnapshotHelpers.php:44 /Users/rommel/Projects/qit-cli/_tests/custom_tests/tests/CompatibilityTest.php:76 4) CompatibilityTest::test_multiple_tags_and_multiple_plugins_with_multiple_tags Failed asserting that two strings are equal. Snapshots can be updated by passing `-d --update-snapshots` through PHPUnit's CLI arguments. --- Expected +++ Actual @@ @@ Bootstrapping automatewoo /qit/tests/e2e/automatewoo/self-test-multiple-test-tags-another/bootstrap/bootstrap.sh\n Exporting DB\n Running E2E Tests\n +\n +What's Next?\n +View summary of image vulnerabilities and recommendations → docker scout quickview automattic/qit-runner-playwright:1.42.1\n Running 7 tests using 1 worker\n [1/7] [woocommerce-self-test-multiple-test-tags] › woocommerce/self-test-multiple-test-tags/example.spec.js:9:5 › I can see my plugin menu\n [2/7] [db-import] › db-import.js:5:5 › Database Import\n /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/Drivers/TextDriver.php:27 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/Snapshot.php:55 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:199 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:57 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:104 /Users/rommel/Projects/qit-cli/_tests/custom_tests/tests/Traits/SnapshotHelpers.php:44 /Users/rommel/Projects/qit-cli/_tests/custom_tests/tests/CompatibilityTest.php:118 5) RunE2ETest::test_runs_scaffolded_e2e Failed asserting that two strings are equal. Snapshots can be updated by passing `-d --update-snapshots` through PHPUnit's CLI arguments. --- Expected +++ Actual @@ @@ Bootstrapping automatewoo /qit/tests/e2e/automatewoo/local/bootstrap/bootstrap.php\n Bootstrapping automatewoo /qit/tests/e2e/automatewoo/local/bootstrap/bootstrap.sh\n Running E2E Tests\n +\n +What's Next?\n +View summary of image vulnerabilities and recommendations → docker scout quickview automattic/qit-runner-playwright:1.42.1\n Running 1 test using 1 worker\n [1/1] [automatewoo-local] › automatewoo/local/example.spec.js:9:5 › I can see my plugin menu\n 1 passed (TIME)\n /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/Drivers/TextDriver.php:27 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/Snapshot.php:55 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:199 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:57 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:104 /Users/rommel/Projects/qit-cli/_tests/custom_tests/tests/Traits/SnapshotHelpers.php:44 /Users/rommel/Projects/qit-cli/_tests/custom_tests/tests/RunE2ETest.php:37 6) RunE2ETest::test_tag_and_run_test Failed asserting that two strings are equal. Snapshots can be updated by passing `-d --update-snapshots` through PHPUnit's CLI arguments. --- Expected +++ Actual @@ @@ Bootstrapping automatewoo /qit/tests/e2e/automatewoo/self-test-tag-and-run/bootstrap/bootstrap.php\n Bootstrapping automatewoo /qit/tests/e2e/automatewoo/self-test-tag-and-run/bootstrap/bootstrap.sh\n Running E2E Tests\n +\n +What's Next?\n +View summary of image vulnerabilities and recommendations → docker scout quickview automattic/qit-runner-playwright:1.42.1\n Running 1 test using 1 worker\n [1/1] [automatewoo-self-test-tag-and-run] › automatewoo/self-test-tag-and-run/example.spec.js:9:5 › I can see my plugin menu\n 1 passed (TIME)\n /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/Drivers/TextDriver.php:27 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/Snapshot.php:55 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:199 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:57 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:104 /Users/rommel/Projects/qit-cli/_tests/custom_tests/tests/Traits/SnapshotHelpers.php:44 /Users/rommel/Projects/qit-cli/_tests/custom_tests/tests/RunE2ETest.php:58 7) RunE2ETest::test_multiple_tags_and_run_tests Failed asserting that two strings are equal. Snapshots can be updated by passing `-d --update-snapshots` through PHPUnit's CLI arguments. --- Expected +++ Actual @@ @@ Bootstrapping automatewoo /qit/tests/e2e/automatewoo/self-test-multiple-test-tags-another/bootstrap/bootstrap.sh\n Exporting DB\n Running E2E Tests\n +\n +What's Next?\n +View summary of image vulnerabilities and recommendations → docker scout quickview automattic/qit-runner-playwright:1.42.1\n Running 3 tests using 1 worker\n [1/3] [automatewoo-self-test-multiple-test-tags] › automatewoo/self-test-multiple-test-tags/example.spec.js:9:5 › I can see my plugin menu\n [2/3] [db-import] › db-import.js:5:5 › Database Import\n /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/Drivers/TextDriver.php:27 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/Snapshot.php:55 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:199 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:57 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:104 /Users/rommel/Projects/qit-cli/_tests/custom_tests/tests/Traits/SnapshotHelpers.php:44 /Users/rommel/Projects/qit-cli/_tests/custom_tests/tests/RunE2ETest.php:86 8) RunE2ETest::test_theme_as_sut Failed asserting that two strings are equal. Snapshots can be updated by passing `-d --update-snapshots` through PHPUnit's CLI arguments. --- Expected +++ Actual @@ @@ Bootstrapping deli /qit/tests/e2e/deli/local/bootstrap/bootstrap.php\n Bootstrapping deli /qit/tests/e2e/deli/local/bootstrap/bootstrap.sh\n Running E2E Tests\n +\n +What's Next?\n +View summary of image vulnerabilities and recommendations → docker scout quickview automattic/qit-runner-playwright:1.42.1\n Running 2 tests using 1 worker\n [1/2] [deli-local] › deli/local/activate-theme.spec.js:4:5 › I can activate Deli\n [2/2] [deli-local] › deli/local/example.spec.js:9:5 › I can see my plugin menu\n /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/Drivers/TextDriver.php:27 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/Snapshot.php:55 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:199 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:57 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:104 /Users/rommel/Projects/qit-cli/_tests/custom_tests/tests/Traits/SnapshotHelpers.php:44 /Users/rommel/Projects/qit-cli/_tests/custom_tests/tests/RunE2ETest.php:124 9) RunE2ETest::test_run_with_snapshot Failed asserting that two strings are equal. Snapshots can be updated by passing `-d --update-snapshots` through PHPUnit's CLI arguments. --- Expected +++ Actual @@ @@ Bootstrapping deli /qit/tests/e2e/deli/local/bootstrap/bootstrap.php\n Bootstrapping deli /qit/tests/e2e/deli/local/bootstrap/bootstrap.sh\n Running E2E Tests\n +\n +What's Next?\n +View summary of image vulnerabilities and recommendations → docker scout quickview automattic/qit-runner-playwright:1.42.1\n Running 2 tests using 1 worker\n [1/2] [deli-local] › deli/local/activate-theme.spec.js:4:5 › I can activate Deli\n [deli-local] › deli/local/activate-theme.spec.js:4:5 › I can activate Deli\n /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/Drivers/TextDriver.php:27 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/Snapshot.php:55 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:199 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:57 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:104 /Users/rommel/Projects/qit-cli/_tests/custom_tests/tests/Traits/SnapshotHelpers.php:44 /Users/rommel/Projects/qit-cli/_tests/custom_tests/tests/RunE2ETest.php:164 10) RunE2ETest::test_playwright_config_override Failed asserting that two strings are equal. Snapshots can be updated by passing `-d --update-snapshots` through PHPUnit's CLI arguments. --- Expected +++ Actual @@ @@ Bootstrapping automatewoo /qit/tests/e2e/automatewoo/local/bootstrap/bootstrap.php\n Bootstrapping automatewoo /qit/tests/e2e/automatewoo/local/bootstrap/bootstrap.sh\n Running E2E Tests\n +\n +What's Next?\n +View summary of image vulnerabilities and recommendations → docker scout quickview automattic/qit-runner-playwright:1.42.1\n Running 1 test using 1 worker\n [1/1] [automatewoo-local] › automatewoo/local/example.spec.js:9:5 › I can see my plugin menu\n Slow test file: [automatewoo-local] › automatewoo/local/example.spec.js (s)\n /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/Drivers/TextDriver.php:27 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/Snapshot.php:55 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:199 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:57 /Users/rommel/Projects/qit-cli/_tests/custom_tests/vendor/spatie/phpunit-snapshot-assertions/src/MatchesSnapshots.php:104 /Users/rommel/Projects/qit-cli/_tests/custom_tests/tests/Traits/SnapshotHelpers.php:44 /Users/rommel/Projects/qit-cli/_tests/custom_tests/tests/RunE2ETest.php:205 11) ScaffoldTest::test_scaffold_e2e Failed asserting that two strings are equal. Snapshots can be updated by passing `-d --update-snapshots` through PHPUnit's CLI arguments. --- Expected +++ Actual @@ @@ -'/tmp-normalized/qit_scaffolded_e2e-NORMALIZED_ID/bootstrap/bootstrap.php: |\n +'/tmp-normalized/qit_scaffolded_e2e-66393ac232e17/bootstrap/bootstrap.php: |\n

./qit scaffold:e2e /tmp/foo ✅

Example E2E test generated in: /tmp/foo/
You can now run your first test with qit run:e2e <your_slug> "/tmp/foo/" --ui
You can start writing your tests with codegen: qit run:e2e --codegen
And when you are ready, you can publish your tests with qit test-tags:upload <your_slug> <path_to_test>
Read more about it on our documentation: https://qit.woo.com/docs/

./qit tag:upload automatewoo /tmp/foo 🔴

Uploading zip...
 0/1 [>---------------------------]   0%
In RequestBuilder.php line 182:

  Error: No route was found matching the URL and request method. (404)  

tag:upload <test_tag> <test_path> [<test_type>]

./qit tag:list 🟡

Command displayed the list at the end, but before printed many instances of the warning:

PHP Warning:  Undefined array key "type" in phar:///Users/rommel/Projects/qit-cli/qit/src/Commands/Tags/ListTestTagsCommand.php on line 46

Warning: Undefined array key "type" in phar:///Users/rommel/Projects/qit-cli/qit/src/Commands/Tags/ListTestTagsCommand.php on line 46
PHP Warning:  Undefined array key "type" in phar:///Users/rommel/Projects/qit-cli/qit/src/Commands/Tags/ListTestTagsCommand.php on line 46

./qit run:e2e automatewoo --plugin woocommerce:activate 🔴

  Command "run:e2e" is not defined.  

  Did you mean one of these?         
      run:activation                 
      run:compatibility              
      run:malware                    
      run:phpcompatibility           
      run:phpstan                    
      run:security                   
      run:woo-api                    
      run:woo-e2e                    
      scaffold:e2e                   

I checked via ./qit and "run:e2e is not in the list of commands

[Luc45]

Howdy @rcstr , can you please confirm that your local Manager is in the correct branch (the counterpart of this one on the Manager repo)? And when you do your manual test where run:e2e is not available, are you connected to the Local instance of that Manager?

[Luc45]

For the self-tests part (not manual testing), I can see some things:

Failed to remove directory "/Users/rommel/Projects/qit-cli/_tests/custom_tests/tmp/tmp_qit_config-qit_custom_tests_66393a99f183f": rmdir(/Users/rommel/Projects/qit-cli/_tests/custom_tests/tmp/._YUT): Directory not empty

This is probably a racing condition where we invoke the method to delete the tmp directory (recursively), but ._YUT still exists, which seems to be resource forks from macOS. Reference 1, Reference 2

macOS stores metadata and resource forks using a technique called AppleDouble, in which the data fork is written as one file, and the resource fork and metadata are written as an entirely separate file preceded by a "._" naming convention. For example: ExampleFile.psd would contain the data fork, and ._ExampleFile.psd would contain the resource fork and metadata.

I think this only happens on the context of self-tests. I'll add a small workaround (probably a 250ms wait before deleting) to see if it addresses it in the context of self-tests.

I won't change anything in the actual production flow, so if there's a bug there about this it will continue to happen on Mac and we can catch it.

As for this failure:

+\n
+What's Next?\n
+View summary of image vulnerabilities and recommendations → docker scout quickview automattic/qit-runner-playwright:1.42.1\n

From the snapshot tests it seems your instance of Docker adds some unexpected output to the terminal, which is nice to catch. I'll add a DOCKER_CLI_HINTS=false env var before running things to normalize this output.