google / site-kit-wp

Site Kit is a one-stop solution for WordPress users to use everything Google has to offer to make them successful on the web.
https://sitekit.withgoogle.com
Apache License 2.0
1.25k stars 290 forks source link

Fix issues where the VRT test fails in CI before the tests have all completed #9528

Open techanvil opened 2 days ago

techanvil commented 2 days ago

Bug Description

Unfortunately, the VRT workflow is still failing before all the tests have completed with some regularity.

Examining some recent failures, nearly all of them are failing with following error in one of two permutations:

      COMMAND | Command "test" ended with an error after [450.482s]
      COMMAND | Error: Failed to launch the browser process!

Some of these are followed by the following line, while others are not:

                [1016/142508.119062:ERROR:bus.cc(399)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory

In the sample of test runs there's also one instance where it's failing without the "Failed to launch the browser" error:

      COMMAND | Command "test" ended with an error after [551.962s]
      COMMAND | [object Object]

Example test failures:

Steps to reproduce

Screenshots

Failed to launch browser, with dbus error:

Image

Failed to launch browser, without dbus error:

Image

Other error:

Image

Additional Context

See the previous issue #9406 for the most recently applied fix, and the Additional Context section on https://github.com/google/site-kit-wp/issues/9211 for some relevant avenues that could yet be explored for the fix.

@benbowler please can you update this section if there are avenues you explored from the issue linked above that we don't need to revisit?


Do not alter or remove anything below. The following sections will be managed by moderators only.

Acceptance criteria

Note, additional issues should be raised if needed to follow up on separate avenues of investigation.

Implementation Brief

Test Coverage

QA Brief

Changelog entry

benbowler commented 2 days ago

Thanks @techanvil, I noticed the reappearance of the browser failed to launch error. It was a little surprising, as in the batch testing the previous fix appeared to prevent this error. I can take a look.

techanvil commented 2 days ago

Thanks @benbowler, yes it's a bit of a shame to see this pop up again as it did seem to have improved for a bit. Thanks for taking a look!