Race Condition when Running Reference Test in Chrome

When a Web Platform Test reference test is requested, the wptrunner tool performs the following actions:

1. Navigates to the document via the "go" command
2. Injects a "wait" script via the "execute async script" command
3. Captures the state of the screen using the "screenshot" command

(source)

(Note: Although the "wait" script in step 2 is authored to defer until the document's readyState is "ready", the delay introduced by the "HTTP command" protocol means that practically speaking, this event always occurs prior to the evaluation of the script. In other words, the "wait" script always signals completion synchronously.)

Despite this, external image resources are not always rendered when the screenshot is captured in step 3.

This project demonstrates the problem in a simplified environment.

To Run

1. Install the following system dependencies:

   • Python 2.7
   • The PIP tool for installing Python packages
   • Google Chrome
   • the Chromedriver binary

2. Install the Python dependencies using the following command:

   $ pip install -r requirements.txt

3. Run the Chromedriver binary, bound to port 4444:

   $ chromedriver --port=4444

4. Run the main.py Python script located in the root of this project:

   $ python main.py

   The script navigates to the same simple document, capturing a screen shot after each navigation. It halts when any two screenshots do not match, ending by saving the screenshots to a file named results.html.

   The runtime behavior of the script may be configured through command-line options. Execute the command python main.py --help for more information.

Analysis

The following images are examples of screenshots captured when running this script:

While it appears that in the second screenshot, the image failed to load at all, a few details indicate that the document is in some transitional state:

• The value of the documentElement's scrollHeight property reflects the height of the document with a fully-rendered image. This is true even when the clientHeight value is captured before the screenshot.
• If another screenshot is captured after an arbitrary delay, the image is rendered as expected

Because this behavior is somewhat counter-intuitive, and because this is not exhibited by Firefox, I am reaching out to the Chromium development team for insight about the internals of the engine and how this indeterminacy might be avoided.

Environment Information

This behavior was observed on a system matching the following description:

• Operating system: Ubuntu 16.04
• Achitecture: x86_64
• Google Chrome: 58.0.3026.3 (Official Build) dev (64-bit)
• ChromeDriver: 2.27.440175 (9bc1d90b8bfa4dd181fbbf769a5eb5e575574320)

License

Copyright 2017 Mike Pennisi under the GNU General Public License v3.0