Add automated tests using Playwright

[simonw]

I'm using pytest-playwright. Here's the first test I figured out which seems to do the right thing.

from playwright.sync_api import Page, expect

def test_initial_load(page: Page):
    page.goto("https://lite.datasette.io/")
    loading = page.locator("#loading-indicator")
    expect(loading).to_have_css("display", "block")
    # Give it up to 60s to finish loading
    expect(loading).to_have_css("display", "none", timeout=60 * 1000)

    # Should load faster the second time thanks to cache
    page.goto("https://lite.datasette.io/")
    expect(loading).to_have_css("display", "none", timeout=20 * 1000)

[simonw]

The expect() helper adds the magic where it keeps polling until the element becomes available. That's not necessary all the time though - this test does some direct assertions against elements on the page that it knows have loaded:

     assert [el.inner_text() for el in page.query_selector_all("h2")] == [
        "fixtures",
        "content",
    ]

[simonw]

Figured out how to use fixtures and run a test that navigates to the /fixtures page and submits the form to execute a custom query:

from playwright.sync_api import Browser, Page, expect
import pytest

@pytest.fixture(scope="module")
def dslite(browser: Browser) -> Page:
    page = browser.new_page()
    page.goto("https://lite.datasette.io/")
    loading = page.locator("#loading-indicator")
    expect(loading).to_have_css("display", "block")
    # Give it up to 60s to finish loading
    expect(loading).to_have_css("display", "none", timeout=60 * 1000)
    return page

def test_initial_load(dslite: Page):
    expect(dslite.locator("#loading-indicator")).to_have_css("display", "none")

def test_has_two_databases(dslite: Page):
    assert [el.inner_text() for el in dslite.query_selector_all("h2")] == [
        "fixtures",
        "content",
    ]

def test_navigate_to_database(dslite: Page):
    h2 = dslite.query_selector("h2")
    assert h2.inner_text() == "fixtures"
    h2.query_selector("a").click()
    expect(dslite).to_have_title("fixtures")
    dslite.query_selector("textarea#sql-editor").fill(
        "SELECT * FROM no_primary_key limit 1"
    )
    dslite.query_selector("input[type=submit]").click()
    expect(dslite).to_have_title("fixtures: SELECT * FROM no_primary_key limit 1")
    table = dslite.query_selector("table.rows-and-columns")
    table_html = "".join(table.inner_html().split())
    assert table_html == (
        '<thead><tr><thclass="col-content"scope="col">content</th>'
        '<thclass="col-a"scope="col">a</th><thclass="col-b"scope="col">b</th>'
        '<thclass="col-c"scope="col">c</th></tr></thead><tbody><tr>'
        '<tdclass="col-content">1</td><tdclass="col-a">a1</td>'
        '<tdclass="col-b">b1</td><tdclass="col-c">c1</td></tr></tbody>'
    )

[simonw]

Next challenge: serve localhost with the current state of the application, rather than running all of the tests directly against https://lite.datasette.io/

This is a bit tricky, because it means I need to spin up a localhost web server for the duration of the pytest run.

Some relevant code:

• https://github.com/microsoft/playwright-python/blob/v1.24.0/tests/server.py is the mechanism Playwright Python itself uses to run a test server - it's built on top of Twisted
• https://pytest-httpserver.readthedocs.io/ is a Pytest tool for running simulated servers - it can be programmed to return specific content from specific URLs but doesn't have a utility mechanism for "just serve these static files"
• https://github.com/ppmdo/pytest-simplehttpserver is closest to what I want - it runs a localhost static server during pytest execution, but it puts it on port 8000 and requires you to set the directory at runtime with pytest --simplehttpserver-directory /home/user/mock_website

I'm inspired by the code that it uses to spin up that server though - it runs Popen(...) to start a python -m http.server --directory x process here: https://github.com/ppmdo/pytest-simplehttpserver/blob/a82ad31912121c074ff1a76c4628a1c42c32b41b/src/pytest_simplehttpserver/simplehttpserver.py

And then in https://github.com/ppmdo/pytest-simplehttpserver/blob/a82ad31912121c074ff1a76c4628a1c42c32b41b/src/pytest_simplehttpserver/pytest_plugin.py#L17-L28 it starts that process, yields it from a fixture, then later calls server_process.terminate() and server_process.wait().

I'm going to try spinning my own fixture that does effectively the same thing.

[simonw]

My version of that fixture:

from subprocess import Popen, PIPE
import pathlib
import pytest
import time
from http.client import HTTPConnection

root = pathlib.Path(__file__).parent.parent.absolute()

@pytest.fixture(scope="module")
def static_server():
    process = Popen(
        ["python", "-m", "http.server", "8123", "--directory", root], stdout=PIPE
    )
    retries = 5
    while retries > 0:
        conn = HTTPConnection("localhost:8123")
        try:
            conn.request("HEAD", "/")
            response = conn.getresponse()
            if response is not None:
                yield process
                break
        except ConnectionRefusedError:
            time.sleep(1)
            retries -= 1

    if not retries:
        raise RuntimeError("Failed to start http server")
    else:
        process.terminate()
        process.wait()

[simonw]

I've been running the tests with:

pytest --headed

So I can watch the browser as they run.

[simonw]

I can model the GitHub workflow on this one I wrote that exercises Datasette using pyodide and Playwright: https://github.com/simonw/datasette/blob/0.62a1/.github/workflows/test-pyodide.yml and https://github.com/simonw/datasette/blob/0.62a1/test-in-pyodide-with-shot-scraper.sh

[simonw]

Next challenge: serve localhost with the current state of the application, rather than running all of the tests directly against https://lite.datasette.io/

Hah, turns out I had a TIL about this that I'd forgotten about! https://til.simonwillison.net/pytest/subprocess-server

I added the static server recipe from above to that existing TIL.

[simonw]

Write this up as a TIL: https://til.simonwillison.net/pytest/playwright-pytest