Start to automate test?

[lemnis]

Seeing that multiple packages are available to automate screen reader testing, wouldn't it be great to start supporting in combination with manual tests? (e.g. @guidepup/guidepup, @accesslint/VoiceOver)

Me trying stuff out:

const { test, expect } = require("@playwright/test");
const { VoiceOver } = require("@accesslint/voiceover");

const voiceOver = new VoiceOver({ log: false });

const og = [
  {
    command: "next_item",
    css_target: "#text1",
    before: {
      mode: "auto",
      virtual_location: "before target",
      focus_location: "before target",
    },
    after: "target",
    output: '"My birthday, M M D D Y Y Y Y, edit text"',
    results: [
      {
        feature_id: "aria/aria-placeholder_attribute",
        feature_assertion_id: "convey_placeholder_labelled_no_value",
        applied_to: "html/input(type-text)_element",
        result: "pass",
      },
    ],
  },
  {
    command: "next_focusable_item",
    css_target: "#text1",
    before: {
      mode: "auto",
      virtual_location: "before target",
      focus_location: "before target",
    },
    after: "target",
    output: '"My birthday, M M D D Y Y Y Y, edit text"',
    results: [
      {
        feature_id: "aria/aria-placeholder_attribute",
        feature_assertion_id: "convey_placeholder_labelled_no_value",
        applied_to: "html/input(type-text)_element",
        result: "pass",
      },
    ],
  },
  {
    command: "next_item",
    css_target: "#text2",
    before: {
      mode: "auto",
      virtual_location: "before target",
      focus_location: "before target",
    },
    after: "target",
    output: '"Your birthday, edit text"',
    results: [
      {
        feature_id: "aria/aria-placeholder_attribute",
        feature_assertion_id: "contribute_to_accessible_name",
        applied_to: "html/input(type-text)_element",
        result: "pass",
      },
    ],
  },
  {
    command: "next_focusable_item",
    css_target: "#text2",
    before: {
      mode: "auto",
      virtual_location: "before target",
      focus_location: "before target",
    },
    after: "target",
    output: '"Your birthday, edit text"',
    results: [
      {
        feature_id: "aria/aria-placeholder_attribute",
        feature_assertion_id: "contribute_to_accessible_name",
        applied_to: "html/input(type-text)_element",
        result: "pass",
      },
    ],
  },
  {
    command: "next_item",
    css_target: "#text3",
    before: {
      mode: "auto",
      virtual_location: "before target",
      focus_location: "before target",
    },
    after: "target",
    output: '"His birthday, M M D D Y Y Y Y, edit text"',
    results: [
      {
        feature_id: "aria/aria-placeholder_attribute",
        feature_assertion_id: "no_aria_placeholder_if_html_placeholder_present",
        applied_to: "html/input(type-text)_element",
        result: "pass",
      },
    ],
  },
  {
    command: "next_focusable_item",
    css_target: "#text3",
    before: {
      mode: "auto",
      virtual_location: "before target",
      focus_location: "before target",
    },
    after: "target",
    output: '"His birthday, M M D D Y Y Y Y, edit text"',
    results: [
      {
        feature_id: "aria/aria-placeholder_attribute",
        feature_assertion_id: "no_aria_placeholder_if_html_placeholder_present",
        applied_to: "html/input(type-text)_element",
        result: "pass",
      },
    ],
  },
  {
    command: "next_item",
    css_target: "#text4",
    before: {
      mode: "auto",
      virtual_location: "before target",
      focus_location: "before target",
    },
    after: "target",
    output: '"12 25 dash 2000, content selected, her birthday, edit text"',
    results: [
      {
        feature_id: "aria/aria-placeholder_attribute",
        feature_assertion_id: "convey_placeholder_labelled_with_value",
        applied_to: "html/input(type-text)_element",
        result: "fail",
      },
    ],
  },
  {
    command: "next_focusable_item",
    css_target: "#text4",
    before: {
      mode: "auto",
      virtual_location: "before target",
      focus_location: "before target",
    },
    after: "target",
    output: '"12 25 dash 2000, content selected, her birthday, edit text"',
    results: [
      {
        feature_id: "aria/aria-placeholder_attribute",
        feature_assertion_id: "convey_placeholder_labelled_with_value",
        applied_to: "html/input(type-text)_element",
        result: "fail",
      },
    ],
  },
];

const result = og.map((i) => ({ ...i }));

test("I can navigate the Playwright website using VoiceOver", async ({
  page,
}) => {
  test.slow();

  // Navigate to Playwright website 🎉
  await page.goto(
    "https://a11ysupport.io/tests/html/aria/aria-placeholder.html",
    {
      headless: false,
      waitUntil: "domcontentloaded",
    }
  );

  await voiceOver.launch();

  let i = 0;

  await voiceOver.advance({
    target: {
      text: "My birthday MM-DD-YYYY",
      role: "edit text",
    },
    steps: 10,
  });
  result[0].output = await voiceOver.lastPhrase();
  await page.keyboard.down("Shift");
  await page.keyboard.press("Tab");
  await page.keyboard.up("Shift");
  await page.keyboard.press("Tab");
  result[1].output = await voiceOver.lastPhrase();

  await voiceOver.advance({
    target: {
      text: "Your birthday",
      role: "edit text",
    },
    steps: 10,
  });
  result[2].output = await voiceOver.lastPhrase();
  await page.keyboard.down("Shift");
  await page.keyboard.press("Tab");
  await page.keyboard.up("Shift");
  await page.keyboard.press("Tab");
  result[3].output = await voiceOver.lastPhrase();

  await voiceOver.advance({
    target: {
      text: "His birthday",
      role: "edit text",
    },
    steps: 10,
  });
  result[4].output = await voiceOver.lastPhrase();
  await page.keyboard.down("Shift");
  await page.keyboard.press("Tab");
  await page.keyboard.up("Shift");
  await page.keyboard.press("Tab");
  result[5].output = await voiceOver.lastPhrase();

  await voiceOver.advance({
    target: {
      text: "Her birthday",
      role: "edit text",
    },
    steps: 10,
  });
  result[6].output = await voiceOver.lastPhrase();
  await page.keyboard.down("Shift");
  await page.keyboard.press("Tab");
  await page.keyboard.up("Shift");
  await page.keyboard.press("Tab");
  result[7].output = await voiceOver.lastPhrase();

  expect(page.url()).toBe(
    "https://a11ysupport.io/tests/html/aria/aria-placeholder.html"
  );
  expect(result).toStrictEqual(og);
});

Output

lisa@lisas-macbook-air vo-playwright % npx playwright test

Running 1 test using 1 worker

  ✘  example.spec.js:164:1 › I can navigate the Playwright website using VoiceOver (8s)

  1) example.spec.js:164:1 › I can navigate the Playwright website using VoiceOver =================

    Error: expect(received).toStrictEqual(expected) // deep equality

    - Expected  - 8
    + Received  + 8

    @@ -6,11 +6,11 @@
            "mode": "auto",
            "virtual_location": "before target",
          },
          "command": "next_item",
          "css_target": "#text1",
    -     "output": "\"My birthday, M M D D Y Y Y Y, edit text\"",
    +     "output": "My birthday MM-DD-YYYY edit text",
          "results": Array [
            Object {
              "applied_to": "html/input(type-text)_element",
              "feature_assertion_id": "convey_placeholder_labelled_no_value",
              "feature_id": "aria/aria-placeholder_attribute",
    @@ -25,11 +25,11 @@
            "mode": "auto",
            "virtual_location": "before target",
          },
          "command": "next_focusable_item",
          "css_target": "#text1",
    -     "output": "\"My birthday, M M D D Y Y Y Y, edit text\"",
    +     "output": "My birthday MM-DD-YYYY edit text",
          "results": Array [
            Object {
              "applied_to": "html/input(type-text)_element",
              "feature_assertion_id": "convey_placeholder_labelled_no_value",
              "feature_id": "aria/aria-placeholder_attribute",
    @@ -44,11 +44,11 @@
            "mode": "auto",
            "virtual_location": "before target",
          },
          "command": "next_item",
          "css_target": "#text2",
    -     "output": "\"Your birthday, edit text\"",
    +     "output": "Your birthday edit text",
          "results": Array [
            Object {
              "applied_to": "html/input(type-text)_element",
              "feature_assertion_id": "contribute_to_accessible_name",
              "feature_id": "aria/aria-placeholder_attribute",
    @@ -63,11 +63,11 @@
            "mode": "auto",
            "virtual_location": "before target",
          },
          "command": "next_focusable_item",
          "css_target": "#text2",
    -     "output": "\"Your birthday, edit text\"",
    +     "output": "Your birthday edit text",
          "results": Array [
            Object {
              "applied_to": "html/input(type-text)_element",
              "feature_assertion_id": "contribute_to_accessible_name",
              "feature_id": "aria/aria-placeholder_attribute",
    @@ -82,11 +82,11 @@
            "mode": "auto",
            "virtual_location": "before target",
          },
          "command": "next_item",
          "css_target": "#text3",
    -     "output": "\"His birthday, M M D D Y Y Y Y, edit text\"",
    +     "output": "His birthday MM-DD-YYYY edit text",
          "results": Array [
            Object {
              "applied_to": "html/input(type-text)_element",
              "feature_assertion_id": "no_aria_placeholder_if_html_placeholder_present",
              "feature_id": "aria/aria-placeholder_attribute",
    @@ -101,11 +101,11 @@
            "mode": "auto",
            "virtual_location": "before target",
          },
          "command": "next_focusable_item",
          "css_target": "#text3",
    -     "output": "\"His birthday, M M D D Y Y Y Y, edit text\"",
    +     "output": "His birthday MM-DD-YYYY edit text",
          "results": Array [
            Object {
              "applied_to": "html/input(type-text)_element",
              "feature_assertion_id": "no_aria_placeholder_if_html_placeholder_present",
              "feature_id": "aria/aria-placeholder_attribute",
    @@ -120,11 +120,11 @@
            "mode": "auto",
            "virtual_location": "before target",
          },
          "command": "next_item",
          "css_target": "#text4",
    -     "output": "\"12 25 dash 2000, content selected, her birthday, edit text\"",
    +     "output": "12-25-2000 Insertion at beginning of text. Her birthday edit text",
          "results": Array [
            Object {
              "applied_to": "html/input(type-text)_element",
              "feature_assertion_id": "convey_placeholder_labelled_with_value",
              "feature_id": "aria/aria-placeholder_attribute",
    @@ -139,11 +139,11 @@
            "mode": "auto",
            "virtual_location": "before target",
          },
          "command": "next_focusable_item",
          "css_target": "#text4",
    -     "output": "\"12 25 dash 2000, content selected, her birthday, edit text\"",
    +     "output": "12-25-2000 contents selected Her birthday edit text",
          "results": Array [
            Object {
              "applied_to": "html/input(type-text)_element",
              "feature_assertion_id": "convey_placeholder_labelled_with_value",
              "feature_id": "aria/aria-placeholder_attribute",

      239 |     "https://a11ysupport.io/tests/html/aria/aria-placeholder.html"
      240 |   );
    > 241 |   expect(result).toStrictEqual(og);
          |                  ^
      242 | });
      243 | // });
      244 | // });

        at /Users/lisa/dev/vo-playwright/example.spec.js:241:18

  1 failed
    example.spec.js:164:1 › I can navigate the Playwright website using VoiceOver ==================
lisa@lisas-macbook-air vo-playwright % npx playwright test

Running 1 test using 1 worker

  ✘  example.spec.js:164:1 › I can navigate the Playwright website using VoiceOver (8s)

  1) example.spec.js:164:1 › I can navigate the Playwright website using VoiceOver =================

    Error: expect(received).toStrictEqual(expected) // deep equality

    - Expected  - 8
    + Received  + 8

    @@ -6,11 +6,11 @@
            "mode": "auto",
            "virtual_location": "before target",
          },
          "command": "next_item",
          "css_target": "#text1",
    -     "output": "\"My birthday, M M D D Y Y Y Y, edit text\"",
    +     "output": "My birthday MM-DD-YYYY edit text",
          "results": Array [
            Object {
              "applied_to": "html/input(type-text)_element",
              "feature_assertion_id": "convey_placeholder_labelled_no_value",
              "feature_id": "aria/aria-placeholder_attribute",
    @@ -25,11 +25,11 @@
            "mode": "auto",
            "virtual_location": "before target",
          },
          "command": "next_focusable_item",
          "css_target": "#text1",
    -     "output": "\"My birthday, M M D D Y Y Y Y, edit text\"",
    +     "output": "My birthday MM-DD-YYYY edit text",
          "results": Array [
            Object {
              "applied_to": "html/input(type-text)_element",
              "feature_assertion_id": "convey_placeholder_labelled_no_value",
              "feature_id": "aria/aria-placeholder_attribute",
    @@ -44,11 +44,11 @@
            "mode": "auto",
            "virtual_location": "before target",
          },
          "command": "next_item",
          "css_target": "#text2",
    -     "output": "\"Your birthday, edit text\"",
    +     "output": "Your birthday edit text",
          "results": Array [
            Object {
              "applied_to": "html/input(type-text)_element",
              "feature_assertion_id": "contribute_to_accessible_name",
              "feature_id": "aria/aria-placeholder_attribute",
    @@ -63,11 +63,11 @@
            "mode": "auto",
            "virtual_location": "before target",
          },
          "command": "next_focusable_item",
          "css_target": "#text2",
    -     "output": "\"Your birthday, edit text\"",
    +     "output": "Your birthday edit text",
          "results": Array [
            Object {
              "applied_to": "html/input(type-text)_element",
              "feature_assertion_id": "contribute_to_accessible_name",
              "feature_id": "aria/aria-placeholder_attribute",
    @@ -82,11 +82,11 @@
            "mode": "auto",
            "virtual_location": "before target",
          },
          "command": "next_item",
          "css_target": "#text3",
    -     "output": "\"His birthday, M M D D Y Y Y Y, edit text\"",
    +     "output": "His birthday MM-DD-YYYY edit text",
          "results": Array [
            Object {
              "applied_to": "html/input(type-text)_element",
              "feature_assertion_id": "no_aria_placeholder_if_html_placeholder_present",
              "feature_id": "aria/aria-placeholder_attribute",
    @@ -101,11 +101,11 @@
            "mode": "auto",
            "virtual_location": "before target",
          },
          "command": "next_focusable_item",
          "css_target": "#text3",
    -     "output": "\"His birthday, M M D D Y Y Y Y, edit text\"",
    +     "output": "His birthday MM-DD-YYYY edit text",
          "results": Array [
            Object {
              "applied_to": "html/input(type-text)_element",
              "feature_assertion_id": "no_aria_placeholder_if_html_placeholder_present",
              "feature_id": "aria/aria-placeholder_attribute",
    @@ -120,11 +120,11 @@
            "mode": "auto",
            "virtual_location": "before target",
          },
          "command": "next_item",
          "css_target": "#text4",
    -     "output": "\"12 25 dash 2000, content selected, her birthday, edit text\"",
    +     "output": "12-25-2000 Insertion at beginning of text. Her birthday edit text",
          "results": Array [
            Object {
              "applied_to": "html/input(type-text)_element",
              "feature_assertion_id": "convey_placeholder_labelled_with_value",
              "feature_id": "aria/aria-placeholder_attribute",
    @@ -139,11 +139,11 @@
            "mode": "auto",
            "virtual_location": "before target",
          },
          "command": "next_focusable_item",
          "css_target": "#text4",
    -     "output": "\"12 25 dash 2000, content selected, her birthday, edit text\"",
    +     "output": "12-25-2000 contents selected Her birthday edit text",
          "results": Array [
            Object {
              "applied_to": "html/input(type-text)_element",
              "feature_assertion_id": "convey_placeholder_labelled_with_value",
              "feature_id": "aria/aria-placeholder_attribute",

      239 |     "https://a11ysupport.io/tests/html/aria/aria-placeholder.html"
      240 |   );
    > 241 |   expect(result).toStrictEqual(og);
          |                  ^
      242 | });
      243 | // });
      244 | // });

        at /Users/lisa/dev/vo-playwright/example.spec.js:241:18

  1 failed
    example.spec.js:164:1 › I can navigate the Playwright website using VoiceOver ==================
lisa@lisas-macbook-air vo-playwright % 

https://user-images.githubusercontent.com/3999815/170102579-e2737535-1cd7-4d91-bdfd-77ad11530cd1.mov

What are thoughts? How valuable do you think it would be and which things should we take into account?

[lemnis]

A small round up of my own personal thoughts:

• It would be nice to automate tests, partially or fully, even if it just detect if a new AT version or browser version results in a different output. And as such detect if a new manual test is needed.
• I am not sure if the current JSON format would work great with automating tests.
• It is feels like a huge project for a single person, and as such I should not even think about doing this 😇

[mfairchild365]

@lemnis Excellent idea and proof of concept! I've explored this in the past and agree that automating the testing is essential to keeping results fresh. It's also not a great fit for the current test format, a lot of work to set up (initially), and in my experience, a lot of the screen reader testing packages are not robust enough to accomplish the goal reliably.

I'd like to point you in the direction of ARIA-AT, specifically the automation workstream. I'm a co-chair of that group, and our goal is to approach screen reader testing in a more standard way, with the involvement of AT developers - and automate all of it. It's a project that will take quite a lot of effort and time, but I'm hopeful that it will eventually make a11ysupport.io obsolete. Might you be interested in helping?

[lemnis]

I would keep an eye out on the ARIA-AT and happy to help where I can.

FYI, I also have been playing a bit with playwright, as all browser developer tools contain info about accessibility. For which tools are currently out to get exposed a11y data, it feels the most stable and reliable. It can be an interesting data point before it gets sent to the AT. Bugs reported improving playwright https://github.com/microsoft/playwright/issues/14332 & https://github.com/microsoft/playwright/issues/14347).