Debugging ATTAs

[halindrome]

In the ATK ATTA, @Joanmarie has included an argument processor that takes 3 arguments: https://github.com/Spec-Ops/web-platform-tests/blob/419a77e21fe985e1d1327dc38e42a81d4ba6145b/wai-aria/tools/atta-atk-atspi.py#L822-L827

I propose that we require that all ATTAs provide for a --debug option that, when enabled, logs all messages and AT API data so that it is possible to more easily analyze what the test is asking for, what is being seen, and how that is being interpreted by the ATTA.

[joanmarie]

Related to this, I noticed that some of my debugging is already appearing in the json response, e.g.:

{
  "results": [
    {
      "test": "/wai-aria/example-cell-manual.html",
      "subtests": [
        {
          "name": "Sample A11Y Test for cell role",
          "status": "FAIL",
          "message": "assert_true: (Found: Accessible, Collection, Component, EditableText, Hypertext, Text);  expected true got false"
        }
      ],
      "status": "OK",
      "message": null
    },
    {
      "test": "/wai-aria/example-main-manual.html",
      "subtests": [
        {
          "name": "Sample A11Y Test for main",
          "status": "FAIL",
          "message": "assert_true: (Found: ROLE_SECTION);  expected true got false"
        }
      ],
      "status": "OK",
      "message": null
    }
  ]
}

In the context of my current console spew, the "Found:" makes sense. In the json above, less so. Thus I plan to make the output make better sense in the response my ATTA sends. Beyond that.... Other suggestions for what you like to see?

[halindrome]

FIrst, FWIW, we do not need to use assert_true. For a complete list of the interfaces available to us that we could use, see testdev.spec-ops.io/wpt/resources/docs/api.md

Second, I like that some information is showing up in the JSON. In the fullness of time, show up in the reports that are generated for each platform. You can see an example of what that might look like at http://testdev.spec-ops.io/test-results/annotation-protocol/all.html - Note that the show/hide messages button in the table header will toggle display of the "messages" from tests that failed.

Finally, as to what I would like to see when debugging is enabled. I dont really know what is available from the ATK API (nor any other AT API). Presumably for every type of thing we are looking for and for the name(s) of those things, we could dump whatever information is found. Is that possible? Is it overwhelming? Is the AT "tree" for a given node just a TON of data? I assume that it is.

[joanmarie]

What would you like to have available? :) Assume I can do it until I tell you otherwise.

Tree isn't a ton of data given the size of our test cases. For regular content, yeah, the tree can be a bit overwhelming. I'll come up with something later (today?) to dump the tree.

[joanmarie]

This just in from the Department of I Don't Know Why I Didn't Think of This Before*: --dry-run

Given the changes we decided to make to the assertions for the testable statements, I did a non-trivial amount of changing to my ATTA (not yet in a branch) which I wanted to test independent of the work you are doing. So I created --dry-run for me, to just dump everything we might want to know about the specified element. The result is quite cool -- and in this context, it is much more useful than a tree dump (imho). I'll finish it up later today.

*People don't usually let me name their departments. Not sure why....

[joanmarie]

Here's sample output: https://gist.github.com/Joanmarie/96c7e722d269d062097eb6d9f49a2fbf

That change is now in my branch (and like the commit message states, tests now spit up when you're not doing a dry run. Once we have raw test files that reflect the new world order, I'll merge your changes into my branch and debug whatever issues remain in my ATTA.

In the meantime, lemme know what you think about the sample output as the fix for this issue. You may find it helpful as for the assertion stuff you're working on (or not....).