Report Data Deletion

[jesdaigle]

Currently in the sandbox we have the ability to delete report data when a test plan has been updated. It is currently solely at the admin's discretion however we need to think about what the logic should be and evaluate all the cases in which a test plan is updated and what of those constitutes a breaking change. Example, if the update is to fix a typo in the instructions for instance, does that really mean that the report data is in valid?

[s3ththompson]

I wanted to expand on @jesdaigle's post with a little more detail about the current behavior of the app. Here is an example scenario.

An admin adds the test plan "Select Only Combobox Example" for NVDA and Firefox to the test queue. This entry corresponds to a particular version from the w3c/aria-at repo, usually the latest version.

Testers begin to record results. For example, two testers may submit draft results for all of the 21 tests in "Select Only Combobox Example".

During the course of testing, an admin realizes that there is an extra assertion in test 10, a change to the instructions in test 11, and a typo in test 12. The test author makes changes to "Select Only Combobox Example" in the w3c/aria-at to address the issues.

An admin decides to update the version of the test plan that is currently in the test queue (to incorporate the latest changes). The app analyzes the existing draft results and realizes that 3 results for each tester (those which corresponded to tests 10, 11, and 12) must be discarded when the test plan is updated, since they correspond to a now-outdated version of those tests.

The testers return to the app to submit new draft results for tests 10, 11, and 12. The results they previously saved for the other tests are still there.

Note: this scenario only applies to results which are in the test queue as drafts. If any results have been published to the reports page, they are unaffected by adding newer versions of a test plan to the test queue. (And indeed multiple versions of published results happily coexist as a way to capture the historical change of results over time).

The open questions about this scenario are:

• Are certain changes (e.g. the typo above) small enough that they might not be "breaking changes" (in other words, they might not need to invalidate previously-saved results)?
  • Right now, the app in not be discriminatory about the type of change within a test.
  • What types of distinctions could be made automatically? What types of distinctions would need to be made by the test plan admin based on guidance in the working mode?

Looking forward to discussing these questions in the next CG meeting. cc @samuelhshaw @jscholes

(We also have some thoughts about retaining outdated test plans in an archive (rather than editing in-place or discarding outdated draft results), but are still working on a proposal.)