[RFC]: Create a testability contract that can be a part of semantic versioning

[stuckless]

Goal: For any given component provide a contract on how that component can be tested. It's a contract like an API contract that will remain consistent from build to build and ideally between major releases.

Semantic versioning is a good tool to convey that are you are not introducing breaking changes in minor and patch releases, and this works great for libraries since they are non-visual and provide a single entry point, the API.

• UI libraries are not like JavaScript libraries and should have a different kind of semantic versioning
• UI has a JavaScript API contract
• UI has a property based API contract (names and types)
• UI has a visual contract (padding around components, how the component looks, etc)
• UI has an interaction contact (what I can click, type into, etc)
• UI has a behaviour contract (what happens when I enter an invalid date)

Can you change the DOM?  Yes, absolutely, provided that it doesn't break your testability contract.  eg, if you say, to find a dialog you'll use role=dialog and title=Your Title but then you change the dom to use <dialog> element and no role, then that is a breaking change.  The testing contract is broken.

To have more flexibility around the component DOM, there are a couple of options: 1) define a set of component testing wrappers for a component.  ie, Dialog("title").click("Send") or Dialog("title").isPrimaryAction("Send").

• Provides a clean testing interface to testers
• Allows for a complete restructuring of components and DOM
• Component wrappers have the knowledge of how to locate components in the DOM
• Can be time-consuming to implement

2) define a testing interface as a set of locators.  ie, 

• ideally a programmatic set of interfaces, but could be a document that explains how to locate parts of a component as a contract. tests should be created to validate the testing interface

Regression fix contract

• Regressions are fixed in the version in which they are found.

For Example Button

• Find a button: role = button and text=TEXT
• Click a button: click( role = button and text =TEXT)
• verify a primary button: role=button and data-kind=primary
• verify a button is showing an icon: role=button and icon alt text=ABC
• Click an icon button: click(role=button and alt text=TEXT)

Dialog

• Find a dialog: role=dialog and text=Dialog Title
• Verify the label in a dialog primary button: Find Dialog; Find child role=button and text = Button Text and data-kind = primary
• Click a button in a dialog: Find Dialog and click(role=button and text=Text)

Dropdown

• Find a dropdown: datatest-id=dropdownid
• Verify an item is selected:  find dropdown; find item textcontent=text and has data-selected=true
• get a count of items in the dropdown: ??
• verify the order of items in a dropdown: ??
• select an item in a dropdown: click(textcontents=text)

When a component is created the testing contract should be created as well.  That contract should be validated in testing-library or cyrpress. The contract would help validate the components, and it would serve as documentation to consumers on how they can build resilient tests around the components that they consume.

Ideally Carbon would adopt the testing contract as well.

[davidicus]

I definitely think option 2 seems like a good idea and easier to maintain and implement. I will create an issue to track this effort.

I passed along this issue to the Carbon team to see what their thoughts are.

[JordanWSmith15]

need to submit a PR to show what I am asking for from the Carbon team

[bjornalm]

We should not underestimate the effort in achieving this for our more complex components. The testable location of a part of a component, i.e. a button somewhere in the table, could require:

• a custom configuration of the component (i.e. the props)
• a set of interactions to get to a specific state (not all states ca be reached by the configuration)
• a custom setup, or change, of the environment (e.g. browser window is small or made smaller)

I assume the above must also be part of the contract. Since we need to write tests to maintain the actual contract, perhaps we can somehow also export the code that delivers the part that should be tested (will be testing framework specific). Kind if like the wrapper suggestion, but less effort from ours side, since we need to write the tests anyway.

[bjornalm]

PR created for Carbon https://github.com/carbon-design-system/carbon/pull/9652

[davidicus]

Carbon says they will support the ability to add a data-testid on all top level components but will not be implementing that in their codebase. Closing this issue as we have completed the contract for our repo and got a conclusion from Carbon. If we want to discuss how we can further improve our testing stability let's open a new issue.