Research and implement a testing strategy for types provided through `@carbon/react`

[tay1orjones]

Background

Once https://github.com/carbon-design-system/carbon/issues/12513 is fully complete, it would be beneficial if we had a way to be more confident in future changes to the component types. Specifically to ensure that types provided through @carbon/react follow the established Semantic Versioning guidelines for the package.

[!IMPORTANT]
Types will continue to be provided on an as-is basis and remain exempt from semver for the forseeable future as we establish a more robust framework around maintaining types. Completing the work in this issue does not mean that we're able to bound types to semver.

Click to expand details

https://github.com/carbon-design-system/carbon/blob/b8701c9578019c1f4fe3f92d4ef0e1a23139e9bd/docs/guides/versioning.md#L75-L86

Potential approach

For component proptypes we have a very helpful setup where the proptypes for all components are collected and snapshotted into a single file, PublicAPI-test.js.snap. As changes to proptypes are made, the snapshot requires being updated, which shows up in the PR diff. This makes it really clear what's changing and makes it much easier to spot potential issues. We could attempt to mirror this setup in some way to provide more explicit visibility on every PR when a potentially semver-incompatible change is made to component types. The prominent solution for this would be to use https://api-extractor.com/

Conversely, rather than snapshotting these, some other options to research include:

• tsd - Seems like a popular option. The programmatic api in particular seems like a way we could automate this testing instead of having to write individual tests for each component.
• https://github.com/JoshuaKGoldberg/eslint-plugin-expect-type
• https://github.com/mmkal/expect-type
• https://github.com/TypeStrong/ts-expect
• Use Vitest, it supports testing types. This could have lots of additional concerns as it would be adding a whole new framework to our testing strategy.
• https://github.com/tstyche/tstyche

Additional concerns

• We should try to avoid having to manually write and maintain tests, instead favoring automation where possible
• Would be awesome to have as much coverage as possible, including testing generics, even though they may only be rarely used in the codebase.
• ...

Related resources

• https://www.totaltypescript.com/how-to-test-your-types
• https://github.com/microsoft/DefinitelyTyped-tools/tree/main/packages/dtslint#just-looking-for-a-type-testing-tool
• https://github.com/JoshuaKGoldberg/eslint-plugin-expect-type?tab=readme-ov-file#references
• An interesting comparison of projects: https://github.com/mmkal/expect-type?tab=readme-ov-file#similar-projects

- [ ] Testing types: Research and explore available options
- [ ] Testing types: Provide a written summary of the research, indicating a preferred approach
- [ ] Testing types: Socialize options with other Carbon library maintainers
- [ ] Testing types: Implement the chosen solution, including adding it to the CI workflows

[mrazauskas]

@tay1orjones It is really nice to see TSTyche in the list. I am its author. Long story short, I was writing and contributing type tests to Jest repo for some time. At the moment there are more than 1000 assertions. Working at this scale was a pain with the available tooling. That is why I wrote TSTyche.

For instance, at large scale there was a need to have describe() and test() helpers with .skip and .only. That’s a must for deducing an issue or building new feature. Also we wanted to test on different versions of TypeScript easily. TSTyche can do all this and much more. All Jest’s type test now are ran by TSTyche.

Differently from all other type testing tools, TSTyche is wrapping around TypeScript’s language service. It allows per file queries instead of working with the whole TS program. And more. For example, another feature I am thinking about is an API which makes possible testing in memory generate files, i.e. passing arbitrary text to the compiler and asking for diagnostics (and/or processing type test assertions).

If this sounds like the automation solution you are looking for, I am more than interested to collaborate. The implementation on TSTyche side is rather trivial. The fs host object must be passed to the language service factory. Obviously, that can be a virtual fs which is generating test files on demand.

To be honest, I never had a chance to work with component type tests. And that sounds even more interesting ;D

[mrazauskas]

Earlier today TSTyche 2.1 shipped with new .toAcceptProps() matcher that is designed to test well.. prop types of JSX components! Thanks for inspiration.

[tay1orjones]

@mrazauskas Thanks for popping in here! The features you listed are definitely intriguing and something we'll consider when we take up this work. toAcceptProps in particular could be really valuable.