🧪 Testing: Establish a preference for small, self-contained mutation tests

[JoshuaKGoldberg]

Overview

This is something I'd wanted to start on ages ago, but never had the time and never got around to writing down...

Today, there are quite a few mutation tests that have a lot of stuff in them. Example: https://github.com/JoshuaKGoldberg/TypeStat/blob/16f76cf106033d5ca45da3eed82de3120aae4c08/test/cases/fixes/noImplicitAny/variableDeclarations/original.ts is just over 200 lines. Large mutation tests were handy when I was getting started on TypeStat many years ago and wanted to quickly iterate on many things at once.

But, the library is a bit more settled now, and these huge tests are unwieldy. It's hard to precisely check one thing at a time with them or search for where one thing is tested for. Example: to test noImplicitAny with variable declarations and Promise resolves, I'd probably temporarily comment out all but the last ~10 lines of the aforementioned variableDeclarations/original.ts file.

Proposal: let's...

1. Quick change: mention in .github/DEVELOPMENT.md a preference for small, self-contained mutation tests
2. Larger change: split up each of these larger original.ts files into families of more granular ones

WDYT @rubiesonthesky?

[rubiesonthesky]

I think it would be good to experiment with mutation test that does not actually need typestat.json and tsconfig.json in the disk. What I mean is that, you could write test somehow like this:

const { actualContent, expectedFilePath } = await runMutationTest(
    // code
    `
        const returnsStringAndNumber = (): number => {
            return BigInt(123n);
        };
    `,
    // typestat
    `
    {
        "fixes": {
            "incompleteTypes": true
        },
        "types": {
            "strictNullChecks": true
        }
    }
    `,
    // tsconfig
    `{
        "files": ["actual.ts"]
    }
    `

);
await expect(actualContent).toMatchFileSnapshot(expectedFilePath);

This would allow creating small tests for specific thing.

So I'm favor of this idea and there are some ways it could be done. :)

[JoshuaKGoldberg]

+1, I would like that very much. https://github.com/JoshuaKGoldberg/TypeStat/discussions/1491#discussioncomment-8973322 mentions long-term hopes around not being tied into a file system.

[rubiesonthesky]

This seems interesting! https://sdorra.dev/posts/2024-02-12-vitest-tmpdir

With writing temp file, we could save the mutations to temp file and then read that. This would work for smaller tests.