Consider using ts-semantic-testing

[SamPruden]

I've just published an early version prototype of an idea to help with writing tests for these kinds of projects, ts-semantic-testing. It's still very early days and potentially buggy and all subject to change, but I thought you might find it interesting to have a look at.

[KiaraGrouwstra]

TypeScript: turn run-time errors into compilation errors

ts-semantic-testing: turn compilation errors back into run-time errors

😅

Honestly, I love this -- it's heart-warming this experiment to do a type lib is already spawning tooling specific to this niche. I'll try it out.

[SamPruden]

Have fun, feel free to request things or contribute or whatever if you'd like.

I've actually been thinking I may completely change around the API though, to something roughly like this. Basically break it away from transforming the full test files and tests, and just provide a single block that is transformed and makes any semantic errors available for assertions, in a completely test framework independent way.

it("uses the new rough API idea", () => {
    tsst(() => {
        // Semantic errors within here are returned as strings
        // Only the contents of tsst are transformed
        type A = B;
    }).expect.toNotFindType("B");
});

So feel free to play around with it, and use it if it seems useful in its current state, but beware this is all very prototype right now and will change drastically.

[SamPruden]

Oh, and while it's designed for this niche, I do wonder if semantic tests may sometimes be useful in more general contexts. The organisation that comes with a testing framework may be a nicer experience than just looking out for compile errors in some situations.

[KiaraGrouwstra]

Features looked for to test types (also see https://github.com/sanctuary-js/sanctuary/issues/254#issuecomment-318912461):

• testing things type-check: originally filled by TypeScript
• testing for expected errors: tested within TS by its harness, outside of it originally by dts-lint
• protecting against any inference: tested within TS by its harness, outside of it originally filled by typings-checker
• getting version control friendly (= no line numbers) error logs for libs that still failed expectations: my typings-checker fork (used for Ramda typings)
• combined testing of types and run-time results: dts-jest (used in impending PR to Ramda typings)
• turning type errors into run-time errors to make named assertions: ts-semantic-testing

Seems to me that for this use-case you already had the right feature-set in mind.

I do wonder if semantic tests may sometimes be useful in more general contexts. The organisation that comes with a testing framework may be a nicer experience than just looking out for compile errors in some situations.

I'd be curious about its potential as well. Then again, normally people make tests to prevent regressions from sneaking in, whereas it's harder for compilation errors to sneak in without noticing upon compilation. In that sense, use-cases elsewhere might be more limited.

On tsst, so current API:

it("works with the number 0", () => {
  type A = ZeroOneToBoolean<0>;
});

What I had now in my attempt to test without names: the<number, 123>();

Imagined way the syntax could collapse:

it("is a number", the<number, 123>);

... Which would fail since generics for functions could only be provided upon calling them. They can be provided for types, but we want an expression-level function here. I guess one could be like

let x = null!; // short expression-level `never`, can be cast to anything
type TheF<T, V extends T> = () => V; // type-testing function type to lift to expression level
// options to invoke without calling function:
x as TheF<number, 123>;
<TheF<number, 123>>x;
// imagined test:
it("is a number", <TheF<
  number,
  123
>>x);

More of a thought experiment than pretty syntax. Also fails to yield a function expression on type error though which I suppose screws it up.

Do you have an example project you've actually already used it on by the way?

[KiaraGrouwstra]

This project is currently horrendously untested because I'm not sure what the best way to test this is.

Put your example snippet in some sub-folder 'repository' and have a script calling it on that folder? Maybe have your CLI propagate the status code of the testing thing so you'd know where the testing thing reported errors? You could e.g. have one test project like that that should pass, another that should result in some error so as to check whether it'd correctly report success/failure based on your transformed types, I dunno.

[KiaraGrouwstra]

issues with line-numbers

Hm. If the it blocks already describe what failed, then a work-around may be to limit it to one type per block or something.

I suppose if you could put your own example project for this up there it might serve simultaneously as both documentation on usage as well as a basis for testing 😅, right now running tsst doesn't seem to have it tell much as to what it's doing (is it outputting transformed files somewhere?), which is a bit confusing.

[SamPruden]

I'll work on getting a proper example out soon sure, that sounds good. And currently tsst is (theoretically, it was a 10 minute job that at least works locally for me) just running the compiler against your tsconfig.json in pretty much the same way just running tsc would, but only emitting files where the source file matches the glob.

It's literally just this:

    const projectPath = project || "./";
    const configPath = ts.findConfigFile(projectPath, ts.sys.fileExists);
    const basePath = path.resolve(path.dirname(configPath)); /*?*/
    const configReadResult = ts.readConfigFile(configPath, ts.sys.readFile);

    if (configReadResult.error) throw new Error("Error reading tsconfig.json");

    const config = ts.parseJsonConfigFileContent(configReadResult.config, ts.sys, basePath);
    const program = ts.createProgram(config.fileNames, config.options);
    const transformer = makeTransformer(program);

    program.getSourceFiles()
        .filter(file => minimatch(file.fileName, glob))
        .forEach(file => program.emit(file, undefined, undefined, undefined, {
            before: [transformer]
        }));

[KiaraGrouwstra]

Thanks! That's illuminating, let me try for a sec then.

[SimonMeskens]

This stuff is great, love to see progress in this space.

[KiaraGrouwstra]

Oh, I installed a local version figuring globals are an anti-pattern with version control, then erroneously tried running index.js instead of build.js, so that explains why mine silently terminated.

Edit: not outputting code, even with noEmit: false. I'll await the example mini-repo for a fool-proof baseline. :D

[SamPruden]

Oops, sorry. Yeah, I'm working on the example now, it's a little tricky to work out how to nicely integrate with an existing test framework because you have to do the transformations before the tests... I think the version I put up will be working but a mess, will have to find/build a better way at some point.

Just to be clear about running the build, tsst is a bin in the package. So if you've installed the package globally you should just be able to run tsst on the command line, or add this in the package.json and npm test.

"scripts": {
    "test": "tsst **/*.test.ts"
}

[SamPruden]

I've clearly explained that awfully, sorry. I'm prototyping/iterating significantly faster than I'm documenting.

[SamPruden]

Sorry, I ended up mostly having to deal with some other boring stuff today, didn't get as much time to work on this as I would have liked. https://github.com/TheOtherSamP/tsst-example-project is now a thing, I think it works.

I've been having a slight issue installing it, I've been intermittently getting a weird

npm ERR! enoent ENOENT: no such file or directory, rename 'D:\tsst-example-project-master\node_modules\.staging\tsst-36171f51\node_modules\@types\minimatch' -> 'D:\tsst-example-project-master\node_modules\.staging\@types\minimatch-b98dca64'

error when npm installing in the project directory. I think this is probably local to my system (I've had a few other problems with NPM, I need to clean that up) but if this happens to you, I've managed to get around it by manually npm install tsst, then npm install. No idea why.

[SamPruden]

Oh, and I completely changed the syntax around, surprise!

[KiaraGrouwstra]

Hm.

• I also get that npm issue. Thanks for the workaround.
• npm test then yields /usr/bin/env: ‘node\r’: No such file or directory for me (Ubuntu) due to the bin file using Windows breaklines (CRLF) over Unix ones (LF). For me VSCode shows an option to fix that in the bottom-left corner, allowing me to get to the next stop. For a long-term solution, it appears that Git automatically fixes CRLF to LF, while npm does not. I wonder if you could publish to npm again after setting your local editor to use Unix breaklines?
• got some Determining test suites to run...(node:5680) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 2): SyntaxError: Invalid regular expression: /js-tests\(.+)\.test\.js$/: Unmatched ')' apparently due to old global install, fixed when using local install. this is why I prefer local installs, less version mismatches. :b
• trying node_modules/.bin/tsst -b tests/*.test.ts -d ./tests terminates silently. minimatch appears to reject the global, file.filename /path/to/tsst-example-project/tests/operators.test.ts, glob somehow tests/operators.test.ts, minimatch then yielding false.
• not sure if it can make that js-tests dir for you, probably blocked by the above

Edit: incidentally '/path/to/tsst-example-project/tests/operators.test.ts'.includes('tests/operators.test.ts') and '/path/to/tsst-example-project/tests/operators.test.ts'.match(/tests\/operators\.test\.ts/) both work. I gues the weird part seems how my glob somehow got filled out. I thought it might be cuz it was the only matching file in there, but adding another matching file doesn't seem to prevent it from becoming tests/operators.test.ts.

[SamPruden]

Huh, I'm on Windows and don't have easy quick access to a Ubuntu environment for testing right now.

I should probably mention that I'm more of a .Net dev historically, I may be making silly mistakes here, I'm learning as I go, sorry.

not sure if it can make that js-tests dir for you, probably blocked by the above

Works for me.

I also get that npm issue. Thanks for the workaround.

Strange. I'll clear everything out and see if I can work out what's going on there. I don't think I've done anything particularly weird with NPM here, so not sure what's up with that.

I wonder if you could publish to npm again after setting your local editor to use Unix breaklines?

On it now, sorry. I should have had that set already, my bad.

trying node_modules/.bin/tsst -b tests/*.test.ts -d ./tests terminates silently.

I'll throw some logging in there quickly. I'm not immediately sure what's going on with that error either. Could that be a Unix thing with path handling somewhere? Apart from the NPM thing, just copying down the folder from github and running locally just works for me.

Aaand this already became a support job. 😂

[KiaraGrouwstra]

Nah it's good. I only switched over half a year ago when I realized my most-wanted Windows 10 had become better BashOnWindows support. We're all learning anyway.

I'll try to figure out what the heck is going on with that glob (edited previous post a bit but still on it). Other two should be fixed with LF and using local install.

[KiaraGrouwstra]

Sorry, seems quotes matter for me, my bad.

node_modules/.bin/tsst -b tests/*.test.ts -d ./tests
glob: 'tests/operators.test.ts'

node_modules/.bin/tsst -b "tests/*.test.ts" -d ./tests
glob: 'tests/*.test.ts'

So file name /path/to/tsst-example-project/tests/operators.test.ts, glob tests/*.test.ts, minimatch still reports false. Hm.

[KiaraGrouwstra]

My bad, swapping tests back to ** fixes the check.

[SamPruden]

Hmm, but I think that should have worked. I just threw the standard minimatch in there and assumed it would just work, but that does seem to be an issue, maybe I need to mess with options for it or something. I'm actually seeing the same thing here when I set that to tests/*.test.ts.

Nothing's ever simple. 😂

[KiaraGrouwstra]

Transpilation now works fine. Running jest still throws the regex error but that might be some version thing. Checking.

[SamPruden]

No, that's actually just a buggy regex I wrote. Your error is write, the confusing thing is that it works for me.

[KiaraGrouwstra]

Thanks, yeah, just saw it. Made a PR, probably at the same time as you fixing it haha.

[KiaraGrouwstra]

Uh, actually, that 'fix' is wrong. Would "js-tests\/(.+)\\.test\\.js$" work on Windows? I'm unsure due to the slash. Works for me.

[SamPruden]

I've been awake for about 30 hours and drank too many coffees, try not to judge me too harshly for how much of a mess I just made of the commit history. No, blame me, that's fair. 😂

[SamPruden]

Actually I'm going to disappear and sleep now before I accidentally destroy this whole project.

[KiaraGrouwstra]

Get some rest, you deserve it 😃, I'll try to see what I can do to incorporate this over here! Heck, I'm pretty sure @gcanti should be interested too!

[SamPruden]

Have fun, thanks for all the help. If I don't spend the whole day battling the weird NPM issue I think I'm going to try to get a basic expectation/assertion system in it tomorrow. And fix the millions of bugs I've probably sleepily put into it today.

[KiaraGrouwstra]

Yeah, I'll try a bit as well, expect another PR or two.

[KiaraGrouwstra]

Note that the setup of tsst-example-project allows one to have tests import types, one cannot import non-type TS (like the) with it.

I tried to chop the .d off the operators.d.ts file name, which makes the behavior of tsst magically change -- suddenly it dumps the resulting JS files not in js-tests, but in js-tests/tests.

Moreover though, as the regular TS compilation process is skipped, meaning it will transpile the import statements, which will then no longer have anything to import from.

Being able to import the from tsst might be one workaround there.

Also tried for a bit to see where I could take this:

tsst(() => {
  the<number, 123>();
}).expectToCompile();

... to something terser too (now I can actually test), but not much luck without touching transformers at least.

[KiaraGrouwstra]

It just occurred to me one part that might not become 'like JS' is types bugged to the extent of failing to terminate.

As the actual type evaluations don't actually occur separately at run-time, a non-terminating type would just make tsst hang.

This isn't actually a regression from what things have been like (compiling separate files commenting things binary-search style to find the culprits), just a point hard to improve on.

Edit: actually, since tsst prints the file names currently, even if you use a glob you'll still know which file hung. Progress!

[KiaraGrouwstra]

Currently trying this out with a few different testing frameworks. My priority there is pretty much stability of output for the purpose of diffing the output with version control:

• jest: kept randomizing order even with --runInBand, ugh. no-go.
• jasmine: errors have numbers :(
• mocha: errors have numbers :(, otherwise seems good. probably fixable with custom reporter but yeah.
• karma: didn't get it to run yet

Anyway... it's working! 😃😃😃

[KiaraGrouwstra]

Finished refactoring everything. Tests are more verbose this way but the superior error reporting was totally worth it.

Made some commits to tsst though I've mostly just been editing the same branch rather than separating everything, so excuse my git hygiene. If you don't like part of it, feel free to kick it out. :P

[KiaraGrouwstra]

It just occurred to me the issue of non-terminating types appears potentially solvable. If the types are transformed in isolation, then perhaps a wrapping the evaluation step in a promise/observable with time-out could serve to cancel the operation before it hits a stack overflow. Being able to incorporate potentially non-terminating types instead of having to comment them might go a long way toward being able to easily test against different TS versions such as to pinpoint potential regressions.

[SamPruden]

Sorry, I haven't been able to get to much of this so far today. Could you explain what you mean by "non-terminating types"? I don't think I've ever come across this phenomenon. It's possible to write types that just hang the compiler?

[KiaraGrouwstra]

It's possible to write types that just hang the compiler?

Yeah. It's a small difference in a sense. Iterating over tuples types for one involves recursion -- you continue until hitting a condition (finishing to iterate over the whole thing). If a TS glitch breaks the condition, it may never terminate, causing the compiler to blow up with a stack overflow.

Still affected by this are array's ListFrom, comp's Max and Min, and number's DivFloor and Modulo. This also affects tests for array's ConcatNumObjs / IncIndexNumbObj and list's ConcatLists, IncIndex and Prepend. Until yesterday there were actually a dozen more. As it stands there is no automated way to judge whether a type terminates.

[SamPruden]

Preview of what I'm sketching out and aiming for with the API:

tsst<NumBoolToBool<"0">>().is<false>();
tsst<number>().accepts<5>();
tsst<AFakeType>().fails().withMissingName("A");
tsst<NumBoolToBool<{}>>.fails().withConstraintMismatch(/*constraint position*/ 1);

The larger block syntax will probably remain as well, it's more versatile but more verbose. These assertions are relatively tricky to add as some of them (the positive ones, is for example) will need to be done in the transformation phase rather than at runtime. I want to do it this way for the superior versatility and better error messages though.

[SamPruden]

Yeah. It's a small difference in a sense. Iterating over tuples types for one involves recursion

Huh, I haven't actually looked at type iteration yet so I haven't really come across that. If the compiler blows up I think we're stuck, the best I could do is catch that from the build script. The transformer isn't "evaluating" the types, it's just reading out the same "diagnostics" from the language service that the IDE is. This will behave in pretty much the same way VSCode behaves for those types, I imagine.

[KiaraGrouwstra]

withMissingName

Having to code in the different errors seems relatively labor-intensive on your end while also making for a more complex API 😅, so unless you have demand for it yourself, I'd suggest not to go full out in that direction just yet while the potential target audience of your lib can be counted on one hand. The is / accepts look like a pretty awesome step forward! Note that 95% of my cases are just in the is category, so yeah. I don't have that many cases intended to fail yet I guess.

If the compiler blows up I think we're stuck

Yeah, I'm thinking it might require a timeout to prevent it from reaching that. Plain promises lack cancelation, but using RxJS e.g. it'd become say let type = await Observable.defer(getType).timeout(200, 'Timeout').toPromise(). The annoying bit there seems that it'd necessitate refactoring the whole thing to become async that way...

Edit:

I haven't actually looked at type iteration yet

Many of my types use recursion 😅, and I believe the same held true for typelevel-ts as well.

[SamPruden]

I'd suggest not to go full out in that direction just yet

Yeah, I basically agree. I'm more designing around having those be possible. I'd also have a generic method on .fails() that accepts a message string or regex to match against. Things like withMissingName() would just be shortcuts for a few of the common regexes. Once that infrastructure is in place more could then be added per demand. Note that .fails() already asserts a failure, the others just add optional extra specificity.

let type = await Observable.defer(getType).timeout(200, 'Timeout').toPromise()

I don't really understand where this would go in the process, but I'll have to wait until I've looked over the iteration stuff to get an understanding of that. Do you have a good little reproduction for that somewhere?

[KiaraGrouwstra]

tsst api

Fair enough. If you're interested in TS errors, you may wanna check their list; if you wanted to make something generic in that direction you might just consider generating it from that. That could probably make for some function catchError(code, ...args), such that to catch e.g. this random excessive stack error you could then be like withExcessiveStack = (s1?: string | RegExp | null, s2?: string | RegExp | null) => catchError(2321, s1, s2) or... something.

[SamPruden]

Oh, that's really useful, I probably could auto generate from that list, and maybe partially use the codes too as they're probably more reliable than the messages. Good find/thought, thanks.

[KiaraGrouwstra]

recursion

Take Length for one. It starts at 0, checks if your tuple has that, until it doesn't. At that point it knows your tuple's length. Sounds silly compared to JS where you can just check length, but you can't use that for types. Or well, you'd just get number.

As for still failing ones, well, the currently commented types I mentioned here. Uncomment any of them and tsc / tsst will blow up.

I don't really understand where this would go in the process

That getType should stand for our () => program.emit(file, undefined, undefined, undefined, { before: [transformer] }).

[KiaraGrouwstra]

with currying, hopefully it could just become withExcessiveStack = catchError(2321). heck, make that a R.map (using Ramda) over { withExcessiveStack: 2321 } // add more then convert. look mom, no custom code!

[KiaraGrouwstra]

I pushed a commit based on the timeout idea, which slightly improves on the situation for non-terminating types by skipping files on timeout, rather than waiting for a minute then having tsst itself blow up with a stack overflow. If only they'd been evaluated on a case-by-case basis perhaps this could just yield a failing test for that type rather than having the entire file not pass the transform. It's a step forward at least I guess.

[KiaraGrouwstra]

I managed to use a sed regex to remove error numbers from mocha output, so test output is version control friendly now. Time to do some version testing! 😃

[KiaraGrouwstra]

By the way, I can't install tsst from github as it errors with ENOENT: no such file or directory, chmod '/blah/node_modules/tsst/dist/build.js' -- the dist directory it's trying to access has not been generated yet if installing from git, so tsc may need to be run before it tries that. It seems if run through install scripts it's run from the parent project's path, so the command may need an explicitly specified relative path.

It seems the preinstall script triggers before that error, so I tried to have that run tsc. That seems too early though -- at that point it just errors there is no such directory yet. However, the other scripts seem not to trigger until after it's already errored about not seeing the build.js though, so I'm having trouble figuring it out.

One other issue: I'd like tsst to respect the TS version of the parent project, so I hope using peerDependencies (with version * for TS) could fix that. Dunno if this issue might be because I'm using an npm link-ed tsst though.

[SamPruden]

Sorry I disappeared for a bit, had a very busy period but I've got a bit of time to put towards this again now.

By the way, I can't install tsst from github

Good point, I hadn't considered installing direct from github when I set that up, I'll take a look at tidying that up and getting it working.

I'd like tsst to respect the TS version of the parent project

Yeah, this one is a little tricky. In principle I agree, but the problem lies in the fact that the compiler API isn't very stable yet. I might be able to get it working with a tested range of peerDependencies versions somehow but ultimately, until the API stabilises, versions of tsst will be tied to versions of typescript.

Speaking of compiler versions, the full features of the error assertion API (mainly the .is<T>() method) are waiting on #9879. That's an old proposal that even has a PR at #9943 but hasn't seen any movement for a while. That's essential for doing transform-time type assertions or checks. I'll be putting in a petition to get that revived at some point. Until then, we'll just have to fall back on the current style of The assertion, though it's far from ideal.

[KiaraGrouwstra]

Thanks for the links. In case they forgot the thread I added my own nag there fwiw. :) Wish I knew why PRs seem to just get stuck in there!

[SamPruden]

Hey, I'm sorry I completely dropped out here. I'm moving house in a few days, very busy at the moment. I've got plans and a new prototype for a new version that I'll hopefully be releasing... soon. It's pretty much going to be a ground-up re-write.

As a basic outline of what I'm working towards, I'm actually stepping back and simplifying. I'm taking out the expectation/testing code, and having tsst<T>() return an object with information about the type (and any errors) that can then be used within any existing testing/expectation framework.

I also have a (prototype) version of the tsst<T>().is<string>() method working. The implementation is entirely ridiculous and inefficient, but it functions. Actually I'm just going to explain it now because it's amusing how terrible it is.

1. The main transformer starts by making a sub-transformer.
2. The sub-transformer transforms a copy of each sourcefile, turning each call to .is<T>() into an anonymous function mimicking the the<string, "hello world">() system.
3. The result of this transformation is then written out by the printer into a string.
4. An entirely new program is created from these transformed files, re-parsing and checking everything.
5. The transformed files are then scanned for any diagnostics within our created the<string, "hello world">() functions, and those are matched up with the original is<T>() calls.
6. The original file is then transformed, using the diagnostics from 5 to pass or fail the is<T>() calls.
7. I either laugh or cry, I haven't decided yet.