can't load zora-reporters with ts-node

[mindplay-dk]

I have pure TypeScript project, running directly on ts-node, e.g. with no build-step and compilation or code output.

Trying to set up zora^5 and not having much luck.

package.json

{
  "type": "commonjs",
  "scripts": {
    "test": "ts-node src/test/test.ts"
  },
  "devDependencies": {
    "ts-node": "^10.2.1",
    "typescript": "^4.4.2",
    "zora": "^5.0.0",
    "zora-reporters": "^1.0.0"
  }
}

src/test/test.ts

import { hold, report, test } from 'zora';
import { createDiffReporter } from 'zora-reporters';

hold(); // prevent the default test harness from running

test(`my very first test`, (assertion) => {
  const input = false;
  assertion.ok(input, 'input should be truthy');
});

report({
  reporter: createDiffReporter(),
});

The CommonJS loader appears to fail...

$ npm run test

> test
> ts-node src/test/test.ts

Error [ERR_PACKAGE_PATH_NOT_EXPORTED]: No "exports" main resolved in /home/mindplay/workspace/Muninn/components/tracker/node_modules/zora-reporters/package.json
    at resolveExportsTarget (internal/modules/cjs/loader.js:602:11)
    at applyExports (internal/modules/cjs/loader.js:460:14)
    at resolveExports (internal/modules/cjs/loader.js:513:23)
    at Function.Module._findPath (internal/modules/cjs/loader.js:641:31)
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:1016:27)
    at Function.Module._load (internal/modules/cjs/loader.js:898:27)
    at Module.require (internal/modules/cjs/loader.js:1089:19)
    at require (internal/modules/cjs/helpers.js:73:18)
    at Object.<anonymous> (/home/mindplay/workspace/Muninn/components/tracker/src/test/test.ts:2:1)
    at Module._compile (internal/modules/cjs/loader.js:1200:30) {
  code: 'ERR_PACKAGE_PATH_NOT_EXPORTED'
}

Going by the line-number in the stack trace, the offending line is this one:

import { createDiffReporter } from 'zora-reporters';

The error message doesn't seem to make any sense - I'm not trying to import zr from 'zora-reporters' or something, which is where you normally get this error-message; when trying to import something that doesn't have a main/default export.

A google search mainly turns up stackoverflow and github articles suggesting an npm update, which didn't help.

If I reduce my dependencies and script to just zora without zora-reporters, it works fine:

import { hold, test } from 'zora';

test(`my very first test`, (assertion) => {
  const input = false;
  assertion.ok(input, 'input should be truthy');
});

I have a mix of CommonJS and native modules in my dependencies already, so this doesn't look like a loader/configuration issue.

Any idea why this wouldn't work?

How come zora loads just fine, but zora-reporters doesn't?

[lorenzofox3]

zora-reporters has no commonjs version in its package (I did not want to introduce yet another build step as its first intent is not to use it programmatically) and ts-node seems to compile to cjs under the hood. You'd better try to use it as a cli:

ZORA_REPORTER=json ts-node src/test/test.ts | zr

[mindplay-dk]

I tried that - didn't work either, but gives a different error message:

$ npm run test

> test
> ts-node src/test/test.ts | zr

(node:32351) UnhandledPromiseRejectionWarning: SyntaxError: Unexpected token T in JSON at position 0
    at parse (<anonymous>)
    at file:///home/mindplay/workspace/Muninn/components/tracker/node_modules/zora-reporters/src/utils.js:76:13
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at async file:///home/mindplay/workspace/Muninn/components/tracker/node_modules/zora-reporters/src/diff/index.js:37:20
    at async file:///home/mindplay/workspace/Muninn/components/tracker/node_modules/zora-reporters/src/bin.js:44:3

[lorenzofox3]

because zr is not meant to interpret TAP, you need to pipe json protocol (note the env var):

ZORA_REPORTER=json ts-node src/test/test.ts | zr

[mindplay-dk]

because zr is not meant to interpret TAP

aah, I see - yeah, because Zora has more of a "schema" for it's event stream than TAP does.

What's the benefit to this kind of process separation?

I mean, the Zora protocol is sort of an implementation detail, right? And you have probably type-safety and some debugging advantages when they're just running in the same process - being able to see in a reporter stack-trace where the data originates in the framework and so on. What's the advantage to running two separate instances of Node?

zora-reporters has no commonjs version in its package (I did not want to introduce yet another build step as its first intent is not to use it programmatically)

Programmatically is the first thing shown in the docs. 😉

I would much prefer to run the reporter in-process.

Shipping packages in different formats seems inconsistent.

I can help set up the build step, if you'd like?

[mindplay-dk]

Sorry to ask annoying question, but what is the advantage of the custom test stream format over TAP?

You diff reporter looks awesome - is there any reason it couldn't work with standard TAP input?

I understand I could use TAP output still, of course - and pick another third-party TAP reporter, but yours looks a lot better than any of the ones I've been able to find. 🙂

[lorenzofox3]

because zr is not meant to interpret TAP

aah, I see - yeah, because Zora has more of a "schema" for it's event stream than TAP does.

What's the benefit to this kind of process separation?

separation of concerns (tends to improve code quality), resilience, flexibility (anyone can build a reporter without the need to touch the zora's code base). Unix philosophy in a word :)

I mean, the Zora protocol is sort of an implementation detail, right? And you have probably type-safety and some debugging advantages when they're just running in the same process - being able to see in a reporter stack-trace where the data originates in the framework and so on. What's the advantage to running two separate instances of Node?

It is not an implementation detail. On the contrary, it is a contract consumers can rely on: the interface between a zora program and any software downstream (does not have to be a nodejs program). You might get some advantages running everything in the same process, but here if you feel the need to, that is problematic: it means the reporting part is coupled to the test engine, although there is no reason for it to be.

zora-reporters has no commonjs version in its package (I did not want to introduce yet another build step as its first intent is not to use it programmatically)

Programmatically is the first thing shown in the docs. 😉

Yes you are right although it is more targeted to people which want to build tools on top of zora and which have their own build process, etc (Pta for example is one of theses tools). I expected test producers would not have to use the reporters programmatically. Maybe I was wrong

I would much prefer to run the reporter in-process. Shipping packages in different formats seems inconsistent.

Actually the target is ES modules (the modern way, node >= 14), and all the packages support ES. Note: some prolific package authors have entirely stopped shipping their pacakges as CJS.

Zora itself supports CJS but for legacy programs. However it seems the cohabitation between TS (again causing troubles :) ) and ES modules is not always smooth. We might want to reintroduce the build step (and the complexity which goes together) to have a better TS support.

I can help set up the build step, if you'd like?

Yes, please. That would be awesome !

[lorenzofox3]

Sorry to ask annoying question, but what is the advantage of the custom test stream format over TAP?

I have found over the time that TAP was lacking some data. For example the diagnostic part is not specified and most of the tools rely on the choices Tape did. A test is defined as "comment" and there is no straightforward way to make the difference with a sub test. Zora's protocol solve these kind of problems I have encountered over the time

[lorenzofox3]

By the way. You can't tweak you ts config to support es modules for a set of packages ?

[mindplay-dk]

separation of concerns (tends to improve code quality), resilience, flexibility (anyone can build a reporter without the need to touch the zora's code base)

Just my understanding, but separation of concerns doesn't need to imply separation of processes - you can get all of the code quality benefits regardless of how you thread/fork the actual work.

You might get some advantages running everything in the same process, but here if you feel the need to, that is problematic: it means the reporting part is coupled to the test engine, although there is no reason for it to be.

Hmm, not the way I understand it. I think we're talking about two different things: how the code is structured, versus how the code executes. I mean, you have the necessary abstractions in place either way - so I get the separation of concerns whether I choose to run the reporters in-process or out-of-process. I don't see the different in code quality terms?

I expected test producers would not have to use the reporters programmatically. Maybe I was wrong

There's no "right or wrong" here, and I can only speak for myself, but I prefer to do everything programmatically - in fact, this was one of the key reasons I was drawn to Zora: there's no magic or conventions or dependencies on scripts or configuration files or anything global; I can open a source-file and see exactly what it means to run the test.

It's part of the reason I still haven't used pta either - having something that finds and runs my tests is a very small convenience to me, and has a few downsides: mainly, there's no longer a source-file where I can see what running the test means or why -it's no longer obvious how to temporarily disable a test, how to control execution order, and so on.

(Side note, the new hold function actually bugs me - having something execute by default, with the option to stop if from executing, is so much less transparent than having a run method that you're required to call. Again, just my philosophy, but it's much harder to understand why something runs without your calling anything to start it, than it is to understand why something doesn't run without your calling it to start. I would have much preferred a line of code for that. I understand why that might not jive with the interplay with something like pta though...)

Anyhow, as said, I know I'm only speaking for myself here. I have a very minimalist point of view - I want as few moving parts as absolutely possible and generally favor explicitness over convenience. I aim to build things that are transparent enough that the next person doesn't have to understand the inner workings of a library or framework. Abstracting, in my books, is good - but hiding is bad. Why a test runs, or how it gets to a reporter - these are not implementation details I want to hide, and not really (in my opinion) difficult or complex enough that hiding or abstracting them is even worth while.

Just my opinion - this is already a long post, and I don't want to take up more of your time than necessary, but I felt like it was worth explaining where I come from. If you disagree, that's totally fine, I won't push the subject. 🙂

(do you enjoy discussing these things, or do you mind? this is open source, so I want to be respectful of taking up your time.)

---

A test is defined as "comment" and there is no straightforward way to make the difference with a sub test.

As far as I know, it's conventional for TAP producers to use YAML to side-band framework-specific information? So, to Zora, side-band information about sub-tests etc. would be meaningful, while to other TAP tools, it would just be text - YAML being reasonably human-readable.

That said, I'm not a fan of YAML by any means. I've kind of wished in the past that TAP would get superseded by something that uses a simple JSON stream like you've done here. It is a much more reliable protocol, and every developer in the world can read JSON - you can't really say that about either TAP or YAML.

By the way. You can't tweak you ts config to support es modules for a set of packages ?

As far as I know, it's an all-or-nothing sort of deal? I use these compiler options:

"module": "CommonJS",
"moduleResolution": "node",
"esModuleInterop": true

I don't know that there's anything specifically you can enable for ES modules specifically. I also don't know how many of these settings are supported by ts-node, which (afaik) uses Babel under the hood?

[mindplay-dk]

A simple interim solution would be to just re-export createDiffReporter from zora, same as you do for the TAP and JSON reporters?

[lorenzofox3]

separation of concerns (tends to improve code quality), resilience, flexibility (anyone can build a reporter without the need to touch the zora's code base)

Just my understanding, but separation of concerns doesn't need to imply separation of processes - you can get all of the code quality benefits regardless of how you thread/fork the actual work.

You might get some advantages running everything in the same process, but here if you feel the need to, that is problematic: it means the reporting part is coupled to the test engine, although there is no reason for it to be.

Hmm, not the way I understand it. I think we're talking about two different things: how the code is structured, versus how the code executes. I mean, you have the necessary abstractions in place either way - so I get the separation of concerns whether I choose to run the reporters in-process or out-of-process. I don't see the different in code quality terms?

You have some valid points. However I am not sure this is the right place to discuss software architecture and general opinions. Zora is designed that way for some reasons and with some tradeoff.

Ideally you should be able to use both way: UNIX like with text streams passing from one process to another, or programmatically. In your case it is not possible to use it programmatically and we need to address the issue.

I am just sad to see the amount of effort and complexity we need to put in place to support a tool that is not able (or you do not manage to successfully configure) to compile your source code into a valid program. I mean you can write the same program in JS and it will run in any targeted platform (nodejs >= 14, Deno, browsers). That's just puzzling to see the burden is on the package maintainers and they have to make some adjustments so TS can work...

I expected test producers would not have to use the reporters programmatically. Maybe I was wrong

There's no "right or wrong" here, and I can only speak for myself, but I prefer to do everything programmatically - in fact, this was one of the key reasons I was drawn to Zora: there's no magic or conventions or dependencies on scripts or configuration files or anything global; I can open a source-file and see exactly what it means to run the test.

It's part of the reason I still haven't used pta either - having something that finds and runs my tests is a very small convenience to me, and has a few downsides: mainly, there's no longer a source-file where I can see what running the test means or why -it's no longer obvious how to temporarily disable a test, how to control execution order, and so on.

(Side note, the new hold function actually bugs me - having something execute by default, with the option to stop if from executing, is so much less transparent than having a run method that you're required to call. Again, just my philosophy, but it's much harder to understand why something runs without your calling anything to start it, than it is to understand why something doesn't run without your calling it to start. I would have much preferred a line of code for that. I understand why that might not jive with the interplay with something like pta though...)

I understand your point, but that's not the default behavior, by design. In the first version of zora, you had to explicitly call run to make the tests run, but that is breaking one of the main goal: "you don't need any test runner". When you write a .run somewhere you are building a test runner(you need to handle how files are loaded, etc). If that's something you want, you can still create the harness manually and you'll have full control on the sequence, it will be however be your responsibility to handle the side effects.

The semantic of hold etc is the same as a readable stream in nodejs: as soon a you .pipe(here as soon as you run) the data start flowing. You can eventually .pause() a Readable stream (here .hold()) and .resume (here report()) but that's not something very common.

If you wish to keep control on that sequence you can just create the test harness manually instead of relying on the default singleton

// test-harness.js
import {createHarness} from 'zora';

const harness = createHarness();

export const test = harness.test
export const report = harness.report

// foo.test.js
import {test} from './test-harness.js'

test('foo', t => t.ok(true))

// your test runner (./index.js)

import {report} from './test-harness.js'

// load test files etc
import './foo.test.js';

report().then(doSomethingAfterTest)

A test is defined as "comment" and there is no straightforward way to make the difference with a sub test.

As far as I know, it's conventional for TAP producers to use YAML to side-band framework-specific information? So, to Zora, side-band information about sub-tests etc. would be meaningful, while to other TAP tools, it would just be text - YAML being reasonably human-readable.

That said, I'm not a fan of YAML by any means. I've kind of wished in the past that TAP would get superseded by something that uses a simple JSON stream like you've done here. It is a much more reliable protocol, and every developer in the world can read JSON - you can't really say that about either TAP or YAML.

The diagnostic part is not specified. Even if YAML is conventional, most of the tools in the JS world make many assumptions (ex: there should be an expected and an actual properties). The test description are TAP comment with no specific distinction whether it is an actual test, a subtest, a comment etc. Just try to build a reporter consuming a TAP stream and you'll see you'll have to make many assumptions along the way. Nevertheless I am ok with TAP format, it just lacks some distinctions I could expect for a messaging protocol defined for testing programs.

[lorenzofox3]

A simple interim solution would be to just re-export createDiffReporter from zora, same as you do for the TAP and JSON reporters?

That is not an option: this reporter is not a simple message passing reporter. It is an opiniated peace of software which brings a lot of nodejs related stuff (TTY, stdout, process) and impact the simplicity of the library and the users who want to use the library in a different environment

[lorenzofox3]

closed in favor of the linked issue

[lorenzofox3]

@mindplay-dk Can you confirm you do not have the problem anymore with the latest verions of zora-reporters ?

Thanks

[mindplay-dk]

Yes! Works beautifully now - I can console.log to my heart's content! 😄

Thank you for your excellent support of this library. 🙏

[tsnobip]

sorry to excavate a closed issue, but @mindplay-dk do you manage to run your TS tests with pta or only calling the tests directly?

[mindplay-dk]

I don't use pta - I don't know where Lorenzo's work as far as allowing mixed JSON/data stream landed?

(I advised him not to, since you could console.log JSON values as well - I don't know if/how he got around that.)

[tsnobip]

OK I'll create a separate issue for pta and typescript then, thanks :)