tape-testing / tape

tap-producing test harness for node and browsers
MIT License
5.77k stars 307 forks source link

Any tips on how to add new assertions? #555

Closed customcommander closed 3 months ago

customcommander commented 3 years ago

Hi all,

I use jsverify with tape. It looks like this:

test('always returns true or false', t => {
  const verify = fn => any => typeof fn(any) == 'boolean';
  jsc.assert(jsc.forall('json', verify(checkDocument)));
  t.end();
});

The problem is that since I'm not actually using an assertion the output is:

TAP version 13
# always returns true or false

1..0
# tests 0
# pass  0

# ok

When a property-based assertion fails, it breaks the test (which is nice) but not in a nice way:

TAP version 13
# always returns true or false
/Users/se/GitHub/schemas/node_modules/jsverify/lib/jsverify.js:360
        throw new Error(formatFailedCase(r));
        ^

Error: Failed after 1 tests and 3 shrinks. rngState: 0016cda478925eb3f0; Counterexample: []; 

Some libraries have support for property-based assertions, ideally with tape it'd look like this:

test('always returns true or false', t => {
  const verify = fn => any => typeof fn(any) == 'boolean';
  t.property('json', verify(checkDocument));
  t.end();
});

So I'm looking into adding a new assertion t.property and have failures properly reported. Ideally I would want this as a plugin e.g. tape-jsverify.

Is there any documentation, code or other tape plugin you would recommend I take a look at? Just looking for some info from the maintainers to get me started in the right direction (hopefully).

If tape doesn't really support third-party assertions, that's good to know too.

Thanks!

ljharb commented 3 years ago

There is no official or supported way to add assertions.

That said, you could import https://github.com/substack/tape/blob/master/lib/test.js and mutate it with your own assertions - I do this in a project or two (example). However, it could break even in a patch release, so use it at your own risk.

ljharb commented 3 years ago

It might be nice/reasonable to have some kind of top-level const wrappedTape = tape.withAssertion(name, handler); kind of API? That way, you could add your own assertions safely, without affecting other consumers of tape in the same application.

customcommander commented 3 years ago

That is pretty much what I was thinking.

Not polluting other tape instances with third-party assertions is definitely the way to go. May I perhaps suggest we can register more than one assertion so that we can integrate third-party assertions as seamlessly as possible (YMMV)? e.g.,

const test = require('tape').withAssertions(['foo', fn], ['bar', fn], ...)

test('...', t => {
  t.foo(...);
  t.bar(...);
  t.end();
});
ljharb commented 3 months ago

I went with a slightly simpler API - t.assertion(fn, ...args). fn is what you would have installed on Test.prototype, and it can take whatever arguments you want.

If a call-this operator existed, this would be native, but this will have to do in the meantime.