A way to hook into showing stack trace and error when test failed

[jcubic]

I have this case: I have Scheme Lisp interpreter written in JavaScript and when tests are failing to show very long no relevant stack trace and line that is not related. Because Ava is tracing only JavaScript and not my Scheme code it don't show the line in tests that are failing but line in my eval function for my language.

Here I've showed the line that was failing and the stack trace don't make any sense to me. I'm only care about stack trace if it's actual Exception throw in my code. This is the line that is usually shown when test is failing of one for macros.

So I was thinking it would be cool to disable the stack trace and show my own, maybe to have hooks that will get e object with stack trace and line and if function return false it will not show default message and I will be able to show my own message, I can think of some code that will show line and stack trace. The hook may require to return value is specific format, e.g.:

 {
   lines: {5939: "..", 5940: "...", 5941: '..."},
   line_number: 5940,
   stack_trace: []  // the stack trace can be exception object
}

and if hook return this it will use it's usual display of line and exception, if it don't get this structure it don't display anything, except "difference" and "1 test failed".

The API can of course be different whatever you think will be the best.

The API can pass this object as argument and user can decide in hook if he want to return same object or return modified version.

I know that this is kind really really special case, not everyone is testing his own language. But maybe it will help some others that don't want stack trace that is quite big and only want to have highlighted line. In my case If I had two failing cases I was not able to see the first one because of this gigantic stack trace that filled my terminal, I needed to use less to see the message.

Another idea

Is to have custom Exception that can be thrown from user code (I'm not sure if this is how t.is and other functions works) and the exception can have usual things like line number and stack trace and I will be able to throw my own exception with my own code and lines that are not JavaScript. With this I will be able to throw this ava.Exception and have my own testing functions that will throw that exception on fail. I'm not sure if this would require to have same hook mention earlier.

Example

to see the issue you can clone and run my tests:

git clone https://github.com/jcubic/lips.git
cd lips
git checkout devel
sed -i 's/(string=? #f "foo" "fooo")/(string=? #t "foo" "fooo")/' tests/strings.scm
make test

this will modify one file that will fail.

this one will show this code, that make any sense:

Unfortunately I don't have small reproducible example, for this I need my own language and my unit tests written in that language. Maybe there are other use cases that will benefit from this I'm not sure.

[novemberborn]

Are you referring to the line highlighted in red? That is meant to show which assertion in your test failed. It shouldn't show something like src/lips.js since that's not a test.

Or maybe you're referring to the stack trace below, in grey?

[jcubic]

I'm referring to both, they came from my source not from unit tests, because my unit tests are not written in JavaScript. I have something like this:

readDir('./tests/').then(function(filenames) {
  return Promise.all(filenames.filter(function(file) {
      return file.match(/.scm$/) && !file.match(/^\.#/);
  }).map(function(file) {
    return readFile(`tests/${file}`).then(d => d.toString());
  })).then(function (files) {
      return lips.exec([`
      (define test (require "ava"))

      (load "./lib/bootstrap.scm")
      (load "./lib/R5RS.scm")
      (load "./examples/helpers.scm")
      (load "./tests/helpers/helpers.scm")
      `
      ].concat(files).join('\n\n')).catch((e) => {
          console.log(e);
      });
  });
});

it load all Scheme files and run tests from Scheme. I'm concatenating all the files including Standard libraries and run exec that execute my JavaScript that in turn call ava tests.

[novemberborn]

Could your interpreter rewrite the stack traces? That would work for thrown errors.

If you use the t.try() assertion you could access all errors, including thrown ones, and rewrite their stack. That'd be an unsupported hack but it should work.

[jcubic]

@novemberborn Can you show how to use this API, I can't find it in source code. I have code like this:

t.try(function(e) {
   if (pass) {
     e.pass();
   } else {
     e.fail(message);
   }
});

but for all tests it just shows:

Error: Test finished, but not all attempts were committed or discarded

Ho to rewrite the stack? The unit tests only show that you can call e.pass() and e.fail() inside t.try nothing with stack.

[novemberborn]

See https://github.com/avajs/ava/blob/master/docs/03-assertions.md#trytitle-implementation--macro--macro-args:

const attempt = await t.try(function(e) {
   if (pass) {
     e.pass();
   } else {
     e.fail(message);
   }
});

if (attempt.passed) {
  attempt.commit()
} else {
  // Modify each error in attempt.errors
  attempt.commit()
}

[jcubic]

Thanks, I think it works.