How to mark a deferred case as failed?

[Finesse]

If a not deferred test case throws an error, the UI shows that the case has failed:

How can I mark a defer test case as failed similar to the example above? I tried deferred.reject(), but the function isn't defined.

[psiho]

Hm I'm not sure why would you want that unless you want to optimize speed of throwing errors :) Anyway, JSBench.me is just a wrapper for benchmark.js so maybe their docs say something about this. Honestly, I don't know. Personally, the code I want to test, i make sure there are no errors inside.

In case you only want to know that something went wrong, you can also open inspector in your browser and error should be logged there. I'm not sure why testing process for deferred tests does not exit on error. I'll take a look. v1.1 should be started soon.

[Finesse]

I'm not sure why would you want that unless you want to optimize speed of throwing errors :)

I don’t want to test expected errors. My example is just an example. At the moment I have few options to deal with errors:

1. Don’t bother with errors. The problem is that in case of an error the test case and the whole process of benchmarking will hang and never complete. Probably, there should be a test case timeout so that the benchmark process can continue even when a test case doesn’t call deferred.resolve() because of an error or any other reason.
2. Catch errors and call deferred.resolve(). I won’t know that the case doesn’t work, and the benchmark will measure the performance of something else, not of what I want. If I can call deferred.reject(error) instead, it will be a solution.

JSBench.me is just a wrapper for benchmark.js so maybe their docs say something about this.

I’ve addressed the question to benchmark.js, thanks.

Personally, the code I want to test, i make sure there are no errors inside.

I agree, but it’s not always possible because:

• A code just can’t work in some browsers due to a lack of API. I want the UI to let know that the case doesn’t work in this case.
• Unexpected errors happen

In case you only want to know that something went wrong, you can also open inspector in your browser and error should be logged there.

It’s a bad UX and not always possible, e.g. in Chrome on iOS.

[psiho]

start on v1.1 will be very soon, so I'll include this on the list and try to see it in more detail. JSBenchme development is kinda cyclic. Once or twice a year I batch all the features/bug requests and do them all at once. So I really cannot say much more about this from the top of my head. It's been months. But it seems a reasonable request to make it impossible for test to hang, even if it is a benchmark.js issue (any maybe it's not).

[Finesse]

@psiho Ok, thank you

[psiho]

This turned out to be more complex than I thought. Benchmark.js just doesn't handle deferred errors. It handles regular ones ok, but deferred... not at all. But thanks to user 'kevinoid' at https://github.com/bestiejs/benchmark.js/issues/123 I saved some time and just implemented solution similar to his. It's a bit hacky, but I think it's solid.

Latest dev version is at https://dev--jsbenchme-v1.netlify.app/ so you can take a look and let me know if you need some tweaking, before v1.1 is published.

[Finesse]

@psiho How is asynchronous error supposed to be reported at https://dev--jsbenchme-v1.netlify.app/? I've tried these code snippets with defer mode, they never complete:

new Promise(resolve => setTimeout(resolve, 100))
  .then(() => {
    deferred.reject()
    throw new Error('Test')
  })

setTimeout(() => {
  deferred.reject()
  throw new Error('Test')
}, 100)

[psiho]

@Finesse , I was testing with something like your 2nd example and if you run that one alone it will throw error (and continue running other tests if any). Error thrown is and displayed in UI is: Uncaught TypeError: deferred.reject is not a function ... because Benchmark.js defer object does not have reject() method and ther's no way for me to add it without changing the library itself.

As for the first example, it's interesting to see why this one was not caught by UI. Error is output to console, but somehow it bypassed window.onerror.

Will continue to tune this.

[psiho]

@Finesse , oh, it seems this is just a different type of error and requires 'unhandledrejection' event listener on window. That catches those.

New dev version v1.0.0-513 dev is up for you to try.

[Finesse]

@psiho Now both examples work 👍

But the second example prints Script error instead of Error: Test. It's not a bit issue for me.

[psiho]

I get proper error. What browser/OS your're using?

[Finesse]

Safari 14.0.3 on macOS 11. Probably, browsers use different security settings for accessing error messages.

[psiho]

yup, tried it with Midori, closest I can get to Apple, and error object contains no mora data. I'll call it a day. Important thing is that this does not interfere with other tests, that they continue while failing test aborts and errors. We're all developers, console contains more details on errors.

Closing this. Will probably go live over the weekend.

[Finesse]

Thank you!