search

vitest-dev / vitest

Next generation testing framework powered by Vite.
https://vitest.dev
MIT License
13.01k stars 1.16k forks source link

[Feature]: Consider `expected` and `actual` properties of `error.cause` #6799

Closed ammut closed 2 days ago

ammut commented 5 days ago

Clear and concise description of the problem

I'm working with fast-check, a Property-Based-Testing library that generates random input to your functions and tries to find inputs for which the supposed assertions about your code don't hold.

I like combining it with vitest's deepEqual assertions. However, fast-check needs to wrap the error thrown by deepEqual so it can provide information about which input specifically caused my assertions to break. To help with debugging, it includes the error thrown by deepEqual as cause, which is already being displayed by vitest. However, the error message can look like this:

Caused by: AssertionError: expected { firstname: 'ammut', …(6) } to deeply equal { firstname: 'ammut', …(6) }

The error thrown by fast-check doesn't have actual and expected, the AssertionError however definitely does.

Suggested solution

When a test fails and the error does not have the actual and expected properties, vitest should try to trace back the error.cause chain (if there is one) to see if one of the root cause errors does have the actual and expected properties, and then use those to print those sweet, sweet diffs.

Alternative

I also suggested in fast-check to just copy over the actual and expected properties to the wrapping error, but that would fix it only for fast-check.

Additional context

No response

Validations

hi-ogawa commented 5 days ago

Can you provide a quick example and also your current workaround? https://github.com/dubzzz/fast-check/issues/660#issuecomment-2435642063 Your suggestion sounds reasonable, but it would be nice to have a concrete code to see the issue better.

ammut commented 5 days ago

Of course, sorry about that!

current behavior Given this code: ```import * as fc from 'fast-check'; const myFunction = (input: { a: string; b: string; c: string; d: string; e: string }) => // FIXME Implement this input; const arbitraryInput = fc.record({ a: fc.string(), b: fc.string(), c: fc.string(), d: fc.string(), e: fc.string(), }); describe('myFunction', () => { it('should add a "foo" property', () => fc.assert( fc.property(arbitraryInput, (i) => { const actual = myFunction(i); const expected = { ...i, foo: 'foo' }; assert.deepEqual(actual, expected); }), { errorWithCause: true }, )); }); ``` I get this output: ``` FAIL src/error-cause-diff-demo.tests.ts > myFunction > should add a "foo" property Error: Property failed after 1 tests { seed: -1776807447, path: "0:0:0:0", endOnFailure: true } Counterexample: [{"a":"","b":"","c":"","d":"","e":""}] Shrunk 3 time(s) Hint: Enable verbose mode in order to have the list of all failing values encountered during the run ❯ buildError ../../node_modules/fast-check/lib/esm/check/runner/utils/RunDetailsFormatter.js:136:19 131| safeObjectAssign(error, { cause: out.errorInstance }); 132| } 133| return error; | ^ 134| } 135| function throwIfFailed(out) { ❯ throwIfFailed ../../node_modules/fast-check/lib/esm/check/runner/utils/RunDetailsFormatter.js:147:11 ❯ Module.reportRunDetails ../../node_modules/fast-check/lib/esm/check/runner/utils/RunDetailsFormatter.js:160:16 ❯ Module.assert ../../node_modules/fast-check/lib/esm/check/runner/Runner.js:78:31 ❯ src/error-cause-diff-demo.tests.ts:17:8 Caused by: AssertionError: expected { a: '', b: '', c: '', d: '', e: '' } to deeply equal { a: '', b: '', c: '', d: '', …(2) } ❯ src/error-cause-diff-demo.tests.ts:21:16 ❯ Property.predicate ../../node_modules/fast-check/lib/esm/check/property/Property.js:19:98 ❯ Property.run ../../node_modules/fast-check/lib/esm/check/property/Property.generic.js:52:33 ❯ runIt ../../node_modules/fast-check/lib/esm/check/runner/Runner.js:27:30 ❯ check ../../node_modules/fast-check/lib/esm/check/runner/Runner.js:71:11 ❯ Module.assert ../../node_modules/fast-check/lib/esm/check/runner/Runner.js:74:17 ❯ src/error-cause-diff-demo.tests.ts:17:8 ```

But it would be way less of a hassle if I had this output:

desired behavior ``` FAIL src/error-cause-diff-demo.tests.ts > myFunction > should add a "foo" property Error: Property failed after 1 tests { seed: 1974126919, path: "0:0:0:0", endOnFailure: true } Counterexample: [{"a":"","b":"","c":"","d":"","e":""}] Shrunk 3 time(s) Hint: Enable verbose mode in order to have the list of all failing values encountered during the run - Expected + Received Object { "a": "", "b": "", "c": "", "d": "", "e": "", - "foo": "foo", } ❯ buildError ../../node_modules/fast-check/lib/esm/check/runner/utils/RunDetailsFormatter.js:133:19 131| safeObjectAssign(error, { cause: out.errorInstance }); 132| } 133| if ('actual' in out.errorInstance) error.actual = out.errorInstance.actual; | ^ 134| if ('expected' in out.errorInstance) error.expected = out.errorInstance.expected; 135| return error; ❯ throwIfFailed ../../node_modules/fast-check/lib/esm/check/runner/utils/RunDetailsFormatter.js:144:11 ❯ Module.reportRunDetails ../../node_modules/fast-check/lib/esm/check/runner/utils/RunDetailsFormatter.js:157:16 ❯ Module.assert ../../node_modules/fast-check/lib/esm/check/runner/Runner.js:78:31 ❯ src/error-cause-diff-demo.tests.ts:17:8 Caused by: AssertionError: expected { a: '', b: '', c: '', d: '', e: '' } to deeply equal { a: '', b: '', c: '', d: '', …(2) } ❯ src/error-cause-diff-demo.tests.ts:21:16 ❯ Property.predicate ../../node_modules/fast-check/lib/esm/check/property/Property.js:19:98 ❯ Property.run ../../node_modules/fast-check/lib/esm/check/property/Property.generic.js:52:33 ❯ runIt ../../node_modules/fast-check/lib/esm/check/runner/Runner.js:27:30 ❯ check ../../node_modules/fast-check/lib/esm/check/runner/Runner.js:71:11 ❯ Module.assert ../../node_modules/fast-check/lib/esm/check/runner/Runner.js:74:17 ❯ src/error-cause-diff-demo.tests.ts:17:8 ``` Notice these additional lines: ``` - Expected + Received Object { "a": "", "b": "", "c": "", "d": "", "e": "", - "foo": "foo", } ```
hi-ogawa commented 4 days ago

Thanks for the code. I moved it to stackblitz https://stackblitz.com/edit/sb1-n2fgtd?file=src%2FmyFunction.test.ts I'm not sure how you're doing the 2nd one. Is it something configurable or are you patching fast-check?

ammut commented 4 days ago

Yes, I have managed to patch fast-check to achieve that result, but haven't tried patching vitest yet.

hi-ogawa commented 4 days ago

Hmm, I just remembered fast-check author did some improvement related to this:

It looks like this is supposed to work now, but maybe it's a regression on Vitest or fast-check.

hi-ogawa commented 4 days ago

Oh, I just noticed my repro is using old Vitest 1.3.1 (also it was private, sorry) :facepalm: Here is a repro with latest Vitest and it looks working now https://stackblitz.com/edit/sb1-ecvswh?file=src%2FmyFunction.test.ts @ammut Can you check which Vitest you use?

github-actions[bot] commented 3 days ago

Hello @ammut. Please provide a minimal reproduction using a GitHub repository or StackBlitz (you can also use examples). Issues marked with needs reproduction will be closed if they have no activity within 3 days.

ammut commented 3 days ago

Oof, that would be embarrassing... But also pretty cool! I'll check as soon as I'm back at work

ammut commented 2 days ago

Jep, that was it. Looks like a lot of stuff happened in the last few months. Thanks a lot of taking your time!