Execute in new context

[alexjeffburke]

The changes appear to run tests in a next context. I've rejigged things to allow specifying globals and an object of functions that can create them - this means that the theming for each test continues to work.

[papandreou]

Looks like this runs into the same problems as I did when I last tried. I tried linking it into unexpected-check:

[unexpected-check]$ npm link unexpected-markdown
/home/andreas/work/unexpected-check/node_modules/unexpected-markdown -> /home/andreas/.nvm/versions/node/v10.16.2/lib/node_modules/unexpected-markdown -> /home/andreas/work/unexpected-markdown
[unexpected-check]$ cd node_modules/metalsmith-unexpected-markdown/
[metalsmith-unexpected-markdown]$ npm link unexpected-markdown
/home/andreas/work/unexpected-check/node_modules/metalsmith-unexpected-markdown/node_modules/unexpected-markdown -> /home/andreas/.nvm/versions/node/v10.16.2/lib/node_modules/unexpected-markdown -> /home/andreas/work/unexpected-markdown
[metalsmith-unexpected-markdown]$ cd ../..

[unexpected-check]$ npm run update-examples

> unexpected-check@2.4.1 update-examples /home/andreas/work/unexpected-check
> generate-site --require ./bootstrap-unexpected-markdown.js --update-examples

wrote site to site-build

This produces:

diff --git a/documentation/assertions/any/when-fuzzed-by.md b/documentation/assertions/any/when-fuzzed-by.md
index d4e9b7c..b55aa08 100644
--- a/documentation/assertions/any/when-fuzzed-by.md
+++ b/documentation/assertions/any/when-fuzzed-by.md
@@ -30,16 +30,5 @@ expect('abc', 'when fuzzed by', makePrefixGenerator, 'to match', /^a/);

-Found an error after 4 iterations
-counterexample:
-
-  Generated input: ''
-  with: fuzz({
-    value: 'abc',
-    mutator: integer({ min: 0, max: 2 }).map(function (prefixLength) {
-      return str.substr(0, prefixLength);
-    })
-  })
-
-  expected '' to match /^a/
+integer is not a function

diff --git a/documentation/assertions/function/to-be-valid-for-all.md b/documentation/assertions/function/to-be-valid-for-all.md index b1d4123..e6fcb05 100644 --- a/documentation/assertions/function/to-be-valid-for-all.md +++ b/documentation/assertions/function/to-be-valid-for-all.md @@ -64,15 +64,7 @@ expect(text => {

-Found an error after 220 iterations
-counterexample:
-
-  Generated input: ''
-  with: string({ min: 0, max: 200 })
-
-  TypeError: Cannot read property 'forEach' of null
-      at rleEncode (/path/to/file.js:x:y)
-      at /path/to/file.js:x:y)
+string is not a function

Something is failing for the empty string input. The problem is that the regular diff --git a/documentation/index.md b/documentation/index.md index 9f8e131..cb1a2bd 100644 --- a/documentation/index.md +++ b/documentation/index.md @@ -93,13 +93,18 @@ But that assumption is actually not true as the build-in sort functions is based on converting items to strings and comparing them. So you will get the following error:

-Found an error after 1 iteration, 107 additional errors found.
-counterexample:
-
-  Generated input: [ -2, -1 ]
-  with: array({ itemGenerator: integer({ min: -20, max: 20 }), min: 1, max: 20 })
-
-  expected [ -1, -2 ] to be sorted
+expected
+function (arr) {
+  var sorted = sort(arr);
+  expect(sorted, 'to have length', arr.length);
+  expect(sorted, 'to be sorted');
+}
+to be valid for all undefined
+  The assertion does not have a matching signature for:
+    <function> to be valid for all <undefined>
+  did you mean:
+    <function> to be valid for all <chance-generator+>
+    <function> to be valid for all <object>

If we wanted to fix the problem, we would need to use a comparison function: @@ -216,13 +221,7 @@ expect((arr) => {

 ```output
-Found an error after 1 iteration, 9 additional errors found.
-counterexample:
-
-  Generated input: [ 0, 0, -2, -1 ]
-  with: array({ itemGenerator: integer({ min: -20, max: 20 }), min: 1, max: 20 })
-
-  expected [ -1, -2, 0, 0 ] to be sorted
+Cannot read property 'values' of undefined

As you can see the input shrinking is worse with less iterations, but it will be

[sunesimonsen]

@papandreou I think we need to expose a way from bootstrap to setup the sandbox:

global.createSandbox = () => ({
  foo: 'bar'
})

This is of cause a breaking change and we need to also accommodate that in the genenrated test code, but I think it will provide a more isolated and stable execution context.

What do you think?

[papandreou]

Hmm, if we're about to make breaking changes, I think we should make sure that we work towards unifying the mocha mode and the update examples mode.

[sunesimonsen]

@papandreou yes I agree, I would like the tests to also execute in the VM.

[alexjeffburke]

I need to look at unexpected-check, seems like it’s a good test bed for this.

Agreed with the sandbox naming, will make the change. As for a global .createSandbox function, I like how explicit that is but I need to see how the expect cloning stuff fits in with it - I’m a touch worried that the globals idea was very tailored to the snippet execution and I want to make shre there isn’t an issue.

[alexjeffburke]

Were executing each snippet in a new context to be implemented, it would be atop https://github.com/alexjeffburke/evaldown which replaces subsumes unexpected-markdown and has already hardened the existing environmental cleanup code. Closing.