search

dubzzz / fast-check

Property based testing framework for JavaScript (like QuickCheck) written in TypeScript
https://fast-check.dev/
MIT License
4.33k stars 181 forks source link

ELIFECYCLE when running fuzz-rest-api test demo #198

Closed DanielJenkyn closed 6 years ago

DanielJenkyn commented 6 years ago

After following the below steps, crash happens after npm run test

npm ERR! code ELIFECYCLE
npm ERR! errno 3
npm ERR! poc-fuzz-rest-api@1.0.0 test: `mocha --require babel-polyfill --require babel-register "test/**/*.js"`
npm ERR! Exit status 3
npm ERR!
npm ERR! Failed at the poc-fuzz-rest-api@1.0.0 test script.

Could be an issue on my end as I'm not super familiar with Javascript/node, any help would be appreciated

dubzzz commented 6 years ago

Hi @DanielJenkyn ,

fuzz-rest-api contains explicitely broken REST API to show that property based testing can be used to detect the weaknesses introduced in this API. As a consequence npm run test is expected to fail with Exit status 3.

You should expect to get something like:

$ npm run test

> poc-fuzz-rest-api@1.0.0 test C:\src\fuzz-rest-api
> mocha --require babel-polyfill --require babel-register "test/**/*.js"

  Fuzzing REST API
    1) /api/login
    2) /api/profile/:uid
    3) /api/comment

  0 passing (432ms)
  3 failing

  1) Fuzzing REST API
       /api/login:
     Error: Property failed after 4 tests
{ seed: 1538426916331, path: "3:0:3:6" }
Counterexample: [{"password":"'"}]
Shrunk 3 time(s)
Got error: Error: Internal Server Error, got: {"data":"<!DOCTYPE html>\n<html lang=\"en\">\n<head>\n<meta charset=\"utf-8\">\n<title>Error</title>\n</head>\n<body>\n<pre>Error: SQLITE_ERROR: unrecognized token: &quot;&#39;&#39;&#39;&quot;</pre>\n</body>\n</html>\n","status":500}

Stack trace: Error: Internal Server Error, got: {"data":"<!DOCTYPE html>\n<html lang=\"en\">\n<head>\n<meta charset=\"utf-8\">\n<title>Error</title>\n</head>\n<body>\n<pre>Error: SQLITE_ERROR: unrecognized token: &quot;&#39;&#39;&#39;&quot;</pre>\n</body>\n</html>\n","status":500}
    at exports.throwIfHttpFailed (C:/src/fuzz-rest-api/test/asyncHttp.js:38:33)
    at process._tickCallback (internal/process/next_tick.js:178:7)

Hint: Enable verbose mode in order to have the list of all failing values encountered during the run
      at throwIfFailed (node_modules\fast-check\lib\check\runner\utils\utils.js:50:15)
      at run (node_modules\core-js\modules\es6.promise.js:75:22)
      at C:\src\fuzz-rest-api\node_modules\core-js\modules\es6.promise.js:92:30
      at flush (node_modules\core-js\modules\_microtask.js:18:9)
      at process._tickCallback (internal/process/next_tick.js:172:11)

  2) Fuzzing REST API
       /api/profile/:uid:
     Error: Property failed after 2 tests
{ seed: 1538426916466, path: "1:1:0:0:0" }
Counterexample: [" "]
Shrunk 4 time(s)
Got error: Error: Internal Server Error, got: {"data":"<!DOCTYPE html>\n<html lang=\"en\">\n<head>\n<meta charset=\"utf-8\">\n<title>Error</title>\n</head>\n<body>\n<pre>Error: SQLITE_ERROR: incomplete input</pre>\n</body>\n</html>\n","status":500}

Stack trace: Error: Internal Server Error, got: {"data":"<!DOCTYPE html>\n<html lang=\"en\">\n<head>\n<meta charset=\"utf-8\">\n<title>Error</title>\n</head>\n<body>\n<pre>Error: SQLITE_ERROR: incomplete input</pre>\n</body>\n</html>\n","status":500}
    at exports.throwIfHttpFailed (C:/src/fuzz-rest-api/test/asyncHttp.js:38:33)
    at process._tickCallback (internal/process/next_tick.js:178:7)

Hint: Enable verbose mode in order to have the list of all failing values encountered during the run
      at throwIfFailed (node_modules\fast-check\lib\check\runner\utils\utils.js:50:15)
      at run (node_modules\core-js\modules\es6.promise.js:75:22)
      at C:\src\fuzz-rest-api\node_modules\core-js\modules\es6.promise.js:92:30
      at flush (node_modules\core-js\modules\_microtask.js:18:9)
      at process._tickCallback (internal/process/next_tick.js:172:11)

  3) Fuzzing REST API
       /api/comment:
     Error: Property failed after 12 tests
{ seed: 1538426916502, path: "11:0:2:2:2:3:4:4:4" }
Counterexample: [{"user":{"login":""},"comment":{"postId":0,"commentId":""}}]
Shrunk 8 time(s)
Got error: Error: Internal Server Error, got: {"data":"<!DOCTYPE html>\n<html lang=\"en\">\n<head>\n<meta charset=\"utf-8\">\n<title>Error</title>\n</head>\n<body>\n<pre>Error: Supposed it failed on this case<br> &nbsp; &nbsp;at router.post.wrap (C:\\src\\fuzz-rest-api\\src\\server.js:62:61)<br> &nbsp; &nbsp;at C:\\src\\fuzz-rest-api\\node_modules\\async-middleware\\dist\\index.js:18:23<br> &nbsp; &nbsp;at Layer.handle [as handle_request] (C:\\src\\fuzz-rest-api\\node_modules\\express\\lib\\router\\layer.js:95:5)<br> &nbsp; &nbsp;at next (C:\\src\\fuzz-rest-api\\node_modules\\express\\lib\\router\\route.js:137:13)<br> &nbsp; &nbsp;at Route.dispatch (C:\\src\\fuzz-rest-api\\node_modules\\express\\lib\\router\\route.js:112:3)<br> &nbsp; &nbsp;at Layer.handle [as handle_request] (C:\\src\\fuzz-rest-api\\node_modules\\express\\lib\\router\\layer.js:95:5)<br> &nbsp; &nbsp;at C:\\src\\fuzz-rest-api\\node_modules\\express\\lib\\router\\index.js:281:22<br> &nbsp; &nbsp;at Function.process_params (C:\\src\\fuzz-rest-api\\node_modules\\express\\lib\\router\\index.js:335:12)<br> &nbsp; &nbsp;at next (C:\\src\\fuzz-rest-api\\node_modules\\express\\lib\\router\\index.js:275:10)<br> &nbsp; &nbsp;at Function.handle (C:\\src\\fuzz-rest-api\\node_modules\\express\\lib\\router\\index.js:174:3)</pre>\n</body>\n</html>\n","status":500}

Stack trace: Error: Internal Server Error, got: {"data":"<!DOCTYPE html>\n<html lang=\"en\">\n<head>\n<meta charset=\"utf-8\">\n<title>Error</title>\n</head>\n<body>\n<pre>Error: Supposed it failed on this case<br> &nbsp; &nbsp;at router.post.wrap (C:\\src\\fuzz-rest-api\\src\\server.js:62:61)<br> &nbsp; &nbsp;at C:\\src\\fuzz-rest-api\\node_modules\\async-middleware\\dist\\index.js:18:23<br> &nbsp; &nbsp;at Layer.handle [as handle_request] (C:\\src\\fuzz-rest-api\\node_modules\\express\\lib\\router\\layer.js:95:5)<br> &nbsp; &nbsp;at next (C:\\src\\fuzz-rest-api\\node_modules\\express\\lib\\router\\route.js:137:13)<br> &nbsp; &nbsp;at Route.dispatch (C:\\src\\fuzz-rest-api\\node_modules\\express\\lib\\router\\route.js:112:3)<br> &nbsp; &nbsp;at Layer.handle [as handle_request] (C:\\src\\fuzz-rest-api\\node_modules\\express\\lib\\router\\layer.js:95:5)<br> &nbsp; &nbsp;at C:\\src\\fuzz-rest-api\\node_modules\\express\\lib\\router\\index.js:281:22<br> &nbsp; &nbsp;at Function.process_params (C:\\src\\fuzz-rest-api\\node_modules\\express\\lib\\router\\index.js:335:12)<br> &nbsp; &nbsp;at next (C:\\src\\fuzz-rest-api\\node_modules\\express\\lib\\router\\index.js:275:10)<br> &nbsp; &nbsp;at Function.handle (C:\\src\\fuzz-rest-api\\node_modules\\express\\lib\\router\\index.js:174:3)</pre>\n</body>\n</html>\n","status":500}
    at exports.throwIfHttpFailed (C:/src/fuzz-rest-api/test/asyncHttp.js:38:33)
    at process._tickCallback (internal/process/next_tick.js:178:7)

Hint: Enable verbose mode in order to have the list of all failing values encountered during the run
      at throwIfFailed (node_modules\fast-check\lib\check\runner\utils\utils.js:50:15)
      at run (node_modules\core-js\modules\es6.promise.js:75:22)
      at C:\src\fuzz-rest-api\node_modules\core-js\modules\es6.promise.js:92:30
      at flush (node_modules\core-js\modules\_microtask.js:18:9)
      at process._tickCallback (internal/process/next_tick.js:172:11)

npm ERR! code ELIFECYCLE
npm ERR! errno 3
npm ERR! poc-fuzz-rest-api@1.0.0 test: `mocha --require babel-polyfill --require babel-register "test/**/*.js"`
npm ERR! Exit status 3
npm ERR!
npm ERR! Failed at the poc-fuzz-rest-api@1.0.0 test script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\user\AppData\Roaming\npm-cache\_logs\2018-10-01T20_48_36_911Z-debug.log

Both /api/login, /api/profile/:uid and /api/comment are broken.

The above log shows that:

If your error log contains Got error: Property timeout: exceeded limit of 100 milliseconds or Uncaught Error: connect ECONNREFUSED 127.0.0.1:8080 it might be because the npm run start is not running anymore.

Please let me know if you need more details.

DanielJenkyn commented 6 years ago

@dubzzz ahh I wasn't sure if it was meant it was meant to fail gracefully. So the errno 3 and Exit status 3 are intended.

Thanks for the help, I'll close the issue.