Very large `npm audit --json` bodies cause parse errors

shockey commented 4 years ago

I have a project which currently has a ton (27,714) of vulnerabilities. It seems that npm-audit-ci-wrapper can't handle that much information coming from npm:

> swagger-client@3.10.0 security-audit:prod /Users/kyle/Code/swagger-api/js
> npm-audit-ci-wrapper -p -t low

undefined:9737
          "id": 1

SyntaxError: Unexpected end of JSON input
    at JSON.parse (<anonymous>)
    at parse_audit_results (/Users/kyle/Code/swagger-api/js/node_modules/npm-audit-ci-wrapper/lib/parser.js:31:21)
    at exec (/Users/kyle/Code/swagger-api/js/node_modules/npm-audit-ci-wrapper/bin/index.js:44:36)
    at ChildProcess.exithandler (child_process.js:282:5)
    at emitTwo (events.js:126:13)
    at ChildProcess.emit (events.js:214:7)
    at maybeClose (internal/child_process.js:925:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:209:5)

Versions

➜ node -v
v8.11.2
➜ npm -v
6.6.0
➜ npm ls npm-audit-ci-wrapper
swagger-client@3.10.0 /Users/kyle/Code/swagger-api/js
└── npm-audit-ci-wrapper@2.3.0

Attachments

My npm audit --json output: swagger-client-npm-audit-2020-03-22-190120.json.zip

InfoSec812 commented 4 years ago

Uhhggg... Yeah, I was hoping to avoid this situation. I will have to see if I can switch over to streaming processing of the results since trying to process the entire result will be problematic. It may take a while before I have time to address this.

Thanks for reporting the issue.

InfoSec812 commented 4 years ago

I'm looking into using JSONStream and JSONPath filtering so that the application can handle arbitrarily large JSON data. You'll be able to see that work in the Issue-51-_-Handle_large_json_response as it progresses.

InfoSec812 commented 4 years ago

Could you try out the branch Issue-51-_-Handle_large_json_response and let me know if it meets your needs?

git clone --single-branch --branch Issue-51-_-Handle_large_json_response https://github.com/InfoSec812/npm-audit-ci-wrapper.git
cd npm-audit-ci-wrapper
npm install
npm link

InfoSec812 commented 4 years ago

@shockey As soon as you can confirm that this works for you, I will merge/tag/release.

InfoSec812 commented 4 years ago

@shockey Bump... Any feedback on this?

shockey commented 4 years ago

@InfoSec812 sorry for the delay here — looks good on my end!

output:

➜ nr security-audit:prod          

> swagger-client@3.10.0 security-audit:prod /Users/kyle/Code/swagger-api/js
> npm-audit-ci-wrapper -p -t low

The following production vulnerabilities are low severity or higher:
┌────────────────┬──────────┬───────────────────────────────────────┐
│ module         │ severity │ overview                              │
├────────────────┼──────────┼───────────────────────────────────────┤
│ minimist@0.0.8 │ low      │ https://www.npmjs.com/advisories/1179 │
├────────────────┼──────────┼───────────────────────────────────────┤
│ kind-of@6.0.2  │ low      │ https://www.npmjs.com/advisories/1490 │
└────────────────┴──────────┴───────────────────────────────────────┘

underlying npm audit metadata:

➜ npm audit --json | jq '.metadata'
{
  "vulnerabilities": {
    "info": 0,
    "low": 27714,
    "moderate": 0,
    "high": 0,
    "critical": 0
  },
  "dependencies": 41,
  "devDependencies": 1377709,
  "optionalDependencies": 14769,
  "totalDependencies": 1377750
}

InfoSec812 commented 4 years ago

Awesome @shockey, I'll get this merged and released.

InfoSec812 / npm-audit-ci-wrapper

Very large `npm audit --json` bodies cause parse errors #51

Versions

Attachments