Open quinnturner opened 5 years ago
Fails again, even with 2.1.0
: https://travis-ci.org/sumup/circuit-ui/builds/559579687
I don't think that's v2.1.0
since readline-transform
was not a dependency at that stage.
I double-checked in the yarn.lock
file and you're right, that was indeed v2.2.0
. Interestingly though, the PR we thought that fixed it (by downgrading to v2.1.0
) was also using v2.2.0
and it passed: https://travis-ci.org/sumup/circuit-ui/builds/559499754.
I'll try downgrading for real and see if that gives us more stable results.
Edit: We're back to the old issue: https://travis-ci.org/sumup/circuit-ui/jobs/559625735
Hmm, I checked the build you sent and its output is unexpected:
Yarn audit report results:
Passed yarn security audit.
Running locally, I get something like:
Yarn audit report results:
{
"vulnerabilities": {
"info": 0,
"low": 0,
"moderate": 0,
"high": 1,
"critical": 0
},
"dependencies": 1,
"devDependencies": 0,
"optionalDependencies": 0,
"totalDependencies": 1
}
Passed yarn security audit.
We're still running into this with the latest version 2.3.0:
❯ yarn list --depth 0 --pattern audit-ci
yarn list v1.13.0
└─ audit-ci@2.3.0
✨ Done in 0.51s.
yarn run v1.13.0
$ node ./node_modules/.bin/audit-ci --config .audit-ci.json --pass-enoaudit
Yarn audit report summary:
events.js:167
throw er; // Unhandled 'error' event
^
Error: Invalid JSON (Unexpected " " at position 4 in state STOP)
Any clues what's going on?
we encountered this problem, turns out the CI environment we're running in (travis) was using npm 5.6.0 which didn't support audit
yet and was causing misleading JSON error. Forcing the correct npm version resolved the problem for us.
We are also still seeing this issue with
audit-ci
-> 2.3.0
node
-> 8.16.0
yarn
-> 1.17.3
$ yarn list --depth 0 --pattern audit-ci
yarn list v1.17.3
yaml@~3.7.0"
└─ audit-ci@2.3.0
✨ Done in 1.33s.
$ yarn rm-audit
yarn run v1.17.3
$ node_modules/audit-ci/bin/audit-ci --config yarn-audit.json
Yarn audit report summary:
events.js:183
throw er; // Unhandled 'error' event
^
Error: Invalid JSON (Unexpected " " at position 4 in state STOP)
Any help on hint would be very cool to have
It seems that this is only happening with Yarn. I am not sure of the issue yet. Can someone who has reproduced this error run yarn audit --json
? If you don't want to post here, I'd also accept an email. 🙂
CC: @mobilutz @terryma
@quinnturner I see the same problem with --json
:
$ ./node_modules/.bin/audit-ci --json
Yarn audit report results:
events.js:183
throw er; // Unhandled 'error' event
^
Error: Invalid JSON (Unexpected " " at position 4 in state STOP)
I tried looking into it a bit, but didn't get too far. I only see why the error is thrown here: https://github.com/creationix/jsonparse/blob/master/jsonparse.js#L266
The problem is, that this.string
is <---
and of course, this is not a number.
if (isNaN(result)){
console.error("result: "+result);
console.error("this.string: "+this.string);
console.error("n: "+n);
return this.charError(buffer, i);
}
result: NaN
this.string: <---
n: 32
And I also found out, that --report-type full
did print the auditAdvisory
s JSON. But then I didn't find where the error really happened after that.
This then
for example was never called:
https://github.com/IBM/audit-ci/blob/master/lib/yarn-auditer.js#L159-L169
same issue here if I use version 2.5.1.
Yarn audit report results:
events.js:167
throw er; // Unhandled 'error' event
^
Error: Invalid JSON (Unexpected " " at position 4 in state STOP)
if I revert to 2.1.0, another error 😂
@jzsplk Thanks for the report. Can you attach your dependencies or run yarn audit --json
and post it here?
we encountered this problem, turns out the CI environment we're running in (travis) was using npm 5.6.0 which didn't support
audit
yet and was causing misleading JSON error. Forcing the correct npm version resolved the problem for us.
This was the issue for us, using circleci, upgrading our build to use the circleci/node:12.16.1 docker image worked.
Hi, I'm getting the same issue in my Dockerfile. It sometimes works and sometimes fails. Using version 12.16.1
On a side note...
When I run yarn audit --json
it literally runs forever, with yarn audit
I get:
38019 vulnerabilities found - Packages audited: 877285
Severity: 38001 Low | 15 Moderate | 3 High
With the vast majority of them linked to "Validation Bypass". It seems to be a circular dependency. Validation Bypass is a dependency of jest. I'd like to note that this issue with low vulnerabilities with jest has been recurrent.
I also see the same output as @kemicofa (yarn audit --json
spinning forever on jest & its dependencies)
https://github.com/yarnpkg/yarn/issues/7404 seems to be the core issue, with https://github.com/facebook/jest/issues/8682 possibly being the trigger
The advisory that triggers this is https://www.npmjs.com/advisories/1490
I have the same problem I think
audit-ci version: 3.1.1
events.js:183
throw er; // Unhandled 'error' event
^
Error: Invalid JSON (Unexpected "U" at position 0 in state STOP)
at Parser.proto.charError (/home/circleci/test/node_modules/jsonparse/jsonparse.js:90:16)
at Parser.proto.write (/home/circleci/test/node_modules/jsonparse/jsonparse.js:154:23)
at Stream.<anonymous> (/home/circleci/test/node_modules/JSONStream/index.js:23:12)
at Stream.stream.write (/home/circleci/test/node_modules/through/index.js:26:11)
at ReadlineTransform.ondata (_stream_readable.js:639:20)
at emitOne (events.js:116:13)
at ReadlineTransform.emit (events.js:211:7)
at addChunk (_stream_readable.js:263:12)
at readableAddChunk (_stream_readable.js:250:11)
at ReadlineTransform.Readable.push (_stream_readable.js:208:10)
Switching back to my own yarn-audit-ci package to run yarn audit in CI. Does exactly what I need in CI: only blocks the CI on the configured severtity level of vulnerabilities found, in the most lightweight manner possible. You guys might wanna try it out.
P.S. As to this issue, after doing some collaboration on the audit-ci project I belive moving towards parsing a heavy json output was a mistake, and even though we switched to JSONStream at some point it is still not good enough as the number of entries is high. The truth is we don't really care for a detailed output in CI, we can just need a fast go/no-go, and we can then do the investigation locally.
This issue is certainly the biggest blocker to workflows for this project. Using {npm|yarn} audit --json
on a project with many advisories and advisory paths can make the program hang and result in the issues above (to the best of my knowledge, which frankly, I have struggled debugging).
@dubbha, I looked into your package and I found it's approach promising for basic use-cases. If a project only uses Yarn, and doesn't need to support allowlist
ing, it's a sufficient package.
I am open to ideas on how to make this issue less of a blocker. We could, for example, incorporate yarn-audit-ci
's simpler checks as a backup audit if we obtain these kinds of errors and warn the user that the allowlist
was not used. Of course, this would have to be under a flag. However, it could reduce the barrier to audits, support allowlist
in the base case, and break the build less often. A similar functionality should work with npm
as well.
We were running into the same problem as described in the first post of this issue. After some debugging, I found out that the string in the buffer where the JSON decoding fails is <--- Last few GCs --->
. Kinda strange, I was not expecting this string. Anyway, I've upped our --max_old_space_size
and can now run audit-ci without issues.
The side effect of this change is that this process now has a significant memory footprint.
I'm using the version 3.1.1. I've upgraded to jest 26
and I'm facing this issue now. Any idea?
@vctormb Try refreshing your yarn.lock
/package-lock.json
(rm yarn.lock && rm -rf ./node_modules && yarn
). You likely have too many existing transitive dependencies that are outdated that have vulnerabilities.
We have the same issue with different versions of audit-ci
:
node: 12.22.1
& audit-ci: 4.0.0
node: 12.22.1
& audit-ci: 3.2.0
Error message:
$ NODE_OPTIONS=--max_old_space_size=4096 audit-ci --config .audit-ci.json
audit-ci version: 4.0.0
Yarn audit report results:
events.js:291
throw er; // Unhandled 'error' event
^
Error: Invalid JSON (Unexpected " " at position 4 in state STOP)
at Parser.proto.charError (/usr/local/share/.config/yarn/global/node_modules/jsonparse/jsonparse.js:90:16)
where .audit-ci.json:
{
"high": true,
"allowlist": ["serialize-javascript", "axios"]
}
@quinnturner Do you have any ideas on how to solve this problem?
Hi @khaleksa, My current understanding of this problem is that you have too many vulnerabilities when performing npm audit. It might have to do with a circular reference, but I am not sure. Can you post the result of your npm audit? If not publicly, can be emailed :)
My current understanding of this problem is that you have too many vulnerabilities when performing npm audit. It might have to do with a circular reference, but I am not sure. Can you post the result of your npm audit? If not publicly, can be emailed :)
Hi @quinnturner,
I followed your suggestion from the previous comment:
run rm yarn.lock && rm -rf ./node_modules && yarn
before calling audit-ci
It fixed our issue!
Thank you 🙏
Hi @quinnturner, I'm glad that you are offering help on our project with the updated audit-ci
deployment flow.
https://github.com/crypto-com/chain-desktop-wallet/pull/1045
Though we're encountering this issue on our side. I tried to upgrade yarn
to version 2 so that we may use yarn dlx
, but this issue happens:
audit-ci version: 6.2.0
Yarn Berry audit report results:
events.js:377
throw er; // Unhandled 'error' event
^
Error: Invalid JSON (Unexpected "I" at position 0 in state STOP)
Seems running rm yarn.lock && rm -rf ./node_modules && yarn
before calling audit-ci
is not helping on our side. Could you advice us more when you have free time? Thanks.
Hey @crypto-matto, thanks for the report. What's the result when you use this command?
yarn npm audit --recursive --json --all
Hey @crypto-matto, thanks for the report. What's the result when you use this command?
yarn npm audit --recursive --json --all
Hi @quinnturner, thanks for the suggestion. This is the result.
{
"actions":[
],
"advisories":{
"1067407":{
"findings":[
{
"version":"1.14.7",
"paths":[
"axios>follow-redirects",
"@crypto-org-chain/chain-jslib>axios>follow-redirects",
"@cosmjs/stargate>@cosmjs/tendermint-rpc>axios>follow-redirects",
"@crypto-org-chain/chain-jslib>@cosmjs/stargate>@cosmjs/tendermint-rpc>axios>follow-redirects"
]
}
],
"metadata":null,
"vulnerable_versions":"<1.14.8",
"module_name":"follow-redirects",
"severity":"moderate",
"github_advisory_id":"GHSA-pw2r-vq6v-hr8c",
"cves":[
"CVE-2022-0536"
],
"access":"public",
"patched_versions":">=1.14.8",
"cvss":{
"score":5.9,
"vectorString":"CVSS:3.1/AV:N/AC:H/PR:N/UI:N/S:U/C:H/I:N/A:N"
},
"updated":"2022-02-14T22:27:57.000Z",
"recommendation":"Upgrade to version 1.14.8 or later",
"cwe":[
"CWE-200"
],
"found_by":null,
"deleted":null,
"id":1067407,
"references":"- https://nvd.nist.gov/vuln/detail/CVE-2022-0536\n- https://github.com/follow-redirects/follow-redirects/commit/62e546a99c07c3ee5e4e0718c84a6ca127c5c445\n- https://huntr.dev/bounties/7cf2bf90-52da-4d59-8028-a73b132de0db\n- https://github.com/advisories/GHSA-pw2r-vq6v-hr8c",
"created":"2022-02-10T00:00:31.000Z",
"reported_by":null,
"title":"Exposure of Sensitive Information to an Unauthorized Actor in follow-redirects",
"npm_advisory_id":null,
"overview":"Exposure of Sensitive Information to an Unauthorized Actor in NPM follow-redirects prior to 1.14.8.",
"url":"https://github.com/advisories/GHSA-pw2r-vq6v-hr8c"
},
"1067654":{
"findings":[
{
"version":"1.0.2",
"paths":[
"@svgr/webpack>@svgr/plugin-svgo>svgo>css-select>nth-check",
"optimize-css-assets-webpack-plugin>cssnano>cssnano-preset-default>postcss-svgo>svgo>css-select>nth-check"
]
}
],
"metadata":null,
"vulnerable_versions":"<2.0.1",
"module_name":"nth-check",
"severity":"moderate",
"github_advisory_id":"GHSA-rp65-9cf3-cjxr",
"cves":[
"CVE-2021-3803"
],
"access":"public",
"patched_versions":">=2.0.1",
"cvss":{
"score":7.5,
"vectorString":"CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H"
},
"updated":"2021-09-20T20:47:31.000Z",
"recommendation":"Upgrade to version 2.0.1 or later",
"cwe":[
"CWE-1333"
],
"found_by":null,
"deleted":null,
"id":1067654,
"references":"- https://nvd.nist.gov/vuln/detail/CVE-2021-3803\n- https://github.com/fb55/nth-check/commit/9894c1d2010870c351f66c6f6efcf656e26bb726\n- https://huntr.dev/bounties/8cf8cc06-d2cf-4b4e-b42c-99fafb0b04d0\n- https://github.com/advisories/GHSA-rp65-9cf3-cjxr",
"created":"2021-09-20T20:47:31.000Z",
"reported_by":null,
"title":"Inefficient Regular Expression Complexity in nth-check",
"npm_advisory_id":null,
"overview":"nth-check is vulnerable to Inefficient Regular Expression Complexity",
"url":"https://github.com/advisories/GHSA-rp65-9cf3-cjxr"
},
"1067669":{
"findings":[
{
"version":"1.5.2",
"paths":[
"web3",
"@crypto-org-chain/chain-jslib>web3"
]
}
],
"metadata":null,
"vulnerable_versions":"<=1.5.2",
"module_name":"web3",
"severity":"low",
"github_advisory_id":"GHSA-27v7-qhfv-rqq8",
"cves":[
],
"access":"public",
"patched_versions":"<0.0.0",
"cvss":{
"score":3.3,
"vectorString":"CVSS:3.1/AV:L/AC:L/PR:L/UI:N/S:U/C:L/I:N/A:N"
},
"updated":"2021-09-16T20:58:49.000Z",
"recommendation":"None",
"cwe":[
],
"found_by":null,
"deleted":null,
"id":1067669,
"references":"- https://github.com/ethereum/web3.js/issues/2739\n- https://snyk.io/vuln/SNYK-JS-WEB3-174533\n- https://www.npmjs.com/advisories/877\n- https://github.com/advisories/GHSA-27v7-qhfv-rqq8",
"created":"2019-05-30T17:26:30.000Z",
"reported_by":null,
"title":"Insecure Credential Storage in web3",
"npm_advisory_id":null,
"overview":"All versions of `web3` are vulnerable to Insecure Credential Storage. The package stores encrypted wallets in local storage and requires a password to load the wallet. Once the wallet is loaded, the private key is accessible via LocalStorage. Exploiting this vulnerability likely requires a Cross-Site Scripting vulnerability to access the private key.\n\n\n## Recommendation\n\nNo fix is currently available. Consider using an alternative module until a fix is made available.",
"url":"https://github.com/advisories/GHSA-27v7-qhfv-rqq8"
},
"1067696":{
"findings":[
{
"version":"2.0.0",
"paths":[
"husky>find-versions>semver-regex"
]
}
],
"metadata":null,
"vulnerable_versions":"<3.1.3",
"module_name":"semver-regex",
"severity":"moderate",
"github_advisory_id":"GHSA-44c6-4v22-4mhx",
"cves":[
"CVE-2021-3795"
],
"access":"public",
"patched_versions":">=3.1.3",
"cvss":{
"score":7.5,
"vectorString":"CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H"
},
"updated":"2021-09-20T20:42:25.000Z",
"recommendation":"Upgrade to version 3.1.3 or later",
"cwe":[
"CWE-400"
],
"found_by":null,
"deleted":null,
"id":1067696,
"references":"- https://nvd.nist.gov/vuln/detail/CVE-2021-3795\n- https://github.com/sindresorhus/semver-regex/commit/11c66245f4e1976dccc52977ed183696a21a3fd7\n- https://huntr.dev/bounties/006624e3-35ac-448f-aab9-7b5183f30e28\n- https://github.com/advisories/GHSA-44c6-4v22-4mhx",
"created":"2021-09-20T20:42:25.000Z",
"reported_by":null,
"title":"Regular Expression Denial of Service (ReDOS)",
"npm_advisory_id":null,
"overview":"npm `semver-regex` is vulnerable to Inefficient Regular Expression Complexity",
"url":"https://github.com/advisories/GHSA-44c6-4v22-4mhx"
},
"1067832":{
"findings":[
{
"version":"7.0.21",
"paths":[
"postcss-safe-parser>postcss",
"stylelint>postcss-less>postcss",
"stylelint-config-css-modules>stylelint>postcss-less>postcss",
"@umijs/fabric>stylelint-config-css-modules>stylelint>postcss-less>postcss",
"@umijs/fabric>stylelint-config-rational-order>stylelint-order>stylelint>postcss-less>postcss",
"@umijs/fabric>stylelint-config-css-modules>stylelint>@stylelint/postcss-css-in-js>postcss-syntax>postcss-less>postcss",
"@umijs/fabric>stylelint-config-rational-order>stylelint-order>stylelint>@stylelint/postcss-css-in-js>postcss-syntax>postcss-less>postcss"
]
}
],
"metadata":null,
"vulnerable_versions":">=7.0.0 <7.0.36",
"module_name":"postcss",
"severity":"moderate",
"github_advisory_id":"GHSA-hwj9-h5mp-3pm3",
"cves":[
"CVE-2021-23368"
],
"access":"public",
"patched_versions":">=7.0.36",
"cvss":{
"score":5.3,
"vectorString":"CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:L"
},
"updated":"2021-06-15T15:12:14.000Z",
"recommendation":"Upgrade to version 7.0.36 or later",
"cwe":[
"CWE-400"
],
"found_by":null,
"deleted":null,
"id":1067832,
"references":"- https://nvd.nist.gov/vuln/detail/CVE-2021-23368\n- https://github.com/postcss/postcss/commit/8682b1e4e328432ba692bed52326e84439cec9e4\n- https://github.com/postcss/postcss/commit/b6f3e4d5a8d7504d553267f80384373af3a3dec5\n- https://lists.apache.org/thread.html/r00158f5d770d75d0655c5eef1bdbc6150531606c8f8bcb778f0627be@%3Cdev.myfaces.apache.org%3E\n- https://lists.apache.org/thread.html/r16e295b4f02d81b79981237d602cb0b9e59709bafaa73ac98be7cef1@%3Cdev.myfaces.apache.org%3E\n- https://lists.apache.org/thread.html/r49afb49b38748897211b1f89c3a64dc27f9049474322b05715695aab@%3Cdev.myfaces.apache.org%3E\n- https://lists.apache.org/thread.html/r5acd89f3827ad9a9cad6d24ed93e377f7114867cd98cfba616c6e013@%3Ccommits.myfaces.apache.org%3E\n- https://lists.apache.org/thread.html/r8def971a66cf3e375178fbee752e1b04a812a047cc478ad292007e33@%3Cdev.myfaces.apache.org%3E\n- https://lists.apache.org/thread.html/rad5af2044afb51668b1008b389ac815a28ecea9eb75ae2cab5a00ebb@%3Ccommits.myfaces.apache.org%3E\n- https://snyk.io/vuln/SNYK-JAVA-ORGWEBJARSNPM-1244795\n- https://snyk.io/vuln/SNYK-JS-POSTCSS-1090595\n- https://github.com/postcss/postcss/commit/54cbf3c4847eb0fb1501b9d2337465439e849734\n- https://github.com/advisories/GHSA-hwj9-h5mp-3pm3",
"created":"2021-05-10T15:29:24.000Z",
"reported_by":null,
"title":"Regular Expression Denial of Service in postcss",
"npm_advisory_id":null,
"overview":"The npm package `postcss` from 7.0.0 and before versions 7.0.36 and 8.2.10 is vulnerable to Regular Expression Denial of Service (ReDoS) during source map parsing.",
"url":"https://github.com/advisories/GHSA-hwj9-h5mp-3pm3"
},
"1070012":{
"findings":[
{
"version":"7.0.21",
"paths":[
"postcss-safe-parser>postcss",
"stylelint>postcss-less>postcss",
"stylelint-config-css-modules>stylelint>postcss-less>postcss",
"@umijs/fabric>stylelint-config-css-modules>stylelint>postcss-less>postcss",
"@umijs/fabric>stylelint-config-rational-order>stylelint-order>stylelint>postcss-less>postcss",
"@umijs/fabric>stylelint-config-css-modules>stylelint>@stylelint/postcss-css-in-js>postcss-syntax>postcss-less>postcss",
"@umijs/fabric>stylelint-config-rational-order>stylelint-order>stylelint>@stylelint/postcss-css-in-js>postcss-syntax>postcss-less>postcss"
]
}
],
"metadata":null,
"vulnerable_versions":"<7.0.36",
"module_name":"postcss",
"severity":"moderate",
"github_advisory_id":"GHSA-566m-qj78-rww5",
"cves":[
"CVE-2021-23382"
],
"access":"public",
"patched_versions":">=7.0.36",
"cvss":{
"score":5.3,
"vectorString":"CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:L"
},
"updated":"2022-04-19T19:03:24.000Z",
"recommendation":"Upgrade to version 7.0.36 or later",
"cwe":[
"CWE-400"
],
"found_by":null,
"deleted":null,
"id":1070012,
"references":"- https://nvd.nist.gov/vuln/detail/CVE-2021-23382\n- https://github.com/postcss/postcss/commit/2b1d04c867995e55124e0a165b7c6622c1735956\n- https://snyk.io/vuln/SNYK-JAVA-ORGWEBJARSNPM-1255641\n- https://snyk.io/vuln/SNYK-JS-POSTCSS-1255640\n- https://github.com/postcss/postcss/releases/tag/7.0.36\n- https://github.com/advisories/GHSA-566m-qj78-rww5",
"created":"2022-01-07T00:21:36.000Z",
"reported_by":null,
"title":"Regular Expression Denial of Service in postcss",
"npm_advisory_id":null,
"overview":"The package postcss versions before 7.0.36 or between 8.0.0 and 8.2.13 are vulnerable to Regular Expression Denial of Service (ReDoS) via getAnnotationURL() and loadAnnotation() in lib/previous-map.js. The vulnerable regexes are caused mainly by the sub-pattern \\/\\*\\s* sourceMappingURL=(.*).",
"url":"https://github.com/advisories/GHSA-566m-qj78-rww5"
}
},
"muted":[
],
"metadata":{
"vulnerabilities":{
"info":0,
"low":2,
"moderate":21,
"high":0,
"critical":0
},
"dependencies":1064,
"devDependencies":894,
"optionalDependencies":0,
"totalDependencies":1958
}
}
After I upgraded yarn
with 3.0.2
and modify the package.json
specifying the non-vulnerable versions, I'm finally able to get this:
audit-ci version: 6.2.0
Yarn Berry audit report results:
{
"vulnerabilities": {
"info": 0,
"low": 0,
"moderate": 0,
"high": 0,
"critical": 0
},
"dependencies": 1064,
"devDependencies": 894,
"optionalDependencies": 0,
"totalDependencies": [19](https://github.com/crypto-com/chain-desktop-wallet/runs/6501364579?check_suite_focus=true#step:8:23)58
}
Passed yarn security audit.
Yet I'm facing a lot of other issues when dealing with migrating yarn 1
to yarn 3
. I can't run the dev command properly on my local project neither. You may check them out.
https://github.com/crypto-com/chain-desktop-wallet/pull/1045
Due to: https://github.com/IBM/audit-ci/issues/96
(It's visible here: https://travis-ci.org/sumup/circuit-ui/jobs/559411807)
Related: #102