search

eclipse-dash / dash-licenses

Extract license information from content.
http://projects.eclipse.org/projects/technology.dash
Eclipse Public License 2.0
47 stars 33 forks source link

Automate filing third-party content review requests #51

Closed mickaelistria closed 2 years ago

mickaelistria commented 3 years ago

I happily run dash-license tool and get a clear and useful output

$ java -jar ~/git/dash-licenses/core/target/org.eclipse.dash.licenses-0.0.1-SNAPSHOT.jar package-lock.json 
Feb 02, 2021 10:29:29 AM org.eclipse.dash.licenses.foundation.EclipseFoundationSupport queryLicenseData
INFO: Querying Eclipse Foundation for license data for 145 items.
Feb 02, 2021 10:29:30 AM org.eclipse.dash.licenses.foundation.EclipseFoundationSupport queryLicenseData
INFO: Found 71 items.
Feb 02, 2021 10:29:30 AM org.eclipse.dash.licenses.clearlydefined.ClearlyDefinedSupport queryLicenseData
INFO: Querying ClearlyDefined for license data for 74 items.
Feb 02, 2021 10:29:32 AM org.eclipse.dash.licenses.clearlydefined.ClearlyDefinedSupport queryLicenseData
INFO: Found 74 items.
License information could not be automatically verified for the following content:

npm/npmjs/-/copy-anything/2.0.1
npm/npmjs/-/less/3.13.1
npm/npmjs/-/sass/1.32.6
npm/npmjs/-/typescript/4.1.3
npm/npmjs/@angular/language-service/11.1.1

This content is either not correctly mapped by the system, or requires review.

Now, I'd like the extra step forward, just like pushing on GitHub suggests a link to directly create a pull request: a prepopulated link to open a CQ for those

waynebeaton commented 3 years ago

Now, I'd like the extra step forward, just like pushing on GitHub suggests a link to directly create a pull request: a prepopulated link to open a CQ for those

I have good news. I've actually already committed the code that does this (0ffba5e798b479157953d9b507b6dcb80d17aa44).

It doesn't actually create a CQ; rather, it creates a GitLab Issue (e.g.). I haven't yet written down how to actually use the feature yet, primarily because I'm not quite ready yet for the deluge of requests that might come (we haven't written the backend code yet, so processing these is entirely manual).

It's still all very experimental, so there will be changes. But it should get the job done for your immediate need.

Short version:

Please do not run this if you have more than dozen or so dependencies identified as requiring review. At least not yet.

mickaelistria commented 3 years ago

Great! Then I suggest it's mostly a matter of adding such info in the standard output to make it more accessible. I'll make a PR that copies your comment here.

mickaelistria commented 3 years ago

Submitted https://github.com/eclipse/dash-licenses/pull/53 to help people leveraging this great capability.

waynebeaton commented 3 years ago

Great! Then I suggest it's mostly a matter of adding such info in the standard output to make it more accessible. I'll make a PR that copies your comment here.

Actually... it's mostly a matter of finding cycles to put some automation support on the backend before somebody runs it on their list of 4,000 NPM dependencies.

At this point, my primary interest is whether or not this actually works for you. There are no new issues in the GitLab repository. Did you try it?

I haven't fully tested the privileges on the repository. Ideally the ability to create issues should be available (and limited) to all committers.

mickaelistria commented 3 years ago

I already opened the CQ yesterday, so I didn't use it.

waynebeaton commented 3 years ago

Try it anyway. It would be helpful.

mickaelistria commented 3 years ago

This works like a charm!

$ java -jar ~/git/dash-licenses/core/target/org.eclipse.dash.licenses-0.0.1-SNAPSHOT.jar -project technology.wildwebdeveloper -review -token XXXXXXXXXXXX package-lock.json 
Feb 03, 2021 2:15:44 PM org.eclipse.dash.licenses.foundation.EclipseFoundationSupport queryLicenseData
INFO: Querying Eclipse Foundation for license data for 32 items.
Feb 03, 2021 2:15:45 PM org.eclipse.dash.licenses.foundation.EclipseFoundationSupport queryLicenseData
INFO: Found 11 items.
Feb 03, 2021 2:15:45 PM org.eclipse.dash.licenses.clearlydefined.ClearlyDefinedSupport queryLicenseData
INFO: Querying ClearlyDefined for license data for 21 items.
Feb 03, 2021 2:15:46 PM org.eclipse.dash.licenses.clearlydefined.ClearlyDefinedSupport queryLicenseData
INFO: Found 21 items.
License information could not be automatically verified for the following content:

npm/npmjs/-/copy-anything/2.0.1
npm/npmjs/-/less/3.13.1
npm/npmjs/-/sass/1.32.6
npm/npmjs/-/typescript/4.1.3
npm/npmjs/@angular/language-service/11.1.1

This content is either not correctly mapped by the system, or requires review.

Setting up a review for npm/npmjs/-/less/3.13.1.
 - Created: https://gitlab.eclipse.org/eclipsefdn/iplab/iplab/-/issues/22
Setting up a review for npm/npmjs/-/typescript/4.1.3.
 - Created: https://gitlab.eclipse.org/eclipsefdn/iplab/iplab/-/issues/23
Setting up a review for npm/npmjs/-/copy-anything/2.0.1.
 - Created: https://gitlab.eclipse.org/eclipsefdn/iplab/iplab/-/issues/24
Setting up a review for npm/npmjs/@angular/language-service/11.1.1.
 - Created: https://gitlab.eclipse.org/eclipsefdn/iplab/iplab/-/issues/25
Setting up a review for npm/npmjs/-/sass/1.32.6.
 - Created: https://gitlab.eclipse.org/eclipsefdn/iplab/iplab/-/issues/26
waynebeaton commented 3 years ago

I'm going to leave this issue open to provide a place to discuss this functionality while we evolve the implementation.

waynebeaton commented 3 years ago

Note that I've added some code in place to throttle creation to five requests. My intention is to open this up, but only after we have some proper automation supporting the backend.

marcdumais-work commented 3 years ago

before somebody runs it on their list of 4,000 NPM dependencies.

hehe :)

Please do not incorporate this feature into your automated builds at this time

Please ping me when you think this is ready for use in a "friendly" project's CI. In the meantime I'll consider adding it as an information step, that scans and informs about suspicious dependencies, but does not automatically submits them for review.

waynebeaton commented 3 years ago

before somebody runs it on their list of 4,000 NPM dependencies.

hehe :)

Don't laugh too hard; I was thinking of you when I typed that. :-)

marcdumais-work commented 3 years ago

As an experiment I did a run, using Theia's yarn.lock directly (like Example: Yarn via yarn.lock), instead of using the helper tool (like Example: Yarn via yarn) or using synp to translate yarn.lock to package-lock.json, and using that as input.

This new run seem to mis-parse packages that start with a @, like the @babel/* bunch and many others. dash-licenses seem to consider them separate packages, form the already known ones that properly start with a @. Looking at automatically opened issues (full list below), I notice that:

Full results:

License information could not be automatically verified for the following content:

npm/npmjs/-/ajv/6.12.6 npm/npmjs/-/autoprefixer/6.7.7 npm/npmjs/-/babel-polyfill/6.26.0 npm/npmjs/-/big.js/3.2.0 npm/npmjs/-/coa/1.0.4 npm/npmjs/-/eslint-plugin-deprecation/1.2.1 npm/npmjs/-/esprima/4.0.1 npm/npmjs/-/esquery/1.4.0 npm/npmjs/-/extsprintf/1.4.0 npm/npmjs/-/from/0.1.7 npm/npmjs/-/fs-extra/4.0.3 npm/npmjs/-/gauge/2.7.4 npm/npmjs/-/gitconfiglocal/1.0.0 npm/npmjs/-/glob/7.1.3 npm/npmjs/-/js-yaml/3.7.0 npm/npmjs/-/jschardet/2.3.0 npm/npmjs/-/jsdom/11.12.0 npm/npmjs/-/jsmin/1.0.1 npm/npmjs/-/json-schema/0.2.3 npm/npmjs/-/json5/0.5.1 npm/npmjs/-/less-loader/2.2.3 npm/npmjs/-/npmlog/4.1.2 npm/npmjs/-/oniguruma/7.2.3 npm/npmjs/-/parse-json/2.2.0 npm/npmjs/-/postcss-reduce-initial/1.0.1 npm/npmjs/-/q/1.5.1 npm/npmjs/-/rc/1.2.8 npm/npmjs/-/readable-stream/2.3.7 npm/npmjs/-/rechoir/0.6.2 npm/npmjs/-/rx/2.3.24 npm/npmjs/-/shelljs/0.8.4 npm/npmjs/-/source-list-map/2.0.1 npm/npmjs/-/spdx-correct/3.1.1 npm/npmjs/-/spdx-license-ids/3.0.9 npm/npmjs/-/style-loader/0.13.2 npm/npmjs/-/through/2.3.8 npm/npmjs/-/ts-md5/1.2.9 npm/npmjs/-/tweetnacl/0.14.5 npm/npmjs/-/typescript/3.9.10 npm/npmjs/-/uc.micro/1.0.6 npm/npmjs/-/uri-js/4.4.1 npm/npmjs/-/wcwidth/1.0.1 npm/npmjs/babel/code-frame/7.12.11 npm/npmjs/babel/code-frame/7.14.5 npm/npmjs/babel/compat-data/7.14.7 npm/npmjs/babel/core/7.14.8 npm/npmjs/babel/generator/7.14.8 npm/npmjs/babel/helper-annotate-as-pure/7.14.5 npm/npmjs/babel/helper-builder-binary-assignment-operator-visitor/7.14.5 npm/npmjs/babel/helper-compilation-targets/7.14.5 npm/npmjs/babel/helper-create-class-features-plugin/7.14.8 npm/npmjs/babel/helper-create-regexp-features-plugin/7.14.5 npm/npmjs/babel/helper-define-polyfill-provider/0.2.3 npm/npmjs/babel/helper-explode-assignable-expression/7.14.5 npm/npmjs/babel/helper-function-name/7.14.5 npm/npmjs/babel/helper-get-function-arity/7.14.5 npm/npmjs/babel/helper-hoist-variables/7.14.5 npm/npmjs/babel/helper-member-expression-to-functions/7.14.7 npm/npmjs/babel/helper-module-imports/7.14.5 npm/npmjs/babel/helper-module-transforms/7.14.8 npm/npmjs/babel/helper-optimise-call-expression/7.14.5 npm/npmjs/babel/helper-plugin-utils/7.14.5 npm/npmjs/babel/helper-remap-async-to-generator/7.14.5 npm/npmjs/babel/helper-replace-supers/7.14.5 npm/npmjs/babel/helper-simple-access/7.14.8 npm/npmjs/babel/helper-skip-transparent-expression-wrappers/7.14.5 npm/npmjs/babel/helper-split-export-declaration/7.14.5 npm/npmjs/babel/helper-validator-identifier/7.14.8 npm/npmjs/babel/helper-validator-option/7.14.5 npm/npmjs/babel/helper-wrap-function/7.14.5 npm/npmjs/babel/helpers/7.14.8 npm/npmjs/babel/highlight/7.14.5 npm/npmjs/babel/parser/7.14.8 npm/npmjs/babel/plugin-bugfix-v8-spread-parameters-in-optional-chaining/7.14.5 npm/npmjs/babel/plugin-proposal-async-generator-functions/7.14.7 npm/npmjs/babel/plugin-proposal-class-properties/7.14.5 npm/npmjs/babel/plugin-proposal-class-static-block/7.14.5 npm/npmjs/babel/plugin-proposal-dynamic-import/7.14.5 npm/npmjs/babel/plugin-proposal-export-namespace-from/7.14.5 npm/npmjs/babel/plugin-proposal-json-strings/7.14.5 npm/npmjs/babel/plugin-proposal-logical-assignment-operators/7.14.5 npm/npmjs/babel/plugin-proposal-nullish-coalescing-operator/7.14.5 npm/npmjs/babel/plugin-proposal-numeric-separator/7.14.5 npm/npmjs/babel/plugin-proposal-object-rest-spread/7.14.7 npm/npmjs/babel/plugin-proposal-optional-catch-binding/7.14.5 npm/npmjs/babel/plugin-proposal-optional-chaining/7.14.5 npm/npmjs/babel/plugin-proposal-private-methods/7.14.5 npm/npmjs/babel/plugin-proposal-private-property-in-object/7.14.5 npm/npmjs/babel/plugin-proposal-unicode-property-regex/7.14.5 npm/npmjs/babel/plugin-syntax-class-properties/7.12.13 npm/npmjs/babel/plugin-syntax-class-static-block/7.14.5 npm/npmjs/babel/plugin-syntax-dynamic-import/7.8.3 npm/npmjs/babel/plugin-syntax-export-namespace-from/7.8.3 npm/npmjs/babel/plugin-syntax-private-property-in-object/7.14.5 npm/npmjs/babel/plugin-syntax-top-level-await/7.14.5 npm/npmjs/babel/plugin-transform-arrow-functions/7.14.5 npm/npmjs/babel/plugin-transform-async-to-generator/7.14.5 npm/npmjs/babel/plugin-transform-block-scoped-functions/7.14.5 npm/npmjs/babel/plugin-transform-block-scoping/7.14.5 npm/npmjs/babel/plugin-transform-classes/7.14.5 npm/npmjs/babel/plugin-transform-computed-properties/7.14.5 npm/npmjs/babel/plugin-transform-destructuring/7.14.7 npm/npmjs/babel/plugin-transform-dotall-regex/7.14.5 npm/npmjs/babel/plugin-transform-duplicate-keys/7.14.5 npm/npmjs/babel/plugin-transform-exponentiation-operator/7.14.5 npm/npmjs/babel/plugin-transform-for-of/7.14.5 npm/npmjs/babel/plugin-transform-function-name/7.14.5 npm/npmjs/babel/plugin-transform-literals/7.14.5 npm/npmjs/babel/plugin-transform-member-expression-literals/7.14.5 npm/npmjs/babel/plugin-transform-modules-amd/7.14.5 npm/npmjs/babel/plugin-transform-modules-commonjs/7.14.5 npm/npmjs/babel/plugin-transform-modules-systemjs/7.14.5 npm/npmjs/babel/plugin-transform-modules-umd/7.14.5 npm/npmjs/babel/plugin-transform-named-capturing-groups-regex/7.14.7 npm/npmjs/babel/plugin-transform-new-target/7.14.5 npm/npmjs/babel/plugin-transform-object-super/7.14.5 npm/npmjs/babel/plugin-transform-parameters/7.14.5 npm/npmjs/babel/plugin-transform-property-literals/7.14.5 npm/npmjs/babel/plugin-transform-regenerator/7.14.5 npm/npmjs/babel/plugin-transform-reserved-words/7.14.5 npm/npmjs/babel/plugin-transform-runtime/7.14.5 npm/npmjs/babel/plugin-transform-shorthand-properties/7.14.5 npm/npmjs/babel/plugin-transform-spread/7.14.6 npm/npmjs/babel/plugin-transform-sticky-regex/7.14.5 npm/npmjs/babel/plugin-transform-template-literals/7.14.5 npm/npmjs/babel/plugin-transform-typeof-symbol/7.14.5 npm/npmjs/babel/plugin-transform-unicode-escapes/7.14.5 npm/npmjs/babel/plugin-transform-unicode-regex/7.14.5 npm/npmjs/babel/preset-env/7.14.8 npm/npmjs/babel/preset-modules/0.1.4 npm/npmjs/babel/runtime/7.14.8 npm/npmjs/babel/template/7.14.5 npm/npmjs/babel/traverse/7.14.8 npm/npmjs/babel/types/7.14.8 npm/npmjs/dabh/diagnostics/2.0.2 npm/npmjs/discoveryjs/json-ext/0.5.3 npm/npmjs/electron/get/1.12.4 npm/npmjs/eslint/eslintrc/0.4.3 npm/npmjs/humanwhocodes/config-array/0.5.0 npm/npmjs/humanwhocodes/object-schema/1.2.0 npm/npmjs/improved/node/1.1.1 npm/npmjs/nodelib/fs.scandir/2.1.5 npm/npmjs/nodelib/fs.stat/2.0.5 npm/npmjs/nodelib/fs.walk/1.2.8 npm/npmjs/phosphor/algorithm/1.2.0 npm/npmjs/phosphor/collections/1.2.0 npm/npmjs/phosphor/commands/1.7.2 npm/npmjs/phosphor/coreutils/1.3.1 npm/npmjs/phosphor/disposable/1.3.1 npm/npmjs/phosphor/domutils/1.1.4 npm/npmjs/phosphor/dragdrop/1.4.1 npm/npmjs/phosphor/keyboard/1.1.3 npm/npmjs/phosphor/messaging/1.3.0 npm/npmjs/phosphor/properties/1.1.3 npm/npmjs/phosphor/signaling/1.3.1 npm/npmjs/phosphor/virtualdom/1.2.0 npm/npmjs/phosphor/widgets/1.9.3 npm/npmjs/primer/octicons-react/9.6.0 npm/npmjs/sindresorhus/df/1.0.1 npm/npmjs/sindresorhus/df/2.1.0 npm/npmjs/sindresorhus/is/0.14.0 npm/npmjs/stroncium/procfs/1.2.1 npm/npmjs/szmarczak/http-timer/1.1.2 npm/npmjs/theia/compression-webpack-plugin/3.0.0 npm/npmjs/theia/monaco-editor-core/0.23.0 npm/npmjs/theia/node-pty/0.9.0-theia.6 npm/npmjs/types/base64-arraybuffer/0.1.0 npm/npmjs/types/bent/7.3.2 npm/npmjs/types/caseless/0.12.2 npm/npmjs/types/chai-string/1.4.2 npm/npmjs/types/chai/4.2.21 npm/npmjs/types/cookie/0.3.3 npm/npmjs/types/decompress/4.2.4 npm/npmjs/types/diff/3.5.4 npm/npmjs/types/dompurify/2.2.3 npm/npmjs/types/escape-html/0.0.20 npm/npmjs/types/eslint-scope/3.7.1 npm/npmjs/types/eslint/7.28.0 npm/npmjs/types/estree/0.0.50 npm/npmjs/types/fs-extra/4.0.12 npm/npmjs/types/highlight.js/10.1.0 npm/npmjs/types/highlight.js/9.12.4 npm/npmjs/types/jsdom/11.12.0 npm/npmjs/types/linkify-it/3.0.2 npm/npmjs/types/lodash.clonedeep/4.5.6 npm/npmjs/types/lodash.debounce/4.0.3 npm/npmjs/types/lodash.throttle/4.1.6 npm/npmjs/types/lodash/4.14.171 npm/npmjs/types/markdown-it-anchor/4.0.4 npm/npmjs/types/markdown-it/12.0.3 npm/npmjs/types/mdurl/1.0.2 npm/npmjs/types/mime-types/2.1.0 npm/npmjs/types/mime/1.3.2 npm/npmjs/types/minipass/2.2.1 npm/npmjs/types/mkdirp/0.5.2 npm/npmjs/types/mocha/5.2.7 npm/npmjs/types/node/12.20.17 npm/npmjs/types/p-queue/2.3.2 npm/npmjs/types/proxy-from-env/1.0.1 npm/npmjs/types/ps-tree/1.1.1 npm/npmjs/types/puppeteer/2.1.6 npm/npmjs/types/react-dom/16.9.14 npm/npmjs/types/react-virtualized/9.21.12 npm/npmjs/types/react/16.14.11 npm/npmjs/types/react/17.0.15 npm/npmjs/types/request/2.48.7 npm/npmjs/types/requestretry/1.12.8 npm/npmjs/types/rimraf/2.0.5 npm/npmjs/types/route-parser/0.1.3 npm/npmjs/types/safer-buffer/2.1.0 npm/npmjs/types/sanitize-html/2.3.2 npm/npmjs/types/scheduler/0.16.2 npm/npmjs/types/semver/5.5.0 npm/npmjs/types/showdown/1.9.4 npm/npmjs/types/sinon/2.3.7 npm/npmjs/types/tar-fs/1.16.3 npm/npmjs/types/tar-stream/2.2.1 npm/npmjs/types/tar/4.0.5 npm/npmjs/types/temp/0.8.34 npm/npmjs/types/tough-cookie/4.0.1 npm/npmjs/types/trusted-types/2.0.2 npm/npmjs/types/uuid/7.0.5 npm/npmjs/types/which/1.3.2 npm/npmjs/types/write-json-file/2.2.1 npm/npmjs/types/ws/5.1.2 npm/npmjs/types/yargs-parser/20.2.1 npm/npmjs/typescript-eslint/eslint-plugin-tslint/4.28.5 npm/npmjs/typescript-eslint/eslint-plugin/4.28.5 npm/npmjs/typescript-eslint/experimental-utils/3.10.1 npm/npmjs/typescript-eslint/experimental-utils/4.28.5 npm/npmjs/typescript-eslint/parser/4.28.5 npm/npmjs/typescript-eslint/scope-manager/4.28.5 npm/npmjs/typescript-eslint/types/3.10.1 npm/npmjs/typescript-eslint/types/4.28.5 npm/npmjs/typescript-eslint/typescript-estree/3.10.1 npm/npmjs/typescript-eslint/typescript-estree/4.28.5 npm/npmjs/typescript-eslint/visitor-keys/3.10.1 npm/npmjs/typescript-eslint/visitor-keys/4.28.5 npm/npmjs/vscode/codicons/0.0.21 npm/npmjs/webassemblyjs/ast/1.11.1 npm/npmjs/webassemblyjs/floating-point-hex-parser/1.11.1 npm/npmjs/webassemblyjs/helper-api-error/1.11.1 npm/npmjs/webassemblyjs/helper-buffer/1.11.1 npm/npmjs/webassemblyjs/helper-numbers/1.11.1 npm/npmjs/webassemblyjs/helper-wasm-bytecode/1.11.1 npm/npmjs/webassemblyjs/helper-wasm-section/1.11.1 npm/npmjs/webassemblyjs/ieee754/1.11.1 npm/npmjs/webassemblyjs/leb128/1.11.1 npm/npmjs/webassemblyjs/utf8/1.11.1 npm/npmjs/webassemblyjs/wasm-edit/1.11.1 npm/npmjs/webassemblyjs/wasm-gen/1.11.1 npm/npmjs/webassemblyjs/wasm-opt/1.11.1 npm/npmjs/webassemblyjs/wasm-parser/1.11.1 npm/npmjs/webassemblyjs/wast-printer/1.11.1 npm/npmjs/webpack-cli/configtest/1.0.4 npm/npmjs/webpack-cli/info/1.3.0 npm/npmjs/webpack-cli/serve/1.5.1

This content is either not correctly mapped by the system, or requires review.

Setting up a review for npm/npmjs/babel/helper-hoist-variables/7.14.5.

More content needs to be reviewed. For now, however, this experimental feature only submits the first 10.

waynebeaton commented 3 years ago

This new run seem to mis-parse packages that start with a @, like the @babel/* bunch and many others. dash-licenses seem to consider them separate packages, form the already known ones that properly start with a @.

It's been a while since I wrote that code, but I swear that I observed the ClearlyDefined ids omitted the '@'... Regardless, that does not appear to be the case. I'll fix it (and the issues that were created).

Thanks for point this out.

waynebeaton commented 3 years ago

I've pushed an update and have kicked off a build. The update should be reflected in repo.eclipse.org in a few minutes.

marcdumais-work commented 3 years ago

Awesome! I'll try this offline first

marcdumais-work commented 3 years ago

I've pushed an update and have kicked off a build. The update should be reflected in repo.eclipse.org in a few minutes.

Confirmed. I now obtain exactly the same "unverified dependencies" list locally, using yarn.lock and using package-lock.json generated from yarn.lock using synp.

mickaelistria commented 2 years ago

This is alive and working.