Evaluate open source scanning tools

kjw000 commented 5 years ago

Goal:

As a contributor I want to understand what type of source code is being used and evaluate several options.

Tasks:

[x] Pick a couple of tools and perform high level evaluation
[x] Pilot 2-3 top tools
[x] Present recommendations to team

Acceptance Criteria:

[ ] Documentation is updated
[ ] Tool meets the needs

Pull Requests:

[ ] TBD

Follow-up:

N/A

Dependencies:

N/A

Accountability:

Owner: TBC
QA/Review: TBC

kjw000 commented 5 years ago

Snyk, SourceClear and OWSAP going to be used as pilot. Not only OWSAP is free but it does not have license monitoring.

HenkKodde commented 5 years ago

Snyk - license reporting/monitoring is not free. Sourceclear - Free trial only for 30 days.... License module only included with Business plan.

kjw000 commented 5 years ago

agree - we have some results and rather go with an OSS tool.

lewisdaly commented 5 years ago

@kjw000 I spent a bit of time evaluating some of the other open source options.

It's not 100% clear to me what we need this tool for. Is it some combination (or all of):

Static code analysis
Dependency and vulnerability scanning
OSS License Scanning to ensure we don't violate the conditions of the dependent licenses

I noticed that SonarCloud was already set up for this project, which does static analysis as well as dependency scanning, but it didn't find any issues with the ml-api-adapter project.

Another option that may have been overlooked is npm audit. It's built in to the latest npm versions, and has a JSON output which I think can be fed into CircleCI. We can run npm audit as a part of the CircleCI build steps that get run (and need to pass) before a PR will be accepted.

For scanning os licences, there were a few open source tools I found that seemed very popular and well maintained on Github:

ScanCode toolkit: https://github.com/nexB/scancode-toolkit
Fossology: https://github.com/fossology/fossology
OSS Review Toolkit: https://github.com/heremaps/oss-review-toolkit

I'll be evaluating these shortly.

kjw000 commented 5 years ago

We want an open source code scanner tech that can automate some of the preventive scanning of contributions - ensure they are fine before merge or part of ci/cd. If we can also do license scanning in addition to static code analysis that would be a good. It is great if we can use OS tools but should not limit ourselves as many commercial tools have OS free licenses. SonarCloud - not sure who set this up, but my guess we installed it with SonarQube which we use quite a bit. Thanks

lewisdaly commented 5 years ago

Dependency + Static Analysis

SonarCloud is the Cloud version of SonarQube, and it seems to do a good job at static analysis, plus it should be pretty easy to integrate into our CI workflow since we already have it mostly set up.

Example from SonarCloud:

I also took a look at DeepScan, but that didn't do as good a job as SonarCloud at picking up potential bugs (it only found a few poorly handled exceptions).

Example from DeepScan:

Snyk seems to only do dependency analysis, and not actual code analysis (someone correct me if I'm wrong), and doesn't seem to find as many issues as a simple npm audit finds from the command line.

GuardRails, which is free for Open Source projects (and PRs only) didn't find any security issues that other tools picked up.

Another option for managing and reviewing security issues for dependencies is dependabot, which just got aquired by GitHub itself. It takes a bit of a different approach; instead of running a check on out of date dependencies or dependencies with security issues during a build, dependabot will just submit pull requests to update the given packages when they are out of date.

License Analysis

As for license analysis, I haven't been able to find a suitable cloud-based tool that will suit out needs. Additionally, the command-line based tools aren't that user friendly or easy to reason about (esp. for non technical people that may want to understand the licenses being used).

FOSSA seems really impressive, but it only supports up to 5 free projects, and then the paid plans are per developer, starting at 5 developers for $230/month. The free scanning is also quite lightweight, and you need the paid plan to do what they call a 'deeper' scan.

I've been playing around with other tools as well:

scancode-toolkit, which listed all of the licenses and copyrights, but didn't tell me anything interesting or actionable about them
license-checker, a simple npm tool that prints out the licenses for the node dependencies. We could couple this with some simple scripts that check for blacklisted/unknown license types and raise warnings, but that's less than ideal.

My Recommendations

Each of these tools can be integrated nto the CI workflow for a go/no go when building code for PRs

Use SonarCloud/SonarCube for the static code analysis
Use npm audit for analysing dependencies and ensuring there are no vulnerabilities being introduced into the codebase. We can combine this with dependabot for automatically updating the out of date packages for us.
I'm still looking into license analysis tools

lewisdaly commented 5 years ago

I spent some more time investigating open source analysis tools. Unfortunately there is a big gap between FOSSA and other open source cli-based tools.

I evaluated a bunch of npm modules, and the best two were:

license-checker and nlf (node license scanner). They both export to CSV files, and have failure conditions by specifying licenses that we don't want in the project, which will help with integrating into CI tools later on.

I also discovered fossa-cli, which I think may be a free version of Fossa (I got in touch with their support), that gets around the 5 project limit. Instead of linking fossa to a github repo, we just provide it with an API key and run the fossa cli locally, which then generates a nice report of all the dependencies.

Here's some examples:

$ license-checker --production --csv     
"module name","license","repository"
"@babel/polyfill@7.4.3","MIT","https://github.com/babel/babel/tree/master/packages/babel-polyfill"
"@mojaloop/central-services-database@5.2.1","Apache-2.0","https://github.com/mojaloop/central-services-database"
"@mojaloop/central-services-error-handling@5.2.0","Apache-2.0","https://github.com/mojaloop/central-services-error-handling"
"@mojaloop/central-services-metrics@5.2.0","Apache-2.0","https://github.com/mojaloop/central-services-metrics"
"@mojaloop/central-services-shared@5.2.0","Apache-2.0","https://github.com/mojaloop/central-services-shared"
"@types/bluebird@3.5.26","MIT","https://github.com/DefinitelyTyped/DefinitelyTyped"
"accept@3.1.3","BSD-3-Clause","https://github.com/hapijs/accept"
"account-lookup-service@6.2.0","Apache-2.0","https://github.com/mojaloop/account-lookup-service"
"ammo@3.0.3","BSD-3-Clause","https://github.com/hapijs/ammo"
"ansi-regex@2.1.1","MIT","https://github.com/chalk/ansi-regex"
"ansi-regex@3.0.0","MIT","https://github.com/chalk/ansi-regex"
"ansi-styles@3.2.1","MIT","https://github.com/chalk/ansi-styles"
"archy@1.0.0","MIT","https://github.com/substack/node-archy"
"argparse@1.0.10","MIT","https://github.com/nodeca/argparse"
"arr-diff@4.0.0","MIT","https://github.com/jonschlinkert/arr-diff"
"arr-flatten@1.1.0","MIT","https://github.com/jonschlinkert/arr-flatten"
"arr-union@3.1.0","MIT","https://github.com/jonschlinkert/arr-union"
"array-each@1.0.1","MIT","https://github.com/jonschlinkert/array-each"
...

$ nlf
...
wrap-ansi@2.1.0 [license(s): MIT]
├── package.json:  MIT
├── license files: MIT
└── readme files: MIT

wrappy@1.0.2 [license(s): ISC]
├── package.json:  ISC
└── license files: ISC

wreck@14.1.3 [license(s): BSD-3-Clause]
└── package.json:  BSD-3-Clause

xmlcreate@1.0.2 [license(s): Apache, Apache-2.0]
├── package.json:  Apache-2.0
└── license files: Apache

y18n@3.2.1 [license(s): ISC]
├── package.json:  ISC
└── readme files: ISC

yallist@2.1.2 [license(s): ISC]
├── package.json:  ISC
└── license files: ISC

yargs@1.2.6 [license(s): MIT/X11]
└── package.json:  MIT/X11

yargs@3.32.0 [license(s): MIT]
└── package.json:  MIT

z-schema@3.25.1 [license(s): MIT]
├── package.json:  MIT
└── license files: MIT

LICENSES: (BSD-2-Clause OR MIT OR Apache-2.0), (MIT AND CC-BY-3.0), (MIT OR Apache-2.0), Apache, Apache 2.0, Apache-2.0, BSD, BSD-2-Clause, BSD-3-Clause, CC-BY-3.0, CC0-1.0, GPL, ISC, LGPL, MIT, MIT/X11, Public Domain, Unknown, WTFPL

kjw000 commented 5 years ago

Thanks @lewisdaly - Originally we also used NLF: https://www.npmjs.com/package/nlf and this worked okay. the only issue as you know you need to do one repo at a time. So we created a XLS with separate tabs for each and then sorted and manually reviewed for restrictive licenses. But I like the idea of doing this and getting 'clean' and then integrating it into CI/CD to 'stay clean'.

Let me know what you find with fossa-cli - as this looks good as well.

Also there is https://www.npmjs.com/package/license-checker which might be worth a look.

I think once we determine the right tool lets pick on repo and do then share with the core team. I can set up a meeting for this.

lewisdaly commented 5 years ago

From the fossa-cli team:

Hi, we do not enforce the number of builds for open source projects. If you have any further questions about Fossa's cloud product please direct them to support@fossa.com. Thank you for reaching out!

So it looks like we can use fossa-cli to at least build some nice reports to begin with, and then either keep using fossa-cli for our ci process or switch to one of these other nodejs based tools.

kjw000 commented 5 years ago

good news on fossa-cli.

lewisdaly commented 5 years ago

I ended up making a tool for running license scans across a number of projects at once:

https://github.com/vessels-tech/license-scanner

I've also completed the scan and made a excel file that @elnyry and I are reviewing at the moment.

Our next step is to integrate these tools into the CI process. I'm not sure if that should be included in this ticket or done elsewhere.

elnyry-sam-k commented 5 years ago

Many thanks for this analysis @lewisdaly .. Lets look at those and create follow-up items along with Kim.

Regarding adding to the CI process, I think its better to do that as a separate new item.

kjw000 commented 5 years ago

Agree we should create a new story to document the tool/process we are using and add it to the CI process.

mojaloop / project