Create new pipeline to fetch dependency provenance data

[mjherzog]

ScanCode Toolkit identifies dependencies from package manifest files, but in most cases what you really want is the provenance data that can be retrieved from a package repository. The enhancement request is to build a new SCIO pipeline with at least one example of fetching the provenance data from a package rep to use as a template. In a common use case you may want to include fetching package repo provenance data for 2 or 3 package types. This is related to:

• https://github.com/nexB/scancode-toolkit/issues/2591
• https://github.com/nexB/scancode.io/issues/579
• https://github.com/nexB/scancode.io/issues/228

I suspect that this enhancement may require Data Model changes.

[pombredanne]

This makes a lot of sense. This is also related to nexB/scancode-toolkit#272 The use cases could be reformulated this way:

• I have Package (e.g. PURLs) but I may be missing the provenance metadata.
• Or I have Package with weak or limited provenance metadata (including pre-built binaries) and I want to either get extra metadata or scan the corresponding source code for completeness.

In the case of dependencies when I do not know the exact versions I may want to further resolve a dependency version constraints to get a concrete version (and this can be very simple such as picking the latest released version).

And to get the extra data I can either:

• fetch extra data from package repository API
• fetch and scan the corresponding source code