Use the SCIO Pipeline Mechanism for Importers/Improvers

[keshav-space]

We should adopt/reuse the pipeline mechanism in SCIO for Importers and Improvers. With this, we would have a common programming model across tools, including ScanCode.io and PurlDB See also:

• https://github.com/nexB/vulnerablecode/issues/1395
• https://github.com/nexB/purldb/issues/521

Migrate the improvers to aboutcode.pipeline:

• [ ] vulnerabilities/improvers/valid_versions.py
• [x] vulnerabilities/improvers/vulnerability_kev.py
• [ ] vulnerabilities/improvers/vulnerability_status.py
• [x] vulnerabilities/improvers/default.py merge this as a step in new importer pipeline
  • Completed in https://github.com/aboutcode-org/vulnerablecode/pull/1559
  • This is now available as a import_new_advisories step in VulnerableCodeBaseImporterPipeline.

Migrate the importers to aboutcode.pipeline:

• [x] nvd https://github.com/aboutcode-org/vulnerablecode/pull/1587
• [x] github https://github.com/aboutcode-org/vulnerablecode/pull/1584
• [x] gitlab https://github.com/aboutcode-org/vulnerablecode/pull/1580
• [x] npm https://github.com/aboutcode-org/vulnerablecode/pull/1574
• [x] pypa https://github.com/aboutcode-org/vulnerablecode/pull/1559
• [x] nginx https://github.com/aboutcode-org/vulnerablecode/pull/1575
• [x] pysec #1627
• [ ] alpine_linux https://github.com/aboutcode-org/vulnerablecode/issues/1633
• [ ] openssl
• [ ] redhat
• [ ] debian
• [ ] postgresql
• [ ] archlinux
• [ ] ubuntu
• [ ] debian_oval
• [ ] retiredotnet
• [ ] apache_httpd
• [ ] mozilla
• [ ] gentoo
• [ ] istio
• [ ] project_kb_msr2019
• [ ] suse_scores
• [ ] elixir_security
• [ ] apache_tomcat
• [ ] xen
• [ ] ubuntu_usn
• [ ] fireeye
• [ ] apache_kafka
• [ ] oss_fuzz
• [ ] ruby
• [ ] github_osv
• [ ] epss

[pombredanne]

@keshav-space do you have any commit for this yet?

[keshav-space]

@keshav-space do you have any commit for this yet?

@pombredanne here is WIP PR https://github.com/nexB/vulnerablecode/pull/1533/

[tdruez]

aboutcode.pipeline 0.1 published to PyPI at https://pypi.org/project/aboutcode.pipeline/