search

aboutcode-org / purldb

Tools to create and expose a database of purls (Package URLs). This project is sponsored by NLnet project https://nlnet.nl/project/vulnerabilitydatabase/ and nexB for https://www.aboutcode.org/ Chat is at https://gitter.im/aboutcode-org/discuss
https://purldb.readthedocs.io/
35 stars 23 forks source link

Mine Rust crates #545

Open pombredanne opened 2 weeks ago

pombredanne commented 2 weeks ago

We need to collect all the crates and index them. There is an API on crates.io and also a git repo with an index:

alok1304 commented 2 weeks ago

@pombredanne, suppose we collected and indexing all the crates from Crates.io. Once we have all crates indexed, could you clarify the next steps? Should we focus on setting up querying functionality, integrating the index into PurlDB, or enriching the data in any particular way? Any guidance on priorities or additional requirements would be greatly appreciated!

pombredanne commented 1 week ago

@alok1304 re:

suppose we collected and indexing all the crates from Crates.io. Once we have all crates indexed, could you clarify the next steps? Should we focus on setting up querying functionality, integrating the index into PurlDB, or enriching the data in any particular way? Any guidance on priorities or additional requirements would be greatly appreciated!

Getting the index is straightforward, this is just a git repo. With this the steps would be:

For batch operations:

  1. Parse the index and populate the PurlDB with basic cargo packages
  2. For one package from 1., further collect detailed metadata from the API
  3. For one package from 2., further integrate the download and scan of the crate code

For the on-demand API endpoint:

  1. For one package PURL as an input, do the 2. and 3. from above, on demand

This would establish the data in PurlDB to collect, index, match and have a comprehensive references for all the crates.

pombredanne commented 1 week ago

In terms of priority, the on-demand operations are likely higher priority to get some results fast to than the full batch operations that come afterwards. Unless Batch 1. is really easy, which it may be.