search

daviddengcn / gcse

Project for Go Search, a search engine for finding popular and relevant packages.
http://go-search.org/
BSD 2-Clause "Simplified" License
277 stars 45 forks source link

Question: Can I download the entire index/database #34

Open OSSIndex-Admin opened 7 years ago

OSSIndex-Admin commented 7 years ago

Hi David,

OSS Index is an open vulnerability database. We are looking at supporting go at some point in the near future, and would like to use your package database as a starting point. Downloading via your API will take a fair while, and use up bandwidth you may not want used.

We are not creating a competing package search, but need a good list of packages in order to start performing vulnerability auditing.

Would it be possible to download the entire database from somewhere, or can I get permission to use your API to download information on all of the packages?

Regards,

Ken

daviddengcn commented 7 years ago

Thanks for reaching out. I'm pretty open with the data sharing. What kind of information do you need for each package exactly?

As mentioned in the API page, I'll appreciate if you can put a link on the website using Go-Search data or service.

OSSIndex-Admin commented 7 years ago

I will definitely add links in a few different places.

Right now the most important information are the following fields for each package:

It is possible that sometime in the future it might be interesting to imports information, but I will not be needing that for quite some time I expect. Far much too much other work to do for now :)

Thanks!

OSSIndex-Admin commented 7 years ago

Hi. Sorry to bug you, I just figure this fell off your radar. Would it be best that I just retrieve data through the API. I can make sure that I only run in "off hours" whenever that might be (let me know).

Thanks.

daviddengcn commented 7 years ago

Sorry, I'm kinda busy recently.

You can use "packages" action to get the list of all packages, then "package" action to get them one by one. I think using a qps of 10, you can fetch all packages within a day.

OSSIndex-Admin commented 7 years ago

NP. I totally understand the being busy thing.

And thanks, I will try and give this a shot over the weekend.