search

daviddengcn / gcse

Project for Go Search, a search engine for finding popular and relevant packages.
http://go-search.org/
BSD 2-Clause "Simplified" License
277 stars 45 forks source link

Get all packages with their dependencies #22

Closed anvaka closed 9 years ago

anvaka commented 9 years ago

Hi David,

I'm going to visualize the go packages universe similar to what I did for npm. Your data source is looking very promising.

After reading your documentation and examples, looks like I will have to:

  1. Get all packages http://go-search.org/api?action=packages
  2. For each found package make a request (e.g. http://go-search.org/api?action=package&id=github.com%2fdaviddengcn%2fgcse), to get Imported field.

This would probably take up to 20 hours of crawler work. Is there any chance you can expose dependencies information in one call? E.g. npm allows you to get full database via single call: curl https://skimdb.npmjs.com/registry/_design/scratch/_view/byField

Thank you!

daviddengcn commented 9 years ago

The one for npm looks pretty cool. I just added a package_depends action in the api for you. Please don't call it too frequently (hopefully no more than once per day if you put it in a continuously running script).

It'll be very nice if you can show a link to http://go-search.org/ in you webpage.

Let me know if you find any issues on GoSearch.

Enjoy!

anvaka commented 9 years ago

I think you are awesome!

Thank you very much, David. Works like a charm. I'll make sure to not use it too often, and will keep your work linked from my websites/presentations.

Cheers, Andrei

anvaka commented 9 years ago

Hi David,

I thought you might enjoy what I did with your data: Visualization of go universe

Thank you so much for making it easily accessible!

daviddengcn commented 9 years ago

It's amazing! So beautiful!