search

golang / gddo

Go Doc Dot Org
https://godoc.org
BSD 3-Clause "New" or "Revised" License
1.1k stars 266 forks source link

Local copy of database? #414

Open lmas opened 8 years ago

lmas commented 8 years ago

Hello, is it possible to get a copy of the live database from godoc.org?

I've been toying with ideas of making a searchable, offline database for when I don't have internet access and/or for making an online mirror for whenever I can't reach godoc.org.

Found a list of all 169710 packages currently indexed by godoc.org, so I assume the database dump would be huge. With that big amount I would like to avoid having to crawl github, and friends, myself and avoid generating that unneeded traffic for them or risk getting rate limited/banned.

dmitshur commented 8 years ago

This doesn't answer your original question, but a comment.

With that big amount I would like to avoid having to crawl github

FWIW, godoc.org does have an API that you could query; so in the worst case you wouldn't have to crawl github for Go packages.

https://github.com/golang/gddo/wiki/API

lmas commented 8 years ago

Yes thanks, that's where I found the API call for grabbing all the packages from. But if possible, I would like to avoid the web crawling completely (not just for github) and spare everyone involved from my extra, periodic high-traffic crawling.