search

daviddengcn / gcse

Project for Go Search, a search engine for finding popular and relevant packages.
http://go-search.org/
BSD 2-Clause "Simplified" License
277 stars 45 forks source link

Some referenced packages are showing as "not found" #25

Open xi2 opened 8 years ago

xi2 commented 8 years ago

Hi. Firstly, thank you for making Go Search. It's a useful tool.

I have noticed that some of my own packages are showing as "not found" on go-search.org, even though they are shown as imports of other packages. They begin "xi2.org/x/..."

For example, if you view the page http://go-search.org/view?id=github.com%2fklauspost%2fpassword and select my package "xi2.org/x/xz" which appears as an import at the bottom of that page you get the "not found" response.

I decided I would try to debug the issue, so I downloaded the source and compiled gcse myself. I ran it and limited the crawl to just a handful of packages including my own, and to my surprise it worked fine when I ran it locally! So I can't help debug it it would seem since there must be something different about my local environment :(

As far as I know the packages download fine with "go get" and they show up ok on godoc.org. Any ideas? Is there something weird about my web server?

It might seem quite conceited of me to be concerned about my own packages, but you never know, it might also be happening to someone else's too.

Best wishes

daviddengcn commented 8 years ago

Checking the logs, here is the reason I guess so far: Request to "xi2.org/x/xz" will be redirected to "github.com", but the crawler does not use autheticated requests, so it leads to a very low rate limit. The fix needs some modification on the crawler package.

xi2 commented 8 years ago

Thanks for replying. I hope the crawler modification is not too big a change.