search

adamdecaf / horizon

A bunch of golang code that doesn't quite fit together yet
0 stars 0 forks source link

reddit: front page crawling #9

Open adamdecaf opened 8 years ago

adamdecaf commented 8 years ago

It's useful to grab out the reddit pages, if not just for domains to crawl later on.

adamdecaf commented 8 years ago

Might want to use https://github.com/hashicorp/go-cleanhttp with net/http