search

earthreader / web

Earth Reader for Web
http://earthreader.org/
GNU Affero General Public License v3.0
41 stars 19 forks source link

Use open_url for crawl feed #60

Closed tribela closed 9 years ago

tribela commented 9 years ago

Some sites are gives us 403 forbidden to default User-Agent. For an example: "explainxlcd.com".