search

quakkels / rssdiscoveryengine

The RSS Discovery Engine exists to encourage people to use RSS for finding and consuming their news and current events.
MIT License
159 stars 9 forks source link

Honor robots.txt #26

Open robmeek opened 2 years ago

robmeek commented 2 years ago

This tool does not seem to honor robots.txt limits(?) e.g. “Disallow” and “crawl-delay”. It sent 400 requests to my site within a few seconds.

quakkels commented 1 month ago

RSS Discovery Engine is no longer live. But I'll be working on the next version soon. It will take a more responsible approach to how it gathers feed information from websites so that site owners bandwidth and processing are respected more.