search

mwmbl / crawler-extension

A browser extension that can be installed by volunteers to participate in mwmbl distributed crawling.
GNU Affero General Public License v3.0
21 stars 2 forks source link

Respect robots.txt #4

Closed daoudclarke closed 2 years ago

daoudclarke commented 2 years ago

Still to do in a future PR: cache the robots.txt to prevent repeated requests.