add check_scraper_allowed function.

hhursev / recipe-scrapers

Python package for scraping recipes data

MIT License

1.73k stars 528 forks source link

Open dragonpop76 opened 1 year ago

dragonpop76 commented 1 year ago

Added an auto-check to see if robots.txt prohibits scraping on a specific url.