search

jacktuck / unfurl

Metadata scraper with support for oEmbed, Twitter Cards and Open Graph Protocol for Node.js :zap:
MIT License
481 stars 52 forks source link

Respect robots.txt #86

Closed phuctm97 closed 2 years ago

phuctm97 commented 2 years ago

Most sites will also look at robots.txt before parsing meta tags, would be great if unfurl can take this into account too!

jacktuck commented 2 years ago

Unfurl itself isn't a crawler. I don't think this is in unfurl's remit really.