search

canonical / canonical.com

Repository for the new version of canonical.com
Other
33 stars 66 forks source link

robots.txt not found for https://canonical.com #1048

Closed venkatsatish07 closed 2 months ago

venkatsatish07 commented 1 year ago

does it mean web scraping is allowed ?

carkod commented 8 months ago

As far as I know robots.txt is an non-required file used by Google to limit crawler requests. Web scraping, if you are talking about writing a script and scraping a website with it will not be affected by the robots.txt file.

britneywwc commented 2 months ago

@venkatsatish07 Closing this issue due to inactivity. If you have anymore questions, please feel free to reopen the issue or create a new one. Thanks!