search

Vizzuality / front-end-scaffold

A scaffold for new projects on Vizzuality
front-end-scaffold-docs.vercel.app
MIT License
3 stars 0 forks source link

robots.txt policy #116

Closed andresgnlez closed 1 year ago

andresgnlez commented 1 year ago

Based on the article Álvaro shared, shall we change our robots.txt policy? Right now, any production application will deploy with robots disabled which may impact SEO results if the file is not changed consciously.

I think like the article: staging/preview sites are development exceptions and should be treated as if. There are better ways to stop robots from crawling and avoid modifying robots.txt per environment.

mbarrenechea commented 1 year ago

https://github.com/vercel/examples/tree/main/edge-functions/basic-auth-password

andresgnlez commented 1 year ago

Closing the issue in favor of https://github.com/Vizzuality/front-end-scaffold/discussions/138