Non-Production Sites Getting Indexed

[YazeedLoonat]

We ran into an issue where a deploy preview from netlify was sticking around and showing up in search results. We don't really want that to happen so we should look at maybe adding a robots.txt or noIndex to non-production sites

see: https://developers.google.com/search/docs/crawling-indexing/robots/intro#:~:text=A%20robots.txt%20file%20tells,web%20page%20out%20of%20Google

https://developers.google.com/search/docs/crawling-indexing/block-indexing

[sarahlazarich]

also investigate why that deploy preview showed up for so long

sep ticket -- google also offers a tool that allows us to re-index our site, we should be familiar with that, see if we can get the deploy preview thats showing up for Detroit

[sarahlazarich]

Can we also just contact netlify directly? Since this is such weird behavior, they're only supposed to be up for 90 days

[jaredcwhite]

A bit of research on this: looks like Netlify automatically adds noindex headers for automated deploy previews, but for other sorts of branch deploys, that won't happen. If we want more fine-grained control over branches vs. prod, we'd need to go with this type of solution: https://docs.netlify.com/routing/headers/#custom-headers-for-different-branch-or-deploy-contexts