search

bloom-housing / bloom

Bloom is Exygy’s affordable housing platform. Bloom's goal is to be a single entry point for affordable housing seekers and a hub for application and listing management for developers.
https://bloomhousing.com
Apache License 2.0
33 stars 25 forks source link

Non-Production Sites Getting Indexed #3991

Open YazeedLoonat opened 7 months ago

YazeedLoonat commented 7 months ago

We ran into an issue where a deploy preview from netlify was sticking around and showing up in search results. We don't really want that to happen so we should look at maybe adding a robots.txt or noIndex to non-production sites

see: https://developers.google.com/search/docs/crawling-indexing/robots/intro#:~:text=A%20robots.txt%20file%20tells,web%20page%20out%20of%20Google

https://developers.google.com/search/docs/crawling-indexing/block-indexing

sarahlazarich commented 7 months ago

also investigate why that deploy preview showed up for so long

sep ticket -- google also offers a tool that allows us to re-index our site, we should be familiar with that, see if we can get the deploy preview thats showing up for Detroit

sarahlazarich commented 7 months ago

Can we also just contact netlify directly? Since this is such weird behavior, they're only supposed to be up for 90 days

jaredcwhite commented 4 months ago

A bit of research on this: looks like Netlify automatically adds noindex headers for automated deploy previews, but for other sorts of branch deploys, that won't happen. If we want more fine-grained control over branches vs. prod, we'd need to go with this type of solution: https://docs.netlify.com/routing/headers/#custom-headers-for-different-branch-or-deploy-contexts