search

mysociety / alaveteli

Provide a Freedom of Information request system for your jurisdiction
https://alaveteli.org
Other
386 stars 196 forks source link

[#8132] Update actions and pages which set "noindex", "nofollow" crawler directives #8223

Open gbp opened 2 months ago

gbp commented 2 months ago

Relevant issue(s)

Fixes #8132

What does this do?

Update actions and pages which set "noindex", "nofollow" crawler directives

Why was this needed?

Snippets of request content often appear on list pages, and create a whack-a-mole situation when unhappy users find that external search engines have indexed a list page (e.g. /body/foo?page=12) that contains a cached snippet of PII that we've removed from the request page itself.

Implementation notes

@garethrees are you happy with changing the number of paginated pages which are indexed? I'm concerned this might impact search ranking due to newer request pages not being indexed at all.

garethrees commented 2 months ago

Reassigning as discussed to make a few tweaks.