search

kubernetes / website

Kubernetes website and documentation repo:
https://kubernetes.io
Creative Commons Attribution 4.0 International
4.49k stars 14.42k forks source link

Search engines indexing old releases #27338

Closed jimangel closed 2 years ago

jimangel commented 3 years ago

Problem:

Google search for "kubernetes secrets v2" returns:

image

(notice the URL subdomains)

Proposed Solution:

I need to do some more research but a few things come to mind:

That is assuming robots tag is the culprit. What's also strange is that the above robot config is set in release-1.19 but doesn't appear in the search results

I also think looking at old robot TXT issues might help. It's possible this problem has been solved and phased out with future releases.

/priority important-soon /triage accepted /help /cc @reylejano

k8s-ci-robot commented 3 years ago

@jimangel: This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed by commenting with the /remove-help command.

In response to [this](https://github.com/kubernetes/website/issues/27338): >**Problem:** > >Google search for "kubernetes secrets v2" returns: > >![image](https://user-images.githubusercontent.com/4601051/113066191-bb0d0980-917f-11eb-9e5b-3271a8a69476.png) > >(notice the URL subdomains) > >**Proposed Solution:** > >I need to do some more research but a few things come to mind: >- `enableRobotsTXT = true` is in [config.toml](https://github.com/kubernetes/website/blob/master/config.toml#L11) and set in the couple `release-1.xx` repos I checked. >- `X-Robots-Tag: noindex` is set in [layouts/index.headers](https://github.com/kubernetes/website/blob/master/layouts/index.headers#L37). I believe this is for non-prod sites that are running, however I believe even the old released sites are considered "production" > >That is assuming robots tag is the culprit. What's also strange is the above is set in release-1.19 but that is not an indexed option. > >I also think looking at [old robottxt issues](https://github.com/kubernetes/website/search?q=robots.txt&type=issues) might help. It's possible this problem has been solved and phased out with future releases. > >/priority important-soon >/triage accepted >/help >/cc @reylejano > Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
sftim commented 3 years ago

https://v1-19.docs.kubernetes.io/robots.txt does look wrong. We could override this file explicitly for the older releases (perhaps just as an interim measure?)

sftim commented 3 years ago

In the v1.19 docs, <META NAME="ROBOTS" CONTENT="INDEX, FOLLOW"> appears in the page head. That's wrong.

fejta-bot commented 3 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale

k8s-triage-robot commented 3 years ago

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle rotten

sftim commented 3 years ago

/lifecycle frozen

sftim commented 3 years ago

/retitle Search engines indexing old releases

Also, the upcoming release is also being indexed and we don't actually want that. Same cause I think.

imoisharma commented 3 years ago

Hi @jimangel and @sftim, I would like to work on this issue. I can see the proposed solution has already been implemented. May I know, what new things we need to work on for this issue? Now, Google search for "kubernetes secrets v2" returns: image

/assign

sftim commented 2 years ago

I've proposed a PR to fix this. :crossed_fingers:

sftim commented 2 years ago

/reopen

We also need to merge 4 backport PRs.

k8s-ci-robot commented 2 years ago

@sftim: Reopened this issue.

In response to [this](https://github.com/kubernetes/website/issues/27338#issuecomment-1014934136): >/reopen > >We also need to merge 4 backport PRs. Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
jihoon-seo commented 2 years ago

[Backport PRs]

sftim commented 2 years ago

It's fixed. I checked. /close

k8s-ci-robot commented 2 years ago

@sftim: Closing this issue.

In response to [this](https://github.com/kubernetes/website/issues/27338#issuecomment-1025254810): >It's fixed. I checked. >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.