Redirect dl.k8s.io traffic to a community-owned GCS bucket instead of kubernetes-release

[spiffxp]

Part of umbrella issues:

• to deprecate and migrate away from gs://kubernetes-release: https://github.com/kubernetes/k8s.io/issues/2396
• to migrate away from GCP project google-containers: https://github.com/kubernetes/k8s.io/issues/1571

Traffic gets to gs://kubernetes-release in one of two ways:

• uri's that have the bucket name hardcoded (e.g. https://cs.k8s.io/?q=kubernetes-release%5B%5E-%5D&i=nope&files=&repos=)
• dl.k8s.io uri's that redirect to the bucket (too many matches for cs.k8s.io to display, but e.g. used in kubernetes/kubernetes CHANGELOGs)

We discussed the idea of cutting over dl.k8s.io traffic first:

• perhaps we could end up cutting over a large chunk of traffic sooner vs. later
• this will help us better plan how to migrate traffic that has the bucket name hardcoded

The sketch is something like:

• decide which project/bucket to use?
• sync from old bucket to new bucket
• change dl.k8s.io to use new bucket instead of old bucket
• see what the delta in traffic is
• decide how to keep buckets in sync?

Which project/bucket to use:

• we have k8s-release hosting gs://k8s-release
• we could choose to use a bucket hosted in k8s-artifacts-prod (simplifies billing)
• we could choose to block until we have similar staging/promotion process for GCS artifacts

How to sync:

• https://cloud.google.com/storage-transfer is pretty simple/fast to setup, I would at least use this for initial sync
  • There is currently no gcloud command, but there is an API and some bindings (ref: https://cloud.google.com/storage-transfer/docs/create-manage-transfer-program#cloud-to-cloud)
  • It supports recurring jobs, but no more frequent than hourly (ref: https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs#TransferJob)
  • It supports one-time jobs, with a limit of 1K per day per project (ref: https://cloud.google.com/storage-transfer/quotas)
• For ongoing sync we could
  • rely on hourly sync
  • periodically create one-time sync jobs more frequently
  • use gcs pubsub notifications to create sync jobs
  • alter release tooling to publish to multiple buckets
  • alter release tooling to create sync jobs

Seeing the delta in traffic:

• we'll be able to see an influx of traffic in our billing report
• @thockin and I need to decide if/how to share the existing traffic gs://kubernetes-release is getting

My preferences would be:

• delete/recreate the gs://k8s-release bucket in k8s-artifacts-prod
• use sts, rely on hourly sync job first, then prowjob that periodically creates one-time syncs

/assign @thockin since you were part of the discussion /assign @justaugustus @hasheddan For @kubernetes/release-engineering input

/wg k8s-infra /sig release /sig testing /area artifacts /milestone v1.21

[justaugustus]

(Opened https://github.com/kubernetes/k8s.io/pull/1857 for dl.k8s.io/ci.)

[spiffxp]

I manually initiated a one-time transfer from gs://kubernetes-release/release to gs://k8s-release/release following instructions at https://cloud.google.com/storage-transfer/docs/create-manage-transfer-console to get a sense of size/cost of transfer

[spiffxp]

delete/recreate the gs://k8s-release bucket in k8s-artifacts-prod

@justaugustus I would still like us to do this before we consider flipping over dl.k8s.io for real

I could be convinced to flip over temporarily to get an estimate of the amount of traffic this flips, but we should ultimately be using a similar approach as we do for GCR, with promotion from a staging bucket etc.

/assign @saschagrunert @puerco @cpanato @jeremyrickard other SIG Release leads for input

[spiffxp]

/milestone v1.23 Definitely not doing this before v1.22 goes out the door.

I've pointed more things at dl.k8s.io as part of migrating away from hardcoding kubernetes-release-dev in URIs.

I'm going to edit the description to put this under the umbrella issue I've created for gs://kubernetes-release deprecation and migration

[spiffxp]

We have concerns that we should hold off on this until we tackle cross-cloud promotion

Would also like to understand why the cost of this is so much larger than cost of container images

[spiffxp]

Make a followup issue for dl.k8s.io log analysis, or look into temporarily enabling access logs and provide a sanitized version

[spiffxp]

Blocked on https://github.com/kubernetes/k8s.io/issues/1375

[spiffxp]

/milestone v1.24

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[ameukam]

/remove-lifecycle stale

[ameukam]

/lifecycle frozen /milestone clear

[ameukam]

/remove-lifecycle frozen /milestone v1.26

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[ameukam]

/remove-lifecycle stale /milestone v1.27 /remove-priority /priority important-longterm

Pending AWS credits (announced here) and decision from Fastly.

[k8s-ci-robot]

@ameukam: Those labels are not set on the issue: priority/important-longterm

In response to [this](https://github.com/kubernetes/k8s.io/issues/1569#issuecomment-1326779496): >/remove-lifecycle stale >/milestone v1.27 >/remove-priority >/priority important-longterm > >Pending AWS credits (announced [here](https://twitter.com/dims/status/1585621637512286211)) and decision from Fastly. Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[ameukam]

Postponed to v1.28

/milestone v1.28

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[ameukam]

/remove-lifecycle stale

[BenTheElder]

We're still not always redirecting away from the bucket and we have most traffic going to the bucket directly (some of which is just skipping the redirect because users have come to depend on the GCS bucket).

We need to: a) ensure dl.k8s.io never redirects to the bucket b) switch publishing to a new bucket, restrict access to fastly only, and send users through dl.k8s.io => cdn.dl.k8s.io

[ameukam]

/milestone v1.29

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[ameukam]

/remove-lifecycle stale

[ameukam]

/milestone v1.31

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[ameukam]

/remove-lifecycle stale

[ameukam]

/milestone v1.32