search

kubernetes / k8s.io

Code and configuration to manage Kubernetes project infrastructure, including various *.k8s.io sites
https://git.k8s.io/community/sig-k8s-infra
Apache License 2.0
714 stars 799 forks source link

Migrate away from google.com gcp project kubernetes-jenkins #1310

Open spiffxp opened 3 years ago

spiffxp commented 3 years ago

Part of umbrella issue to migrate away from google.com gcp projects: https://github.com/kubernetes/k8s.io/issues/1469

Umbrella issue for migrating or removing dependence on all of the kubernetes project infra that lives under kubernetes-jenkins:

GCS buckets: gsutil ls -p kubernetes-jenkins | grep -v kubernetes-staging- | sed -e 's/.*/- [ ] `&`/'

Service Accounts: I'm not sure of everything I am (not) allowed to list here. So this isn't an exhaustive list. But we should make sure none of the service accounts in this project are used in any IAM bindings in kubernetes.io. Googlers will need to help identify this.

spiffxp commented 3 years ago

At the moment it's not clear to me whether we can update prow.k8s.io to support different jobs writing to different buckets, or if we're going to have to make the change in lockstep. This is why migrating gs://kubernetes-jenkins needs a proposal

spiffxp commented 3 years ago

/lifecycle frozen

tosi3k commented 3 years ago

On the occasion of moving things out of the kubernetes-jenkins GCP project, gs://sig-scalability-logs bucket should also be migrated.

SIG scalability uses this bucket to store master and node logs separately from the gs://kubernetes-jenkins bucket for two reasons:

  1. it was needed for us in order to finish migration of our Prow jobs to pod-utils (https://github.com/kubernetes/test-infra/pull/17215)
  2. speeding up loading the Prow GCS viewer of Prow jobs where many artifacts were stored in gs://kubernetes-jenkins bucket (we produce lots of them in our tests given the test cluster' sizes)
ameukam commented 3 years ago

On the occasion of moving things out of the kubernetes-jenkins GCP project, gs://sig-scalability-logs bucket should also be migrated.

SIG scalability uses this bucket to store master and node logs separately from the gs://kubernetes-jenkins bucket for two reasons:

  1. it was needed for us in order to finish migration of our Prow jobs to pod-utils (https://github.com/kubernetes/test-infra/pull/17215)
  2. speeding up loading the Prow GCS viewer of Prow jobs where many artifacts were stored in gs://kubernetes-jenkins bucket (we produce lots of them in our tests given the test cluster' sizes)

@tosi3k Do you want the contents of this bucket to be preserved? if yes, what is the estimated size of sig-scalability-logs ?

tosi3k commented 3 years ago

@tosi3k Do you want the contents of this bucket to be preserved?

We just need the last 90 days of the contents therein.

I wonder - would applying some retention policy to gs://sig-scalability-logs bucket make sense? I'm not a big GCS / test-infra expert but I think that some kind of mechanism like this is already done for gs://kubernetes-jenkins bucket where we store the logs for jobs that have finished in the last 90 days IIUC.

After we migrate the scalability job configs to use gs://k8s-infra-scalability-tests-logs instead and 90 days pass, we could remove the old bucket (gs://sig-scalability-logs) completely as there would be no need for it anymore. Would that make sense here?

if yes, what is the estimated size of sig-scalability-logs?

I don't know but if we were to introduce a 90d retention policy mechanism we would probably comfortably fit in 5 TBs. This is the size of the bucket after 97 days of existence: image

spiffxp commented 3 years ago

/milestone v1.23 I don't know that we can get all of this done within v1.23 but I'd like to see us start pruning the random buckets away at the very least

spiffxp commented 2 years ago

/milestone v1.24

ameukam commented 2 years ago

/milestone clear

BenTheElder commented 1 month ago

I'm working on eliminating old unused buckets from this project.

kubernetes-jenkins and sig-scalability-logs still need a plan.

There's also the mysterious kubernetes-jenkins-gcslogs which has content written to it recently judging by the file names but we can't tell where from yet ...

BenTheElder commented 1 month ago

Circled back:

$ gsutil logging get gs://kubernetes-jenkins
{"logBucket": "kubernetes-jenkins-gcslogs", "logObjectPrefix": "kubernetes-jenkins"}

Cleaning that up now.

BenTheElder commented 1 month ago

gs://kubernetes-jenkins/ and gs://sig-scalability-logs/ remain, I cleaned up the rest, including the many many gs://kubernetes-staging-*

tosi3k commented 1 month ago

Opened https://github.com/kubernetes/test-infra/pull/33248 for getting rid of gs://sig-scalability-logs from the scalability jobs configs.

BenTheElder commented 1 month ago

Thank you! The only remaining reference is https://github.com/kubernetes/k8s.io/blob/a554228abb7c8134b2b7adef881ee1c8d9150d14/apps/gcsweb/deployment.yaml#L36 which we should leave for a bit.

Later we should drop write permissions to this bucket, and then when we still don't see issues then we should drop it from gcsweb (no rush ...)

BenTheElder commented 1 month ago

so mostly we need to migrate off of gs://kubernetes-jenkins, I think with everything else going on there's a high chance we'll defer this to after the migration, as far as I know aside from the prow control plane there's that one and gs://kubernetes-release and then we're pretty much out of google.com into kubernetes.io 🤞

ameukam commented 3 weeks ago

/remove-lifecycle frozen /milestone v1.32