search

kubernetes / k8s.io

Code and configuration to manage Kubernetes project infrastructure, including various *.k8s.io sites
https://git.k8s.io/community/sig-k8s-infra
Apache License 2.0
730 stars 808 forks source link

GCR to Artifact Registry tracking #1343

Open jimangel opened 3 years ago

jimangel commented 3 years ago

Tracking issue for moving from Google Container Registry to Google Artifact Registry.

Artifact Registry is "The next generation of Container Registry. Store, manage, and secure your build artifacts." Once Artifact Registry graduates from beta, there will be a minimum of 6 months supported on GCR.

The general impact is: us.gcr.io/my-project/webapp becomes us-docker.pkg.dev/my-project/team1/webapp

https://cloud.google.com/artifact-registry https://cloud.google.com/artifact-registry/docs/transition/transition-from-gcr#top_of_page

/cc @kubernetes/release-managers @spiffxp @dims @bartsmykla /priority important-soon /area artifacts /sig release /sig testing /sig k8s-infra

Update (@ameukam): Sucess criteria:

spiffxp commented 3 years ago

Looks like this first went beta in March 2020: https://cloud.google.com/artifact-registry/docs/release-notes#March_16_2020

I'm not inclined to suggest we move on this until at least "Container Analysis and vulnerability scanning" is available: https://cloud.google.com/artifact-registry/docs/transition/transition-from-gcr#feature-availability

I would like to understand if k8s.gcr.io already is or can become one of the backwards compatible repos mentioned here: https://cloud.google.com/artifact-registry/docs/transition/transition-from-gcr#compatibility

We should consider whether container-image-promoter and any related scripts would need to be rewritten to support new APIs / commands: https://cloud.google.com/artifact-registry/docs/transition/setup-repo#gcloud-comparison

FYI @listx

ameukam commented 3 years ago

ICYMI, Artifact Registry is now GA: https://cloud.google.com/artifact-registry/docs/transition/transition-from-gcr#feature-availability

dims commented 3 years ago

Update from @saschagrunert : "We have a research topic on our board about "Support for Google Artifact Registry and how do we transition?"" ( from https://kubernetes.slack.com/archives/C2C40FMNF/p1610377375328600 )

justaugustus commented 3 years ago

This will be part of SIG Release's Artifact Mgmt epic, which I'll be working to flesh out this cycle.

ref: https://github.com/kubernetes/sig-release/issues/1372 /assign /milestone v1.21

spiffxp commented 3 years ago

I notice the language on https://cloud.google.com/artifact-registry/docs/transition/transition-from-gcr#top_of_page has changed from "minimum 6 months of GCR support post-GA" to "Both services will continue to co-exist after Artifact Registry becomes generally available" so I feel like there is less urgency to this

/remove-priority important-soon /priority important-longterm

dims commented 3 years ago

@spiffxp Whew!!!! thanks.

spiffxp commented 3 years ago

I think we're having problems using docker buildx build to push to GCR (https://github.com/kubernetes/test-infra/issues/20884#issuecomment-783840975) due to a containerd bug. GCR's not the only service impacted by the bug, but other services seem to have rolled out a workaround by now (including GAR?).

I'm not saying this is impetus enough to migrate now but may be a sign that GCR is headed toward some kind of maintenance mode vs. getting fixes.

fejta-bot commented 3 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale

fejta-bot commented 3 years ago

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle rotten

ameukam commented 3 years ago

/remove-lifecycle rotten /lifecycle frozen

spiffxp commented 3 years ago

/remove-priority important-longterm /priority awaiting-more-evidence

ameukam commented 3 years ago

GCP published a transition plan to Artifact registry : https://cloud.google.com/artifact-registry/docs/transition/setup-gcr-repo#push-pull

spiffxp commented 3 years ago

/remove-priority awaiting-more-evidence /priority backlog GCR is getting no new features, so there is a risk we may need to move if some essential feature comes up

justaugustus commented 2 years ago

/milestone v1.24

justaugustus commented 2 years ago

Enabling Artifact Registry for RelEng projects:

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

ameukam commented 2 years ago

/remove-lifecycle stale /milestone v1.25 /priority important-longterm

BenTheElder commented 2 years ago

This topic came up again, adding some more up to date notes ...

We'll have to start discussing this more soon.

I'm not inclined to suggest we move on this until at least "Container Analysis and vulnerability scanning" is available: https://cloud.google.com/artifact-registry/docs/transition/transition-from-gcr#feature-availability

It is now, and we've disabled it anyhow due to cost anyhow https://github.com/kubernetes/k8s.io/issues/1963

I would like to understand if k8s.gcr.io already is or can become one of the backwards compatible repos mentioned here: https://cloud.google.com/artifact-registry/docs/transition/transition-from-gcr#compatibility

Not automatically / with anything standard ... k8s.gcr.io is special (it performs regionalizing to {eu,us,asia}.gcr.io/k8s-artifacts-prod), however, as long as the endpoint speaks OCI, the clients don't care.

We have https://github.com/kubernetes-sigs/oci-proxy coming. We can perform the AR switch behind that.

This also gives us room to regionalize ourselves and stop using multi-regional storage, which is going to become a big cost issue in October of this year (also because GCR exposes the GCS bucket, AR does not).

https://cloud.google.com/storage/pricing-announce

We should consider whether container-image-promoter and any related scripts would need to be rewritten to support new APIs / commands: https://cloud.google.com/artifact-registry/docs/transition/setup-repo#gcloud-comparison

Creating registries changed, but pushing / tagging / ... remains OCI, drop-in. If we're shelling out to gcloud anywhere we s/containers images/artifacts docker/g.

We already have to push to the three actual registries today (k8s.gcr.io is just a regionalizing alias), so it should be a pretty small change to additionally push to some AR registries in preparation to switch.

GCR is getting no new features, so there is a risk we may need to move if some essential feature comes up

This. GCR is in maintenance mode.

At some point we're going to want new features (immutable tags? 🤞 ), or as previously mentioned pushing from buildx ... we'll have to switch to get these. Plus the incoming cost increase from all our multi-regional storage, which we could just regionalize ourselves like oci-proxy is doing for S3, and have fine-grained control over cost.

upodroid commented 2 years ago

One additional piece of info:

Audit Logging doesn't work for Public AR registries. That deprecates the gcs logs processing component of the data pipelines at https://ii.nz/post/building-a-data-pipline-for-displaying-kubernetes-public-artifact-traffic/

We will need oci-proxy to send a structured json log entry with the relevant details to Cloud Logging(writing to stdout is sufficient) which we can easily sink to a bigquery for analysis.

FYI: GCS Usage Logs has a subset of the information available through Audit Logging.

riaankleinhans commented 2 years ago

One additional piece of info:

Audit Logging doesn't work for Public AR registries. That deprecates the gcs logs processing component of the data pipelines fr https://ii.nz/post/building-a-data-pipline-for-displaying-kubernetes-public-artifact-traffic/

We will need oci-proxy to send a structured json log entry with the relevant details to Cloud Logging(writing to stdout is sufficient) which we can easily sink to a bigquery for analysis.

FYI: GCS Usage Logs has a subset of the information available through Audit Logging.

/cc @hh @BobyMCbobs

ameukam commented 2 years ago

/milestone v.126

k8s-ci-robot commented 2 years ago

@ameukam: The provided milestone is not valid for this repository. Milestones in this repository: [v1.24, v1.25, v1.26]

Use /milestone clear to clear the milestone.

In response to [this](https://github.com/kubernetes/k8s.io/issues/1343#issuecomment-1221129129): >/milestone v.126 Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
ameukam commented 2 years ago

/milestone v1.26

upodroid commented 2 years ago

Hi All

This migration is now complete.

ameukam commented 2 years ago

/reopen

Moving to Artifact Registry also needs the additional step to remove the GCR container registries used by k8s.gcr.io. remove those container registries will not be done right away.

k8s-ci-robot commented 2 years ago

@ameukam: Reopened this issue.

In response to [this](https://github.com/kubernetes/k8s.io/issues/1343#issuecomment-1270190944): >/reopen > >Moving to Artifact Registry also needs the additional step to remove the GCR container registries used by `k8s.gcr.io`. Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
ameukam commented 1 year ago

/milestone v1.27

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

BenTheElder commented 1 year ago

/remove-lifecycle stale We still have a boatload of staging registries to migrate. The prod registries are migrated. (https://registry.k8s.io uses AR not GCR)

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

ameukam commented 1 year ago

/remove-lifecycle stale /milestone v1.29

ameukam commented 8 months ago

/milestone v1.30

k8s-triage-robot commented 5 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

dims commented 5 months ago

/remove-lifecycle stale

ameukam commented 1 month ago

/milestone v1.32