cloudfoundry-incubator / kubecf

Cloud Foundry on Kubernetes
Apache License 2.0
115 stars 62 forks source link

CI is broken because of Dockerhub bandwidth limits #1557

Closed jandubois closed 4 years ago

jandubois commented 4 years ago

The last few PR builds failed to report status back to Github:

failed to fetch digest for image 'resource/github-status:release': 429 Too Many Requests
does the image exist?

We should move all images used by Concourse to the Github registry instead. The above image is from https://github.com/colstrom/concourse-github-status

But we are still pulling many more images from Dockerhub, at the very least:

cloudfoundry-incubator/kubecf
resource/github-status
splatform/base-ci
splatform/catapult
splatform/fissile-stemcell-sle
splatform/github-pr-resource
viccuad commented 4 years ago

splatform/catapult is being built on concourse.suse.de, from https://github.com/SUSE/cloudfoundry/blob/master/ci/pipelines/catapult/pipeline.yml. This may be the case for all the splatform org images.

jandubois commented 4 years ago

I've moved the the 2 images used by the pipeline to ghcr.io/jandubois:

resource/github-status
splatform/github-pr-resource

Especially the github-pr-resource is downloaded way more often than I expected.

Anyways, we need to create a Github organization that can become the owner of all these images; they should not be stored under a personal account.

The name should not mention concourse but be generic enough to include also all the images for bosh releases, for stemcells, and for the quarks operator.

viccuad commented 4 years ago

@jandubois honest thanks for moving the images; that should have been me, and I was blocking CI.

@viovanov could we create the GH org now? or should we reuse cloudfoundry-incubator as quarks does? In the second case, we would need credentials for it.

Once we have a target org, we can change the target of suse-buildpacks-ci, eirinix, and the rest of imagelist.txt to also publish there.

viovanov commented 4 years ago

@viccuad please use the incubator org

jandubois commented 4 years ago

I've added some comments on #1581 about images for autoscaler and eirini.

viccuad commented 4 years ago

Prereequisites:

For all images to migrate:

  1. They need to be pushed to ghcr.io/cloudfoundry-incubator
  2. They need to be made public manually on ghcr.io
  3. We need to consume them in the helm chart

Possible incomplete list of images to push to ghcr.io:

mook-as commented 4 years ago

With #1581 merged, the current list of images I have (from an amalgamation of two clusters, one diego and one eirini, both in the middle of 🐱), including any system pods:

List of images ``` 127.0.0.1:31666/cloudfoundry/5c5778eb-0853-4c97-92e1-efe32412f1b2:2e553bf5996a2501f9b5ad4afa3e599625907ddf 127.0.0.1:31666/cloudfoundry/c081ef4d-10f7-4abc-89dd-85897da90ef6:c9932a53a1e529e375a8aea0a4c34f0aa1c36a7a cfcontainerization/cf-operator:v6.1.17-0.gec409fd7 cfcontainerization/coredns:0.1.0-1.6.7-bp152.1.19 cfcontainerization/quarks-job:v1.0.206 cfcontainerization/quarks-secret:v1.0.744 docker.io/bitnami/external-dns:0.7.4-debian-10-r41 docker.io/bitnami/nginx-ingress-controller:0.40.2-debian-10-r27 docker.io/bitnami/nginx:1.19.4-debian-10-r4 gcr.io/gke-release/gke-metrics-agent:0.1.3-gke.0 gcr.io/stackdriver-agents/metadata-agent-go:1.2.0 gcr.io/stackdriver-agents/stackdriver-logging-agent:1.6.36 ghcr.io/cfcontainerizationbot/eirini-dns-aliases:0.0.0-32.g585bc34d ghcr.io/cfcontainerizationbot/kubecf-apps-dns:0.1.0 ghcr.io/cloudfoundry-incubator/app-autoscaler:SLE_15_SP1-27.11-7.0.0_374.gb8e8e6af-3.0.1 ghcr.io/cloudfoundry-incubator/capi:SLE_15_SP1-27.11-7.0.0_374.gb8e8e6af-1.98.0 ghcr.io/cloudfoundry-incubator/cf-acceptance-tests:SLE_15_SP1-27.11-7.0.0_374.gb8e8e6af-0.0.22 ghcr.io/cloudfoundry-incubator/cf-cli:SLE_15_SP1-27.11-7.0.0_374.gb8e8e6af-1.29.0 ghcr.io/cloudfoundry-incubator/cf-networking:SLE_15_SP1-27.11-7.0.0_374.gb8e8e6af-2.33.0 ghcr.io/cloudfoundry-incubator/cf-smoke-tests:SLE_15_SP1-27.11-7.0.0_374.gb8e8e6af-41.0.1 ghcr.io/cloudfoundry-incubator/credhub:SLE_15_SP1-27.11-7.0.0_374.gb8e8e6af-2.8.0 ghcr.io/cloudfoundry-incubator/diego:SLE_15_SP1-27.11-7.0.0_374.gb8e8e6af-2.48.0 ghcr.io/cloudfoundry-incubator/eirinix-loggregator-bridge:v0.0.0-0.g7da9e04 ghcr.io/cloudfoundry-incubator/eirinix-persi-broker:v0.0.0-g0c241e7 ghcr.io/cloudfoundry-incubator/eirinix-persi:v0.0.0-73.g8201eba ghcr.io/cloudfoundry-incubator/eirinix-ssh:v0.0.0-0.g09b53c0 ghcr.io/cloudfoundry-incubator/garden-runc:SLE_15_SP1-27.11-7.0.0_374.gb8e8e6af-1.19.16 ghcr.io/cloudfoundry-incubator/log-cache:SLE_15_SP1-27.11-7.0.0_374.gb8e8e6af-2.8.0 ghcr.io/cloudfoundry-incubator/loggregator-agent:SLE_15_SP1-27.11-7.0.0_374.gb8e8e6af-6.1.1 ghcr.io/cloudfoundry-incubator/loggregator:SLE_15_SP1-27.11-7.0.0_374.gb8e8e6af-106.3.10 ghcr.io/cloudfoundry-incubator/nats:SLE_15_SP1-27.11-7.0.0_374.gb8e8e6af-34 ghcr.io/cloudfoundry-incubator/postgres:SLE_15_SP1-27.11-7.0.0_374.gb8e8e6af-42 ghcr.io/cloudfoundry-incubator/pxc:0.9.11 ghcr.io/cloudfoundry-incubator/routing:SLE_15_SP1-27.11-7.0.0_374.gb8e8e6af-0.206.0 ghcr.io/cloudfoundry-incubator/silk:SLE_15_SP1-27.11-7.0.0_374.gb8e8e6af-2.33.0 ghcr.io/cloudfoundry-incubator/statsd-injector:SLE_15_SP1-27.11-7.0.0_374.gb8e8e6af-1.11.15 ghcr.io/cloudfoundry-incubator/sync-integration-tests:SLE_15_SP1-27.11-7.0.0_374.gb8e8e6af-v0.0.2 ghcr.io/cloudfoundry-incubator/uaa:SLE_15_SP1-27.11-7.0.0_374.gb8e8e6af-74.24.0 gke.gcr.io/addon-resizer:1.8.11-gke.1 gke.gcr.io/addon-resizer:1.8.8-gke.1 gke.gcr.io/calico/node:v3.8.8-1-gke.0-amd64 gke.gcr.io/calico/typha:v3.8.8-1-gke.0-amd64 gke.gcr.io/cluster-proportional-autoscaler-amd64:1.7.1-gke.0 gke.gcr.io/cluster-proportional-autoscaler-amd64:1.8.1-gke.0 gke.gcr.io/cpvpa-amd64:v0.8.3-gke.0 gke.gcr.io/event-exporter:v0.3.3-gke.0 gke.gcr.io/k8s-dns-dnsmasq-nanny-amd64:1.15.13 gke.gcr.io/k8s-dns-kube-dns-amd64:1.15.13 gke.gcr.io/k8s-dns-sidecar-amd64:1.15.13 gke.gcr.io/kube-proxy-amd64:v1.16.13-gke.401 gke.gcr.io/prometheus-to-sd:v0.10.0-gke.0 gke.gcr.io/prometheus-to-sd:v0.4.2 k8s.gcr.io/fluentd-gcp-scaler:0.5.2 k8s.gcr.io/ip-masq-agent-amd64:v2.4.1 k8s.gcr.io/metrics-server-amd64:v0.3.6 k8s.gcr.io/prometheus-to-sd:v0.8.2 registry.suse.com/cap-staging/bits-service:bits-1.0.21-15.1.6.2.298-24.56 registry.suse.com/cap-staging/event-reporter:1.8.0 registry.suse.com/cap-staging/instance-index-env-injector:c101e581d7f5ec01b6970eb2ae1a58df4600f872 registry.suse.com/cap-staging/metrics-collector:1.8.0.1 registry.suse.com/cap-staging/opi:1.8.0.1 registry.suse.com/cap-staging/recipe-uploader:1.8.0-24.56 registry.suse.com/cap-staging/route-collector:1.8.0.1 registry.suse.com/cap-staging/route-pod-informer:1.8.0.1 registry.suse.com/cap-staging/route-statefulset-informer:1.8.0.1 registry.suse.com/cap-staging/staging-reporter:1.8.0 ```

The only relevant ones are:

cfcontainerization/cf-operator:v6.1.17-0.gec409fd7
cfcontainerization/coredns:0.1.0-1.6.7-bp152.1.19
cfcontainerization/quarks-job:v1.0.206
cfcontainerization/quarks-secret:v1.0.744
docker.io/bitnami/external-dns:0.7.4-debian-10-r41
docker.io/bitnami/nginx-ingress-controller:0.40.2-debian-10-r27
docker.io/bitnami/nginx:1.19.4-debian-10-r4

This boils down to quarks (we'll need to consumer a newer release) and the ingress chart.

viccuad commented 4 years ago

The following

docker.io/bitnami/external-dns:0.7.4-debian-10-r41
docker.io/bitnami/nginx-ingress-controller:0.40.2-debian-10-r27
docker.io/bitnami/nginx:1.19.4-debian-10-r4

come from https://github.com/SUSE/cap-terraform/tree/cap-ci and its upstsream helm charts being installed in the terraform scripts. Those charts get installed on the Terraform call, so it happens directly on the public cloud provider, hitting their container mirrors. I think it's safe to ignore them.

From the quarks ones:

cfcontainerization/cf-operator:v6.1.17-0.gec409fd7
cfcontainerization/coredns:0.1.0-1.6.7-bp152.1.19
cfcontainerization/quarks-job:v1.0.206
cfcontainerization/quarks-secret:v1.0.744

Only coredns is not changed yet. The image comes from registry.opensuse.org, https://build.opensuse.org/package/show/Cloud:Platform:quarks:images/coredns, and is pushed here: https://concourse.suse.de/teams/main/pipelines/obs-to-dockerhub-pusher. Tim and I updated that pipeline and renamed it, now it is in https://concourse.suse.de/teams/main/pipelines/obs-to-ghcr-pusher, and coredns images are being published.

I will open a PR against quarks and link it here.

viccuad commented 4 years ago

I suppose we just need to consume a new quarks-operator release now, to close this card.

mook-as commented 4 years ago

Added https://github.com/SUSE/cap-terraform/pull/102 to make cap-terraform consume the upstream ingress-nginx chart instead of the bitnami one; that one uses gcr.io, so it should work around any potential Docker Hub quota issues.