Flux doesn't pull images data

acarmel commented 3 years ago

Flux is unable to scan image data and does't update my workloads when a new image is deployed.

list-images

WORKLOAD                                   CONTAINER  IMAGE           CREATED
my-namespace:deployment/backend         backend                    image data not available
                                                      '-> (untagged)  ?
my-namespace:deployment/flux            flux                       image data not available
                                                      '-> (untagged)  ?
my-namespace:deployment/flux-memcached  memcached                  image data not available
                                                      '-> (untagged)  ?
my-namespace:deployment/podinfo         podinfod                   image data not available
                                                      '-> (untagged)  ?

list-wokloads

WORKLOAD                                   CONTAINER  IMAGE                                                             RELEASE  POLICY
my-namespace:deployment/backend         backend    XXXXXXXXXXXX.dkr.ecr.us-east-1.amazonaws.com/my-project/backend  ready    automated
my-namespace:deployment/flux            flux       docker.io/fluxcd/flux:1.23.1                                      ready
my-namespace:deployment/flux-memcached  memcached  memcached:1.6.9-alpine                                            ready
my-namespace:deployment/podinfo         podinfod   ghcr.io/stefanprodan/podinfo:6.0.0                                ready    automated

The only relevant line I find in the log every few minutes.

ts=2021-07-31T20:54:26.372983478Z caller=images.go:17 component=sync-loop msg="polling for new images for automated workloads"

How I've installed flux

helm install flux fluxcd/flux -f my_values.yaml

# my_values.yaml

allowedNamespace: my-namespace
git:
  path: develop
  pollInterval: "1m"
  label: flux-sync-develop
  url: git@github.com:XXXXXX/XXXXXXX.git
  timeout: "2m"
manifestGeneration: true
rbac:
  create: false
serviceAccount:
  create: false
  name: flux
syncGarbageCollection:
  enabled: true

kubectl describe flux

Name:               flux
Namespace:          my-namespace
CreationTimestamp:  Sat, 31 Jul 2021 23:39:20 +0300
Labels:             app=flux
                    app.kubernetes.io/managed-by=Helm
                    chart=flux-1.10.1
                    heritage=Helm
                    release=flux
Annotations:        deployment.kubernetes.io/revision: 1
                    meta.helm.sh/release-name: flux
                    meta.helm.sh/release-namespace: my-namespace
Selector:           app=flux,release=flux
Replicas:           1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:       Recreate
MinReadySeconds:    0
Pod Template:
  Labels:           app=flux
                    release=flux
  Service Account:  flux
  Containers:
   flux:
    Image:      docker.io/fluxcd/flux:1.23.1
    Port:       3030/TCP
    Host Port:  0/TCP
    Args:
      --log-format=fmt
      --ssh-keygen-dir=/var/fluxd/keygen
      --ssh-keygen-format=RFC4716
      --k8s-secret-name=flux-git-deploy
      --memcached-hostname=flux-memcached
      --sync-state=git
      --memcached-service=
      --git-url=git@github.com:XXXXXX/XXXXXXX.git
      --git-branch=master
      --git-path=develop
      --git-readonly=false
      --git-user=Weave Flux
      --git-email=support@weave.works
      --git-verify-signatures=false
      --git-set-author=false
      --git-poll-interval=1m
      --git-timeout=2m
      --sync-interval=1m
      --git-ci-skip=false
      --git-label=flux-sync-develop
      --manifest-generation=true
      --automation-interval=5m
      --registry-rps=200
      --registry-burst=125
      --registry-trace=false
      --sync-garbage-collection=true
    Requests:
      cpu:      50m
      memory:   64Mi
    Liveness:   http-get http://:3030/api/flux/v6/identity.pub delay=5s timeout=5s period=10s #success=1 #failure=3
    Readiness:  http-get http://:3030/api/flux/v6/identity.pub delay=5s timeout=5s period=10s #success=1 #failure=3
    Environment:
      KUBECONFIG:  /root/.kubectl/config
    Mounts:
      /etc/fluxd/ssh from git-key (ro)
      /root/.kubectl from kubedir (rw)
      /var/fluxd/keygen from git-keygen (rw)
  Volumes:
   kubedir:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      flux-kube-config
    Optional:  false
   git-key:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  flux-git-deploy
    Optional:    false
   git-keygen:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   flux-694dd8dc5 (1/1 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  20m   deployment-controller  Scaled up replica set flux-694dd8dc5 to 1

The ec2 node's role where Flux runs on has this policy

What should I do to solve it?

kingdonb commented 3 years ago

Hello, thanks for submitting this issue. First, are you sure you're not being rate limited? There is not enough log here to judge, there are many problems that could be affecting you which are not possible to resolve in Flux v1, as clarified in the Flux v1 FAQ: https://fluxcd.io/legacy/flux/faq/#why-should-i-upgrade – the initial startup sequence of Flux v1 when automation is in play is long and slow, especially if any involved image registries are rate limited.

That might not be your issue though, we've received a lot of reports from ECR users lately and some may be fixed in the next release:

There are recent issues and merges that might be related to your problem here,

3492
and ~#3124~
(eventually merged as #3485),
which is addressing #3015

Please include a larger section of the log, ~as well as the specific version of Flux that you are using~ (I found it in the workloads list, version 1.23.1 is still the latest version for the moment...) It will also be helpful to get more details about your cluster, but as you showed you are on AWS, I think I would guess you are using EKS.

You may try setting your Flux daemon deployment to use the fluxcd/flux-prerelease:master-7218abf9 image to see if #3485 solves your issue. If we can see more users are affected, I think it will affect the priority of this release, so if you can test it out and let us know if it helps or not, this will be beneficial.

From experience debugging this issue with other users recently, if your cluster is on EKS and you are following best practices, then you are probably affected by #3015 and need this fix. The flux pod cannot reach the AWS instance metadata API and so it balks. You might find the message AWS auth implied by ECR image, but AWS API is not available in your logs if that is the case, but I think you might have mentioned that if it appeared, so maybe you're having a different issue here.

Thanks for your report, I will look forward to your reply.

kingdonb commented 3 years ago

Based on the log line you showed, I would be more inclined to blame rate limiting than the ECR issue.

It is important to know that Flux may scan all tags for every image across every registry, and syncing+automation will not be performed until the scan is complete.

This is the design limitation addressed in the FAQ entry that I linked. Pulling every image header to retrieve the build-time metadata is slow and costly, this was the driving factor in the new requirement found in Flux v2, to enable automation you must use sortable image tags one way or another. It is not enough to simply tag your images with branch and SHA, as it has been done with Flux v1 in the past. The tag string must contain a serial number or timestamp, or other component providing a lexical sort order, so they can be ordered without pulling the metadata and header of each image layer separately; this avoids the rate limiting issue we commonly see reported here, affecting Flux v1.

Users of Flux are all encouraged to upgrade to Flux v2 as soon as possible. While Flux v1 remains in support for now, the hope is to migrate all users according to the migration timetable. As you can see from the timetable, Flux v1 was marked as "superseded" when the Flux v2 API reached the stability milestone late last quarter, around the end of June.

It will be helpful for us to know also, is this a new deployment of Flux or is it a long-standing deployment that you have been using for a while? (If it was a new deployment, I'd like to ask where you got the idea of installing Flux v1, the legacy version?) As I explained we're trying to gently push all users towards the new Flux version, as it is actively developed, has been pushed across the feature parity milestone for a while already, and gets a lot more attention from our team than Flux v1 at this point.

kingdonb commented 3 years ago

Sorry, I missed an important detail about #3015

You will need to set the ECR region in your Flux config in order to take advantage of this. Flux does a pre-flight check to see if it can reach the AWS Instance Metadata API, and if it fails, it will mark AWS ECR API as "not OK to use". The change for #3015 circumvents this by assuming it is OK to use the ECR API, as long as you've provided some region(s) that it can be used in.

So you will need to include the ECR region as well, or upgrading to the pre-release image I suggested will not do anything.

This is --registry-ecr-region=us-east-1 in Flux Daemon, or in your Helm values file:

registry:
  ecr:
    region: us-east-1

kingdonb commented 3 years ago

I've just spent some time with the e2e tests while verifying that Flux v1 will work against the next Kubernetes release due out this coming week, and I have come back around to this report because I'm afraid I've presented too much info, and much of it is not helpful to your case. My explanation related to (#3015) may not have been related to your issue at all.

My apologies for taking you down the wrong road. In my testing, I ran into this log line which you mentioned having seen:

ts=2021-07-31T20:54:26.372983478Z caller=images.go:17 component=sync-loop msg="polling for new images for automated workloads"

This indicates simply that Flux isn't finished populating the memcache yet, which is where I began. I misunderstood a little something about the rate limit issue: Flux will also rate-limit itself, which means that not all users experience the rate limiting in the same way. You don't always get "backoff" messages – those are only if you are rate limited by the registry server, it looks different than when Flux has rate limited itself.

What you are seeing, if that message repeats for a long time, is the effect of Flux rate limiting itself. By the time you read this, it may have even already resolved itself.

You can adjust the rate limit parameters in your deployment with chart values corresponding to these items:

      --registry-rps=200
      --registry-burst=125

... but the rate limit behavior can also come from the registry server, which in that case, sees Flux making too many requests and asks it to slow down. It can be both sources of rate limiting, Flux itself, and the registry server. So, setting those values too high increases the risk you will rapidly be rate limited by the registry server, and it doesn't necessarily finish faster then.

So, I haven't seen your whole log, and I'm not sure what other parts might be relevant, so I'm guessing a bit, but if everything is as I expect, and you really haven't omitted anything important, then it was really this issue:

Flux v1 runtime behavior doesn’t scale well

Granted the other issue might also present. It can be both issues.

If you are only using images on your cluster that are pulled from registries that you own, then you can probably pump up the registry-rps and burst setting values safely, at least until it swamps the registry, but ideally until image automation performs acceptably.

The primary recommendation remains the same in any case, upgrading to Flux v2 is the best recommendation I can give that will help resolve this.

acarmel commented 3 years ago

Hey, First of all WOW huge thanks for the quick and detailed answer.

Regarding your questions:

1) Yes, we run on EKS. server api version is 1.18. 2) I'm attaching more log lines, but the log repeats itself and I don't see anything relevant other than the polling for new images for automated workloads line.

ts=2021-08-01T20:56:39.909123265Z caller=sync.go:64 component=daemon info="trying to sync git changes to the cluster" old=aef066a44f8f0c385ffabd5a2795e063962ccfa7 new=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T20:56:40.896419153Z caller=sync.go:540 method=Sync cmd=apply args= count=7
ts=2021-08-01T20:56:41.216244562Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=319.757197ms err=null output="service/backend unchanged\nserviceaccount/backend unchanged\ningress.networking.k8s.io/backend configured\nconfigmap/backend-env-55d52ggc45 unchanged\ndeployment.apps/backend unchanged\nservicemonitor.monitoring.coreos.com/backend-monitor unchanged\nsealedsecret.bitnami.com/backend-secrets unchanged"
ts=2021-08-01T20:56:49.284013509Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T20:57:42.302472814Z caller=sync.go:64 component=daemon info="trying to sync git changes to the cluster" old=aef066a44f8f0c385ffabd5a2795e063962ccfa7 new=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T20:57:43.331512041Z caller=sync.go:540 method=Sync cmd=apply args= count=7
ts=2021-08-01T20:57:43.728462891Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=396.881787ms err=null output="service/backend unchanged\nserviceaccount/backend unchanged\ningress.networking.k8s.io/backend configured\nconfigmap/backend-env-55d52ggc45 unchanged\ndeployment.apps/backend unchanged\nservicemonitor.monitoring.coreos.com/backend-monitor unchanged\nsealedsecret.bitnami.com/backend-secrets unchanged"
ts=2021-08-01T20:57:49.570083571Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T20:58:44.688594689Z caller=sync.go:64 component=daemon info="trying to sync git changes to the cluster" old=aef066a44f8f0c385ffabd5a2795e063962ccfa7 new=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T20:58:45.672249264Z caller=sync.go:540 method=Sync cmd=apply args= count=7
ts=2021-08-01T20:58:45.966031188Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=293.726063ms err=null output="serviceaccount/backend unchanged\nservice/backend unchanged\ningress.networking.k8s.io/backend configured\nconfigmap/backend-env-55d52ggc45 unchanged\ndeployment.apps/backend unchanged\nservicemonitor.monitoring.coreos.com/backend-monitor unchanged\nsealedsecret.bitnami.com/backend-secrets unchanged"
ts=2021-08-01T20:58:49.868956279Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T20:59:38.216085451Z caller=images.go:17 component=sync-loop msg="polling for new images for automated workloads"
ts=2021-08-01T20:59:47.074491916Z caller=sync.go:64 component=daemon info="trying to sync git changes to the cluster" old=aef066a44f8f0c385ffabd5a2795e063962ccfa7 new=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T20:59:48.035723295Z caller=sync.go:540 method=Sync cmd=apply args= count=7
ts=2021-08-01T20:59:48.398748094Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=362.972688ms err=null output="service/backend unchanged\nserviceaccount/backend unchanged\ningress.networking.k8s.io/backend configured\nconfigmap/backend-env-55d52ggc45 unchanged\ndeployment.apps/backend unchanged\nservicemonitor.monitoring.coreos.com/backend-monitor unchanged\nsealedsecret.bitnami.com/backend-secrets unchanged"
ts=2021-08-01T20:59:50.413735125Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:00:49.44801872Z caller=sync.go:64 component=daemon info="trying to sync git changes to the cluster" old=aef066a44f8f0c385ffabd5a2795e063962ccfa7 new=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:00:50.659166006Z caller=sync.go:540 method=Sync cmd=apply args= count=7
ts=2021-08-01T21:00:51.370645602Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=711.411336ms err=null output="serviceaccount/backend unchanged\nservice/backend unchanged\ningress.networking.k8s.io/backend configured\nconfigmap/backend-env-55d52ggc45 unchanged\ndeployment.apps/backend unchanged\nservicemonitor.monitoring.coreos.com/backend-monitor unchanged\nsealedsecret.bitnami.com/backend-secrets unchanged"
ts=2021-08-01T21:00:52.067532915Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:01:51.083924699Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:01:52.06974697Z caller=sync.go:64 component=daemon info="trying to sync git changes to the cluster" old=aef066a44f8f0c385ffabd5a2795e063962ccfa7 new=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:01:53.074135001Z caller=sync.go:540 method=Sync cmd=apply args= count=7
ts=2021-08-01T21:01:54.075157413Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=1.000964055s err=null output="service/backend unchanged\nserviceaccount/backend unchanged\ningress.networking.k8s.io/backend configured\nconfigmap/backend-env-55d52ggc45 unchanged\ndeployment.apps/backend unchanged\nservicemonitor.monitoring.coreos.com/backend-monitor unchanged\nsealedsecret.bitnami.com/backend-secrets unchanged"
ts=2021-08-01T21:02:51.389522339Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:02:54.454999645Z caller=sync.go:64 component=daemon info="trying to sync git changes to the cluster" old=aef066a44f8f0c385ffabd5a2795e063962ccfa7 new=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:02:55.881457366Z caller=sync.go:540 method=Sync cmd=apply args= count=7
ts=2021-08-01T21:02:56.295431283Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=413.907946ms err=null output="serviceaccount/backend unchanged\nservice/backend unchanged\ningress.networking.k8s.io/backend configured\nconfigmap/backend-env-55d52ggc45 unchanged\ndeployment.apps/backend unchanged\nservicemonitor.monitoring.coreos.com/backend-monitor unchanged\nsealedsecret.bitnami.com/backend-secrets unchanged"
ts=2021-08-01T21:03:51.712866238Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:03:57.205717459Z caller=sync.go:64 component=daemon info="trying to sync git changes to the cluster" old=aef066a44f8f0c385ffabd5a2795e063962ccfa7 new=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:03:58.304139247Z caller=sync.go:540 method=Sync cmd=apply args= count=7
ts=2021-08-01T21:03:58.677555562Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=373.344913ms err=null output="serviceaccount/backend unchanged\nservice/backend unchanged\ningress.networking.k8s.io/backend configured\nconfigmap/backend-env-55d52ggc45 unchanged\ndeployment.apps/backend unchanged\nservicemonitor.monitoring.coreos.com/backend-monitor unchanged\nsealedsecret.bitnami.com/backend-secrets unchanged"
ts=2021-08-01T21:04:39.104865813Z caller=images.go:17 component=sync-loop msg="polling for new images for automated workloads"
ts=2021-08-01T21:04:52.019427287Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:04:59.722458445Z caller=sync.go:64 component=daemon info="trying to sync git changes to the cluster" old=aef066a44f8f0c385ffabd5a2795e063962ccfa7 new=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:05:00.878612982Z caller=sync.go:540 method=Sync cmd=apply args= count=7
ts=2021-08-01T21:05:01.214617486Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=335.948379ms err=null output="service/backend unchanged\nserviceaccount/backend unchanged\ningress.networking.k8s.io/backend configured\nconfigmap/backend-env-55d52ggc45 unchanged\ndeployment.apps/backend unchanged\nservicemonitor.monitoring.coreos.com/backend-monitor unchanged\nsealedsecret.bitnami.com/backend-secrets unchanged"
ts=2021-08-01T21:05:52.321897527Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:06:02.278157351Z caller=sync.go:64 component=daemon info="trying to sync git changes to the cluster" old=aef066a44f8f0c385ffabd5a2795e063962ccfa7 new=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:06:03.332789012Z caller=sync.go:540 method=Sync cmd=apply args= count=7
ts=2021-08-01T21:06:03.756006379Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=423.163111ms err=null output="service/backend unchanged\nserviceaccount/backend unchanged\ningress.networking.k8s.io/backend configured\nconfigmap/backend-env-55d52ggc45 unchanged\ndeployment.apps/backend unchanged\nservicemonitor.monitoring.coreos.com/backend-monitor unchanged\nsealedsecret.bitnami.com/backend-secrets unchanged"
ts=2021-08-01T21:06:52.622832049Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:07:04.754037025Z caller=sync.go:64 component=daemon info="trying to sync git changes to the cluster" old=aef066a44f8f0c385ffabd5a2795e063962ccfa7 new=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:07:05.763362366Z caller=sync.go:540 method=Sync cmd=apply args= count=7
ts=2021-08-01T21:07:06.161167306Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=397.746148ms err=null output="service/backend unchanged\nserviceaccount/backend unchanged\ningress.networking.k8s.io/backend configured\nconfigmap/backend-env-55d52ggc45 unchanged\ndeployment.apps/backend unchanged\nservicemonitor.monitoring.coreos.com/backend-monitor unchanged\nsealedsecret.bitnami.com/backend-secrets unchanged"
ts=2021-08-01T21:07:52.927858219Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:08:07.191160992Z caller=sync.go:64 component=daemon info="trying to sync git changes to the cluster" old=aef066a44f8f0c385ffabd5a2795e063962ccfa7 new=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:08:08.179469147Z caller=sync.go:540 method=Sync cmd=apply args= count=7
ts=2021-08-01T21:08:08.487463161Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=307.929979ms err=null output="service/backend unchanged\nserviceaccount/backend unchanged\ningress.networking.k8s.io/backend configured\nconfigmap/backend-env-55d52ggc45 unchanged\ndeployment.apps/backend unchanged\nservicemonitor.monitoring.coreos.com/backend-monitor unchanged\nsealedsecret.bitnami.com/backend-secrets unchanged"
ts=2021-08-01T21:08:53.269127215Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:09:09.606843931Z caller=sync.go:64 component=daemon info="trying to sync git changes to the cluster" old=aef066a44f8f0c385ffabd5a2795e063962ccfa7 new=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:09:10.683535787Z caller=sync.go:540 method=Sync cmd=apply args= count=7
ts=2021-08-01T21:09:11.070913097Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=387.324344ms err=null output="service/backend unchanged\nserviceaccount/backend unchanged\ningress.networking.k8s.io/backend configured\nconfigmap/backend-env-55d52ggc45 unchanged\ndeployment.apps/backend unchanged\nservicemonitor.monitoring.coreos.com/backend-monitor unchanged\nsealedsecret.bitnami.com/backend-secrets unchanged"
ts=2021-08-01T21:09:40.035857866Z caller=images.go:17 component=sync-loop msg="polling for new images for automated workloads"
ts=2021-08-01T21:09:53.589126963Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:10:12.158856606Z caller=sync.go:64 component=daemon info="trying to sync git changes to the cluster" old=aef066a44f8f0c385ffabd5a2795e063962ccfa7 new=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:10:13.16620866Z caller=sync.go:540 method=Sync cmd=apply args= count=7
ts=2021-08-01T21:10:13.628069706Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=461.795448ms err=null output="serviceaccount/backend unchanged\nservice/backend unchanged\ningress.networking.k8s.io/backend configured\nconfigmap/backend-env-55d52ggc45 unchanged\ndeployment.apps/backend unchanged\nservicemonitor.monitoring.coreos.com/backend-monitor unchanged\nsealedsecret.bitnami.com/backend-secrets unchanged"
ts=2021-08-01T21:10:53.902899672Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:11:14.571512581Z caller=sync.go:64 component=daemon info="trying to sync git changes to the cluster" old=aef066a44f8f0c385ffabd5a2795e063962ccfa7 new=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:11:15.591292262Z caller=sync.go:540 method=Sync cmd=apply args= count=7
ts=2021-08-01T21:11:16.145367348Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=554.019752ms err=null output="service/backend unchanged\nserviceaccount/backend unchanged\ningress.networking.k8s.io/backend configured\nconfigmap/backend-env-55d52ggc45 unchanged\ndeployment.apps/backend unchanged\nservicemonitor.monitoring.coreos.com/backend-monitor unchanged\nsealedsecret.bitnami.com/backend-secrets unchanged"
ts=2021-08-01T21:11:54.197163579Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:12:17.021414984Z caller=sync.go:64 component=daemon info="trying to sync git changes to the cluster" old=aef066a44f8f0c385ffabd5a2795e063962ccfa7 new=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:12:18.235318431Z caller=sync.go:540 method=Sync cmd=apply args= count=7
ts=2021-08-01T21:12:18.531076019Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=295.685556ms err=null output="service/backend unchanged\nserviceaccount/backend unchanged\ningress.networking.k8s.io/backend configured\nconfigmap/backend-env-55d52ggc45 unchanged\ndeployment.apps/backend unchanged\nservicemonitor.monitoring.coreos.com/backend-monitor unchanged\nsealedsecret.bitnami.com/backend-secrets unchanged"
ts=2021-08-01T21:12:54.499534536Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:13:19.640855705Z caller=sync.go:64 component=daemon info="trying to sync git changes to the cluster" old=aef066a44f8f0c385ffabd5a2795e063962ccfa7 new=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:13:20.625496518Z caller=sync.go:540 method=Sync cmd=apply args= count=7
ts=2021-08-01T21:13:21.14518115Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=519.633887ms err=null output="service/backend unchanged\nserviceaccount/backend unchanged\ningress.networking.k8s.io/backend configured\nconfigmap/backend-env-55d52ggc45 unchanged\ndeployment.apps/backend unchanged\nservicemonitor.monitoring.coreos.com/backend-monitor unchanged\nsealedsecret.bitnami.com/backend-secrets unchanged"
ts=2021-08-01T21:13:54.802310936Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:14:22.048321723Z caller=sync.go:64 component=daemon info="trying to sync git changes to the cluster" old=aef066a44f8f0c385ffabd5a2795e063962ccfa7 new=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:14:23.026535316Z caller=sync.go:540 method=Sync cmd=apply args= count=7
ts=2021-08-01T21:14:23.409100701Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=382.507024ms err=null output="service/backend unchanged\nserviceaccount/backend unchanged\ningress.networking.k8s.io/backend configured\nconfigmap/backend-env-55d52ggc45 unchanged\ndeployment.apps/backend unchanged\nservicemonitor.monitoring.coreos.com/backend-monitor unchanged\nsealedsecret.bitnami.com/backend-secrets unchanged"
ts=2021-08-01T21:14:41.061585083Z caller=images.go:17 component=sync-loop msg="polling for new images for automated workloads"
ts=2021-08-01T21:14:55.103600562Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:15:24.440967767Z caller=sync.go:64 component=daemon info="trying to sync git changes to the cluster" old=aef066a44f8f0c385ffabd5a2795e063962ccfa7 new=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:15:25.537890667Z caller=sync.go:540 method=Sync cmd=apply args= count=7
ts=2021-08-01T21:15:25.872723981Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=334.779407ms err=null output="service/backend unchanged\nserviceaccount/backend unchanged\ningress.networking.k8s.io/backend configured\nconfigmap/backend-env-55d52ggc45 unchanged\ndeployment.apps/backend unchanged\nservicemonitor.monitoring.coreos.com/backend-monitor unchanged\nsealedsecret.bitnami.com/backend-secrets unchanged"
ts=2021-08-01T21:15:55.405359717Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:16:26.955478618Z caller=sync.go:64 component=daemon info="trying to sync git changes to the cluster" old=aef066a44f8f0c385ffabd5a2795e063962ccfa7 new=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:16:27.935202365Z caller=sync.go:540 method=Sync cmd=apply args= count=7
ts=2021-08-01T21:16:28.237996147Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=302.73827ms err=null output="service/backend unchanged\nserviceaccount/backend unchanged\ningress.networking.k8s.io/backend configured\nconfigmap/backend-env-55d52ggc45 unchanged\ndeployment.apps/backend unchanged\nservicemonitor.monitoring.coreos.com/backend-monitor unchanged\nsealedsecret.bitnami.com/backend-secrets unchanged"
ts=2021-08-01T21:16:55.717832796Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:17:29.34634747Z caller=sync.go:64 component=daemon info="trying to sync git changes to the cluster" old=aef066a44f8f0c385ffabd5a2795e063962ccfa7 new=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:17:30.485404593Z caller=sync.go:540 method=Sync cmd=apply args= count=7
ts=2021-08-01T21:17:30.783516355Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=298.051958ms err=null output="service/backend unchanged\nserviceaccount/backend unchanged\ningress.networking.k8s.io/backend configured\nconfigmap/backend-env-55d52ggc45 unchanged\ndeployment.apps/backend unchanged\nservicemonitor.monitoring.coreos.com/backend-monitor unchanged\nsealedsecret.bitnami.com/backend-secrets unchanged"
ts=2021-08-01T21:17:56.066249557Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:18:31.906086664Z caller=sync.go:64 component=daemon info="trying to sync git changes to the cluster" old=aef066a44f8f0c385ffabd5a2795e063962ccfa7 new=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:18:32.896554103Z caller=sync.go:540 method=Sync cmd=apply args= count=7
ts=2021-08-01T21:18:33.198508002Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=301.886633ms err=null output="serviceaccount/backend unchanged\nservice/backend unchanged\ningress.networking.k8s.io/backend configured\nconfigmap/backend-env-55d52ggc45 unchanged\ndeployment.apps/backend unchanged\nservicemonitor.monitoring.coreos.com/backend-monitor unchanged\nsealedsecret.bitnami.com/backend-secrets unchanged"
ts=2021-08-01T21:18:56.369218244Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:19:34.294417214Z caller=sync.go:64 component=daemon info="trying to sync git changes to the cluster" old=aef066a44f8f0c385ffabd5a2795e063962ccfa7 new=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:19:35.261969661Z caller=sync.go:540 method=Sync cmd=apply args= count=7
ts=2021-08-01T21:19:35.781910951Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=519.875951ms err=null output="service/backend unchanged\nserviceaccount/backend unchanged\ningress.networking.k8s.io/backend configured\nconfigmap/backend-env-55d52ggc45 unchanged\ndeployment.apps/backend unchanged\nservicemonitor.monitoring.coreos.com/backend-monitor unchanged\nsealedsecret.bitnami.com/backend-secrets unchanged"
ts=2021-08-01T21:19:41.96020793Z caller=images.go:17 component=sync-loop msg="polling for new images for automated workloads"
ts=2021-08-01T21:19:56.669146716Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:20:36.681272445Z caller=sync.go:64 component=daemon info="trying to sync git changes to the cluster" old=aef066a44f8f0c385ffabd5a2795e063962ccfa7 new=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:20:37.690838366Z caller=sync.go:540 method=Sync cmd=apply args= count=7
ts=2021-08-01T21:20:37.975595806Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=284.706833ms err=null output="service/backend unchanged\nserviceaccount/backend unchanged\ningress.networking.k8s.io/backend configured\nconfigmap/backend-env-55d52ggc45 unchanged\ndeployment.apps/backend unchanged\nservicemonitor.monitoring.coreos.com/backend-monitor unchanged\nsealedsecret.bitnami.com/backend-secrets unchanged"
ts=2021-08-01T21:20:56.97140291Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:21:39.08459241Z caller=sync.go:64 component=daemon info="trying to sync git changes to the cluster" old=aef066a44f8f0c385ffabd5a2795e063962ccfa7 new=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:21:40.159859004Z caller=sync.go:540 method=Sync cmd=apply args= count=7
ts=2021-08-01T21:21:40.710433827Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=550.517387ms err=null output="service/backend unchanged\nserviceaccount/backend unchanged\ningress.networking.k8s.io/backend configured\nconfigmap/backend-env-55d52ggc45 unchanged\ndeployment.apps/backend unchanged\nservicemonitor.monitoring.coreos.com/backend-monitor unchanged\nsealedsecret.bitnami.com/backend-secrets unchanged"
ts=2021-08-01T21:21:57.289100063Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:22:41.562995314Z caller=sync.go:64 component=daemon info="trying to sync git changes to the cluster" old=aef066a44f8f0c385ffabd5a2795e063962ccfa7 new=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:22:42.55215405Z caller=sync.go:540 method=Sync cmd=apply args= count=7
ts=2021-08-01T21:22:43.00499271Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=452.770616ms err=null output="service/backend unchanged\nserviceaccount/backend unchanged\ningress.networking.k8s.io/backend configured\nconfigmap/backend-env-55d52ggc45 unchanged\ndeployment.apps/backend unchanged\nservicemonitor.monitoring.coreos.com/backend-monitor unchanged\nsealedsecret.bitnami.com/backend-secrets unchanged"
ts=2021-08-01T21:22:57.591316539Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:23:43.963655885Z caller=sync.go:64 component=daemon info="trying to sync git changes to the cluster" old=aef066a44f8f0c385ffabd5a2795e063962ccfa7 new=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:23:44.916133448Z caller=sync.go:540 method=Sync cmd=apply args= count=7
ts=2021-08-01T21:23:45.300646746Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=384.454114ms err=null output="service/backend unchanged\nserviceaccount/backend unchanged\ningress.networking.k8s.io/backend configured\nconfigmap/backend-env-55d52ggc45 unchanged\ndeployment.apps/backend unchanged\nservicemonitor.monitoring.coreos.com/backend-monitor unchanged\nsealedsecret.bitnami.com/backend-secrets unchanged"
ts=2021-08-01T21:23:57.904386245Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:24:43.251963774Z caller=images.go:17 component=sync-loop msg="polling for new images for automated workloads"
ts=2021-08-01T21:24:46.346900516Z caller=sync.go:64 component=daemon info="trying to sync git changes to the cluster" old=aef066a44f8f0c385ffabd5a2795e063962ccfa7 new=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:24:47.489961001Z caller=sync.go:540 method=Sync cmd=apply args= count=7
ts=2021-08-01T21:24:47.877870842Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=387.851347ms err=null output="service/backend unchanged\nserviceaccount/backend unchanged\ningress.networking.k8s.io/backend configured\nconfigmap/backend-env-55d52ggc45 unchanged\ndeployment.apps/backend unchanged\nservicemonitor.monitoring.coreos.com/backend-monitor unchanged\nsealedsecret.bitnami.com/backend-secrets unchanged"
ts=2021-08-01T21:24:58.220738373Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:25:48.920841851Z caller=sync.go:64 component=daemon info="trying to sync git changes to the cluster" old=aef066a44f8f0c385ffabd5a2795e063962ccfa7 new=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:25:50.114431056Z caller=sync.go:540 method=Sync cmd=apply args= count=7
ts=2021-08-01T21:25:50.399022161Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=284.536328ms err=null output="service/backend unchanged\nserviceaccount/backend unchanged\ningress.networking.k8s.io/backend configured\nconfigmap/backend-env-55d52ggc45 unchanged\ndeployment.apps/backend unchanged\nservicemonitor.monitoring.coreos.com/backend-monitor unchanged\nsealedsecret.bitnami.com/backend-secrets unchanged"
ts=2021-08-01T21:25:58.526450556Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:26:51.528128536Z caller=sync.go:64 component=daemon info="trying to sync git changes to the cluster" old=aef066a44f8f0c385ffabd5a2795e063962ccfa7 new=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:26:52.675566565Z caller=sync.go:540 method=Sync cmd=apply args= count=7
ts=2021-08-01T21:26:52.979909236Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=304.281163ms err=null output="serviceaccount/backend unchanged\nservice/backend unchanged\ningress.networking.k8s.io/backend configured\nconfigmap/backend-env-55d52ggc45 unchanged\ndeployment.apps/backend unchanged\nservicemonitor.monitoring.coreos.com/backend-monitor unchanged\nsealedsecret.bitnami.com/backend-secrets unchanged"
ts=2021-08-01T21:26:58.827420768Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:27:54.078624513Z caller=sync.go:64 component=daemon info="trying to sync git changes to the cluster" old=aef066a44f8f0c385ffabd5a2795e063962ccfa7 new=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:27:55.200168753Z caller=sync.go:540 method=Sync cmd=apply args= count=7
ts=2021-08-01T21:27:55.590986826Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=390.745675ms err=null output="serviceaccount/backend unchanged\nservice/backend unchanged\ningress.networking.k8s.io/backend configured\nconfigmap/backend-env-55d52ggc45 unchanged\ndeployment.apps/backend unchanged\nservicemonitor.monitoring.coreos.com/backend-monitor unchanged\nsealedsecret.bitnami.com/backend-secrets unchanged"
ts=2021-08-01T21:27:59.135261989Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:28:56.485123578Z caller=sync.go:64 component=daemon info="trying to sync git changes to the cluster" old=aef066a44f8f0c385ffabd5a2795e063962ccfa7 new=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:28:57.518103763Z caller=sync.go:540 method=Sync cmd=apply args= count=7
ts=2021-08-01T21:28:57.959596Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=441.432739ms err=null output="service/backend unchanged\nserviceaccount/backend unchanged\ningress.networking.k8s.io/backend configured\nconfigmap/backend-env-55d52ggc45 unchanged\ndeployment.apps/backend unchanged\nservicemonitor.monitoring.coreos.com/backend-monitor unchanged\nsealedsecret.bitnami.com/backend-secrets unchanged"
ts=2021-08-01T21:28:59.432210294Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:29:44.187984608Z caller=images.go:17 component=sync-loop msg="polling for new images for automated workloads"
ts=2021-08-01T21:29:58.899472426Z caller=sync.go:64 component=daemon info="trying to sync git changes to the cluster" old=aef066a44f8f0c385ffabd5a2795e063962ccfa7 new=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:29:59.867586151Z caller=sync.go:540 method=Sync cmd=apply args= count=7
ts=2021-08-01T21:30:00.270957531Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=403.294217ms err=null output="service/backend unchanged\nserviceaccount/backend unchanged\ningress.networking.k8s.io/backend configured\nconfigmap/backend-env-55d52ggc45 unchanged\ndeployment.apps/backend unchanged\nservicemonitor.monitoring.coreos.com/backend-monitor unchanged\nsealedsecret.bitnami.com/backend-secrets unchanged"
ts=2021-08-01T21:30:01.291269807Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:31:00.06452102Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:31:01.292643236Z caller=sync.go:64 component=daemon info="trying to sync git changes to the cluster" old=aef066a44f8f0c385ffabd5a2795e063962ccfa7 new=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:31:02.352991834Z caller=sync.go:540 method=Sync cmd=apply args= count=7
ts=2021-08-01T21:31:02.760741963Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=407.680374ms err=null output="service/backend unchanged\nserviceaccount/backend unchanged\ningress.networking.k8s.io/backend configured\nconfigmap/backend-env-55d52ggc45 unchanged\ndeployment.apps/backend unchanged\nservicemonitor.monitoring.coreos.com/backend-monitor unchanged\nsealedsecret.bitnami.com/backend-secrets unchanged"
ts=2021-08-01T21:32:00.362042324Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:32:03.761913435Z caller=sync.go:64 component=daemon info="trying to sync git changes to the cluster" old=aef066a44f8f0c385ffabd5a2795e063962ccfa7 new=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:32:04.804951992Z caller=sync.go:540 method=Sync cmd=apply args= count=7
ts=2021-08-01T21:32:05.318453643Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=513.440387ms err=null output="service/backend unchanged\nserviceaccount/backend unchanged\ningress.networking.k8s.io/backend configured\nconfigmap/backend-env-55d52ggc45 unchanged\ndeployment.apps/backend unchanged\nservicemonitor.monitoring.coreos.com/backend-monitor unchanged\nsealedsecret.bitnami.com/backend-secrets unchanged"
ts=2021-08-01T21:33:00.685527771Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:33:06.170026833Z caller=sync.go:64 component=daemon info="trying to sync git changes to the cluster" old=aef066a44f8f0c385ffabd5a2795e063962ccfa7 new=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:33:07.163628863Z caller=sync.go:540 method=Sync cmd=apply args= count=7
ts=2021-08-01T21:33:07.544289349Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=380.605509ms err=null output="service/backend unchanged\nserviceaccount/backend unchanged\ningress.networking.k8s.io/backend configured\nconfigmap/backend-env-55d52ggc45 unchanged\ndeployment.apps/backend unchanged\nservicemonitor.monitoring.coreos.com/backend-monitor unchanged\nsealedsecret.bitnami.com/backend-secrets unchanged"
ts=2021-08-01T21:34:01.026864034Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:34:08.573202201Z caller=sync.go:64 component=daemon info="trying to sync git changes to the cluster" old=aef066a44f8f0c385ffabd5a2795e063962ccfa7 new=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:34:09.62541923Z caller=sync.go:540 method=Sync cmd=apply args= count=7
ts=2021-08-01T21:34:10.361202874Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=735.725683ms err=null output="service/backend unchanged\nserviceaccount/backend unchanged\ningress.networking.k8s.io/backend configured\nconfigmap/backend-env-55d52ggc45 unchanged\ndeployment.apps/backend unchanged\nservicemonitor.monitoring.coreos.com/backend-monitor unchanged\nsealedsecret.bitnami.com/backend-secrets unchanged"
ts=2021-08-01T21:34:45.074657164Z caller=images.go:17 component=sync-loop msg="polling for new images for automated workloads"
ts=2021-08-01T21:35:01.761527848Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:35:11.04525228Z caller=sync.go:64 component=daemon info="trying to sync git changes to the cluster" old=aef066a44f8f0c385ffabd5a2795e063962ccfa7 new=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:35:12.098170952Z caller=sync.go:540 method=Sync cmd=apply args= count=7
ts=2021-08-01T21:35:12.42411077Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=325.886317ms err=null output="service/backend unchanged\nserviceaccount/backend unchanged\ningress.networking.k8s.io/backend configured\nconfigmap/backend-env-55d52ggc45 unchanged\ndeployment.apps/backend unchanged\nservicemonitor.monitoring.coreos.com/backend-monitor unchanged\nsealedsecret.bitnami.com/backend-secrets unchanged"
ts=2021-08-01T21:36:02.47707467Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:36:13.51181359Z caller=sync.go:64 component=daemon info="trying to sync git changes to the cluster" old=aef066a44f8f0c385ffabd5a2795e063962ccfa7 new=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:36:14.514580267Z caller=sync.go:540 method=Sync cmd=apply args= count=7
ts=2021-08-01T21:36:14.814187814Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=299.55105ms err=null output="serviceaccount/backend unchanged\nservice/backend unchanged\ningress.networking.k8s.io/backend configured\nconfigmap/backend-env-55d52ggc45 unchanged\ndeployment.apps/backend unchanged\nservicemonitor.monitoring.coreos.com/backend-monitor unchanged\nsealedsecret.bitnami.com/backend-secrets unchanged"
ts=2021-08-01T21:37:02.87420869Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:37:15.941960954Z caller=sync.go:64 component=daemon info="trying to sync git changes to the cluster" old=aef066a44f8f0c385ffabd5a2795e063962ccfa7 new=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:37:17.059716866Z caller=sync.go:540 method=Sync cmd=apply args= count=7
ts=2021-08-01T21:37:17.458027494Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=398.25546ms err=null output="service/backend unchanged\nserviceaccount/backend unchanged\ningress.networking.k8s.io/backend configured\nconfigmap/backend-env-55d52ggc45 unchanged\ndeployment.apps/backend unchanged\nservicemonitor.monitoring.coreos.com/backend-monitor unchanged\nsealedsecret.bitnami.com/backend-secrets unchanged"
ts=2021-08-01T21:38:03.251880961Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:38:18.464738302Z caller=sync.go:64 component=daemon info="trying to sync git changes to the cluster" old=aef066a44f8f0c385ffabd5a2795e063962ccfa7 new=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:38:19.663080169Z caller=sync.go:540 method=Sync cmd=apply args= count=7
ts=2021-08-01T21:38:20.207803989Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=544.666265ms err=null output="serviceaccount/backend unchanged\nservice/backend unchanged\ningress.networking.k8s.io/backend configured\nconfigmap/backend-env-55d52ggc45 unchanged\ndeployment.apps/backend unchanged\nservicemonitor.monitoring.coreos.com/backend-monitor unchanged\nsealedsecret.bitnami.com/backend-secrets unchanged"
ts=2021-08-01T21:39:03.598406075Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:39:21.078762017Z caller=sync.go:64 component=daemon info="trying to sync git changes to the cluster" old=aef066a44f8f0c385ffabd5a2795e063962ccfa7 new=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:39:22.182339753Z caller=sync.go:540 method=Sync cmd=apply args= count=7
ts=2021-08-01T21:39:24.061753666Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=1.879358211s err=null output="serviceaccount/backend unchanged\nservice/backend unchanged\ningress.networking.k8s.io/backend configured\nconfigmap/backend-env-55d52ggc45 unchanged\ndeployment.apps/backend unchanged\nservicemonitor.monitoring.coreos.com/backend-monitor unchanged\nsealedsecret.bitnami.com/backend-secrets unchanged"
ts=2021-08-01T21:39:46.065539046Z caller=images.go:17 component=sync-loop msg="polling for new images for automated workloads"
ts=2021-08-01T21:40:04.584712595Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:40:24.14197195Z caller=sync.go:64 component=daemon info="trying to sync git changes to the cluster" old=aef066a44f8f0c385ffabd5a2795e063962ccfa7 new=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:40:25.221222591Z caller=sync.go:540 method=Sync cmd=apply args= count=7
ts=2021-08-01T21:40:25.947745605Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=726.465457ms err=null output="serviceaccount/backend unchanged\nservice/backend unchanged\ningress.networking.k8s.io/backend configured\nconfigmap/backend-env-55d52ggc45 unchanged\ndeployment.apps/backend unchanged\nservicemonitor.monitoring.coreos.com/backend-monitor unchanged\nsealedsecret.bitnami.com/backend-secrets unchanged"
ts=2021-08-01T21:41:04.999941937Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:41:26.636431147Z caller=sync.go:64 component=daemon info="trying to sync git changes to the cluster" old=aef066a44f8f0c385ffabd5a2795e063962ccfa7 new=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:41:27.592027816Z caller=sync.go:540 method=Sync cmd=apply args= count=7
ts=2021-08-01T21:41:28.284997865Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=692.9155ms err=null output="service/backend unchanged\nserviceaccount/backend unchanged\ningress.networking.k8s.io/backend configured\nconfigmap/backend-env-55d52ggc45 unchanged\ndeployment.apps/backend unchanged\nservicemonitor.monitoring.coreos.com/backend-monitor unchanged\nsealedsecret.bitnami.com/backend-secrets unchanged"
ts=2021-08-01T21:42:05.318110912Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:42:29.424278425Z caller=sync.go:64 component=daemon info="trying to sync git changes to the cluster" old=aef066a44f8f0c385ffabd5a2795e063962ccfa7 new=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:42:30.418028319Z caller=sync.go:540 method=Sync cmd=apply args= count=7
ts=2021-08-01T21:42:31.863666584Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=1.445581549s err=null output="serviceaccount/backend unchanged\nservice/backend unchanged\ningress.networking.k8s.io/backend configured\nconfigmap/backend-env-55d52ggc45 unchanged\ndeployment.apps/backend unchanged\nservicemonitor.monitoring.coreos.com/backend-monitor unchanged\nsealedsecret.bitnami.com/backend-secrets unchanged"
ts=2021-08-01T21:43:05.635369167Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:43:31.954134203Z caller=sync.go:64 component=daemon info="trying to sync git changes to the cluster" old=aef066a44f8f0c385ffabd5a2795e063962ccfa7 new=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:43:32.945807332Z caller=sync.go:540 method=Sync cmd=apply args= count=7
ts=2021-08-01T21:43:34.810769804Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=1.864906859s err=null output="service/backend unchanged\nserviceaccount/backend unchanged\ningress.networking.k8s.io/backend configured\nconfigmap/backend-env-55d52ggc45 unchanged\ndeployment.apps/backend unchanged\nservicemonitor.monitoring.coreos.com/backend-monitor unchanged\nsealedsecret.bitnami.com/backend-secrets unchanged"
ts=2021-08-01T21:44:05.95181597Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:44:35.171924511Z caller=sync.go:64 component=daemon info="trying to sync git changes to the cluster" old=aef066a44f8f0c385ffabd5a2795e063962ccfa7 new=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:44:36.331083004Z caller=sync.go:540 method=Sync cmd=apply args= count=7
ts=2021-08-01T21:44:37.577106688Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=1.24596716s err=null output="serviceaccount/backend unchanged\nservice/backend unchanged\ningress.networking.k8s.io/backend configured\nconfigmap/backend-env-55d52ggc45 unchanged\ndeployment.apps/backend unchanged\nservicemonitor.monitoring.coreos.com/backend-monitor unchanged\nsealedsecret.bitnami.com/backend-secrets unchanged"
ts=2021-08-01T21:44:46.999931664Z caller=images.go:17 component=sync-loop msg="polling for new images for automated workloads"
ts=2021-08-01T21:45:06.268047935Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:45:37.735095189Z caller=sync.go:64 component=daemon info="trying to sync git changes to the cluster" old=aef066a44f8f0c385ffabd5a2795e063962ccfa7 new=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:45:38.991452746Z caller=sync.go:540 method=Sync cmd=apply args= count=7
ts=2021-08-01T21:45:39.680266769Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=688.747923ms err=null output="service/backend unchanged\nserviceaccount/backend unchanged\ningress.networking.k8s.io/backend configured\nconfigmap/backend-env-55d52ggc45 unchanged\ndeployment.apps/backend unchanged\nservicemonitor.monitoring.coreos.com/backend-monitor unchanged\nsealedsecret.bitnami.com/backend-secrets unchanged"
ts=2021-08-01T21:46:06.594604824Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:46:40.383469852Z caller=sync.go:64 component=daemon info="trying to sync git changes to the cluster" old=aef066a44f8f0c385ffabd5a2795e063962ccfa7 new=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:46:41.402974649Z caller=sync.go:540 method=Sync cmd=apply args= count=7
ts=2021-08-01T21:46:41.747467137Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=344.370212ms err=null output="service/backend unchanged\nserviceaccount/backend unchanged\ningress.networking.k8s.io/backend configured\nconfigmap/backend-env-55d52ggc45 unchanged\ndeployment.apps/backend unchanged\nservicemonitor.monitoring.coreos.com/backend-monitor unchanged\nsealedsecret.bitnami.com/backend-secrets unchanged"
ts=2021-08-01T21:47:06.904631995Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:47:42.825388085Z caller=sync.go:64 component=daemon info="trying to sync git changes to the cluster" old=aef066a44f8f0c385ffabd5a2795e063962ccfa7 new=aef066a44f8f0c385ffabd5a2795e063962ccfa7
ts=2021-08-01T21:47:43.825518097Z caller=sync.go:540 method=Sync cmd=apply args= count=7
ts=2021-08-01T21:47:44.114531733Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=288.956963ms err=null output="service/backend unchanged\nserviceaccount/backend unchanged\ningress.networking.k8s.io/backend configured\nconfigmap/backend-env-55d52ggc45 unchanged\ndeployment.apps/backend unchanged\nservicemonitor.monitoring.coreos.com/backend-monitor unchanged\nsealedsecret.bitnami.com/backend-secrets unchanged"
ts=2021-08-01T21:48:07.223926575Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/XXXXXX/YYYYYYY.git branch=master HEAD=aef066a44f8f0c385ffabd5a2795e063962ccfa7

3) What is strange in theory about the problem with ECR is that only one image of my workloads is coming from ECR, The three other are not. And I still don't get data for all of them. 4) I found something interesting about rate limit. The logs of this Flux instance mention nothing about "rate limit" but I checked out the logs of another flux instance which runs on the cluster in another namespace and found tons of log lines on "rate limit". Is it possible that they are affecting each other? After all they run on the same, not big (8 nodes) cluster.

ts=2021-08-01T22:02:47.163098644Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/6.0-rc1-32bit-buster: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.163237052Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/6.0-rc1-32bit: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.163375661Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/3.2.6-windowsservercore: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.163433864Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/5.0-rc3-alpine: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.163761649Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/6.0-alpine3.12: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.164172279Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/5.0-rc6-32bit-stretch: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.164576606Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/6.0-rc-buster: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.164584133Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/3.2-alpine: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.164761262Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/3.2.12: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.164716747Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/5.0.9-alpine3.12: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.164996542Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/6.0.15-alpine3.14: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.165125353Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/3.2.7: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.166490798Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/3.2.1: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.167395581Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/6.0.12: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.167708556Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/6.0.14-buster: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.167728758Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/6.0.12-buster: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.167728525Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/6.0.13-alpine: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.168507445Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/5.0.3-32bit: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.16852109Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/6.0.2-buster: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.169331746Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/5.0.4-alpine: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.170500117Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/5.0.5-32bit-buster: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.170609819Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/5.0.4-stretch: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.170747951Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/5.0.5-32bit-stretch: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.17105754Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/5.0.4-32bit: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.171253188Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/5.0.5-alpine3.10: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.171642339Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/3-windowsservercore: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.171866119Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/5.0.4: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.172285654Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/5.0.5-alpine3.9: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.172479446Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/3.2.3-alpine: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.173464252Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/5.0.6-32bit-buster: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.173629776Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/5.0.5-alpine: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.173901444Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/4.0.9-32bit: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.174312291Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/3.2.0-alpine: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.175187064Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/3.2.5: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.175553987Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/5.0.5-buster: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.17564267Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/5.0.5-stretch: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.175731928Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/5.0.5: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.176139389Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/6.2-rc-buster: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.176547947Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/6.2.1-buster: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.176771503Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/4.0.11-stretch: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.177712396Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/6.2.2-alpine: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.178512027Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/5.0.5-32bit: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.178771917Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/6.0.8: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.178942052Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/3.2.6-alpine: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.179164854Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/4.0.10: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.179265465Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/3.2-nanoserver: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.179324067Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/4.0.12-32bit: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.179018061Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/6.2.3-alpine3.13: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.179876239Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/6.2.4-alpine3.14: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.180032569Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/6.2.4-buster: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.180051904Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/3.2.12-alpine: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.180339754Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/5.0.7-32bit: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.18038753Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/5.0.7-alpine: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.179234959Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/6.2-rc2-buster: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.180588113Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/5.0.7-alpine3.10: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.181039946Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/6.2.4-alpine3.13: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.181145228Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/4.0.11-alpine: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.181446827Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/5.0.7-buster: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.181454511Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/3.2.1-alpine: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.181567014Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/5.0.9-32bit-buster: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.182668914Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/3.0.6-32bit: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.182990143Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/4-alpine3.8: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.183941363Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/4.0.11: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.186794772Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/3-nanoserver: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.187051997Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/4.0-32bit-buster: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.187186167Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/3.2.12-32bit: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.187409852Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/4-32bit-stretch: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.187607709Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/3.2.3-32bit: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.18761711Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/4.0.10-alpine: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.187804405Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/6.0-rc-alpine: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:47.188413277Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="Get https://index.docker.io/v2/library/redis/manifests/3.2.10-alpine: rate limited: rate: Wait(n=1) would exceed context deadline" ref=redis:windowsservercore
ts=2021-08-01T22:02:56.508344372Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" ref=redis:windowsservercore
ts=2021-08-01T22:03:06.507224014Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" ref=redis:windowsservercore
ts=2021-08-01T22:03:11.663263618Z caller=sync.go:60 component=daemon info="trying to sync git changes to the cluster" old=09721420fb012ecb6107df5c6473b29346e95785 new=09721420fb012ecb6107df5c6473b29346e95785
ts=2021-08-01T22:03:12.835172201Z caller=sync.go:86 method=Sync info="not applying resource; ignore annotation in cluster resource" resource=some-ns-develop:deployment/another-project1
ts=2021-08-01T22:03:12.844699996Z caller=sync.go:540 method=Sync cmd=apply args= count=65
ts=2021-08-01T22:03:14.176154708Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=1.331345408s err=null output="service/accounts-service unchanged\nrole.rbac.authorization.k8s.io/allow-port-forward unchanged\nrole.rbac.authorization.k8s.io/allow-run-jobs unchanged\nservice/assets-events-metrics unchanged\nservice/assets-service unchanged\nserviceaccount/another-project1 unchanged\nservice/another-project1 unchanged\nservice/test-website-dev unchanged\nserviceaccount/r-service unchanged\nservice/r-service unchanged\nrole.rbac.authorization.k8s.io/deploy-render-pr unchanged\nservice/exports-service unchanged\nservice/zz-service unchanged\nservice/proxy unchanged\nservice/projects-service unchanged\nservice/redis unchanged\nservice/render-service unchanged\nservice/render-worker-metrics unchanged\nservice/web-gateway unchanged\nconfigmap/accounts-env-9cfkt8td7c unchanged\nconfigmap/asset-source-map unchanged\nconfigmap/assets-env-b5kkfgt4fd unchanged\ningress.networking.k8s.io/another-project1 configured\nconfigmap/another-project1-env-tdgtg2d897 unchanged\ningress.networking.k8s.io/test-ingress-dev configured\nconfigmap/r-env-997c7mc965 unchanged\nconfigmap/exports-env-gkb5bkh8ck unchanged\ningress.networking.k8s.io/zz-proxy configured\ningress.networking.k8s.io/zz-with- configured\nconfigmap/gateway-env-8k2b4fgbdk unchanged\ningress.networking.k8s.io/gateway-ingress configured\nconfigmap/projects-env-9mtm6552mh unchanged\nconfigmap/render-pipeline-env-d45g8gfgd9 unchanged\nconfigmap/service-env-fg6hdb454g unchanged\nrolebinding.rbac.authorization.k8s.io/another-ns-build-dev-user-allow-deploy configured\nrolebinding.rbac.authorization.k8s.io/another-ns-build-dev-user-allow-port-forward configured\nrolebinding.rbac.authorization.k8s.io/another-ns-build-dev-user-allow-run-jobs configured\ndeployment.apps/accounts-service unchanged\ncronjob.batch/accounts-verify unchanged\ndeployment.apps/assets-events unchanged\ndeployment.apps/assets-service unchanged\ncronjob.batch/assets-verify unchanged\ncronjob.batch/another-project1-deploy-to-preview unchanged\ncronjob.batch/another-project1-parse-configuration-jsons unchanged\ndeployment.apps/test-website-dev unchanged\ndeployment.apps/r-service unchanged\ncronjob.batch/r-verify unchanged\ndeployment.apps/exports-service unchanged\ncronjob.batch/exports-verify unchanged\ndeployment.apps/zz-service unchanged\ndeployment.apps/proxy unchanged\ndeployment.apps/projects-service unchanged\ncronjob.batch/projects-verify unchanged\ndeployment.apps/redis unchanged\ndeployment.apps/render-service unchanged\ndeployment.apps/render-service-worker unchanged\ncronjob.batch/render-verify unchanged\ndeployment.apps/web-gateway unchanged\nservicemonitor.monitoring.coreos.com/backend-service-monitor unchanged\nsealedsecret.bitnami.com/another-project1-secrets unchanged\nsealedsecret.bitnami.com/email-subscription-api-key unchanged\nservicemonitor.monitoring.coreos.com/gateway-monitor unchanged\nsealedsecret.bitnami.com/gateway-secrets unchanged\nsealedsecret.bitnami.com/leanplum-client-key unchanged\nsealedsecret.bitnami.com/-proxy unchanged"
ts=2021-08-01T22:03:16.508071617Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" ref=redis:windowsservercore
ts=2021-08-01T22:03:26.50518021Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" ref=redis:windowsservercore
ts=2021-08-01T22:03:29.404604582Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/tttttttttt/hhhhhhhhh.git branch=master HEAD=09721420fb012ecb6107df5c6473b29346e95785
ts=2021-08-01T22:03:36.515691755Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" ref=redis:windowsservercore
ts=2021-08-01T22:03:46.515736976Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/library/redis auth={map[]} err="toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" ref=redis:windowsservercore
ts=2021-08-01T22:03:46.515785089Z caller=warming.go:206 component=warmer updated=redis successful=0 attempted=496
ts=2021-08-01T22:03:46.516009846Z caller=images.go:17 component=sync-loop msg="polling for new images for automated workloads"
ts=2021-08-01T22:03:46.94904276Z caller=warming.go:198 component=warmer info="refreshing image" image=xxxxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/another-ns/render-service tag_count=262 to_update=2 of_which_refresh=2 of_which_missing=0
ts=2021-08-01T22:03:47.033355017Z caller=images.go:106 component=sync-loop workload=some-ns-develop:cronjob/another-project1-parse-configuration-jsons container=parse-configuration-jsons repo=xxxxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/another-ns/another-project1 pattern=glob:master-* current=xxxxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/another-ns/another-project1 warning="image with zero created timestamp" current="xxxxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/another-ns/another-project1 (0001-01-01 00:00:00 +0000 UTC)" latest="xxxxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/another-ns/another-project1:master-1c8ab7a (2021-07-27 19:14:26.881091109 +0000 UTC)" action="skip container"
ts=2021-08-01T22:03:47.03384068Z caller=images.go:106 component=sync-loop workload=some-ns-develop:deployment/another-project1 container=cms repo=xxxxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/another-ns/another-project1 pattern=glob:master-* current=xxxxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/another-ns/another-project1 warning="image with zero created timestamp" current="xxxxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/another-ns/another-project1 (0001-01-01 00:00:00 +0000 UTC)" latest="xxxxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/another-ns/another-project1:master-1c8ab7a (2021-07-27 19:14:26.881091109 +0000 UTC)" action="skip container"
ts=2021-08-01T22:03:47.03390967Z caller=images.go:106 component=sync-loop workload=some-ns-develop:deployment/another-project1 container=db-migration repo=xxxxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/another-ns/another-project1 pattern=glob:master-* current=xxxxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/another-ns/another-project1 warning="image with zero created timestamp" current="xxxxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/another-ns/another-project1 (0001-01-01 00:00:00 +0000 UTC)" latest="xxxxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/another-ns/another-project1:master-1c8ab7a (2021-07-27 19:14:26.881091109 +0000 UTC)" action="skip container"
ts=2021-08-01T22:03:47.035728823Z caller=images.go:106 component=sync-loop workload=some-ns-develop:cronjob/another-project1-deploy-to-preview container=deploy-to-preview repo=xxxxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/another-ns/another-project1 pattern=glob:master-* current=xxxxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/another-ns/another-project1 warning="image with zero created timestamp" current="xxxxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/another-ns/another-project1 (0001-01-01 00:00:00 +0000 UTC)" latest="xxxxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/another-ns/another-project1:master-1c8ab7a (2021-07-27 19:14:26.881091109 +0000 UTC)" action="skip container"
ts=2021-08-01T22:03:47.108641658Z caller=warming.go:206 component=warmer updated=xxxxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/another-ns/render-service successful=2 attempted=2
ts=2021-08-01T22:03:47.563081282Z caller=warming.go:198 component=warmer info="refreshing image" image=xxxxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/another-ns/render-service-worker tag_count=262 to_update=4 of_which_refresh=4 of_which_missing=0
ts=2021-08-01T22:03:47.823467449Z caller=warming.go:206 component=warmer updated=xxxxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/another-ns/render-service-worker successful=4 attempted=4
ts=2021-08-01T22:04:14.885501882Z caller=sync.go:60 component=daemon info="trying to sync git changes to the cluster" old=09721420fb012ecb6107df5c6473b29346e95785 new=09721420fb012ecb6107df5c6473b29346e95785
ts=2021-08-01T22:04:16.118361619Z caller=sync.go:86 method=Sync info="not applying resource; ignore annotation in cluster resource" resource=some-ns-develop:deployment/another-project1
ts=2021-08-01T22:04:16.120240225Z caller=sync.go:540 method=Sync cmd=apply args= count=65
ts=2021-08-01T22:04:17.167497038Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" took=1.047170174s err=null output="service/accounts-service unchanged\nrole.rbac.authorization.k8s.io/allow-port-forward unchanged\nrole.rbac.authorization.k8s.io/allow-run-jobs unchanged\nservice/assets-events-metrics unchanged\nservice/assets-service unchanged\nserviceaccount/another-project1 unchanged\nservice/another-project1 unchanged\nservice/test-website-dev unchanged\nserviceaccount/r-service unchanged\nservice/r-service unchanged\nrole.rbac.authorization.k8s.io/deploy-render-pr unchanged\nservice/exports-service unchanged\nservice/zz-service unchanged\nservice/proxy unchanged\nservice/projects-service unchanged\nservice/redis unchanged\nservice/render-service unchanged\nservice/render-worker-metrics unchanged\nservice/web-gateway unchanged\nconfigmap/accounts-env-9cfkt8td7c unchanged\nconfigmap/asset-source-map unchanged\nconfigmap/assets-env-b5kkfgt4fd unchanged\ningress.networking.k8s.io/another-project1 configured\nconfigmap/another-project1-env-tdgtg2d897 unchanged\ningress.networking.k8s.io/test-ingress-dev configured\nconfigmap/r-env-997c7mc965 unchanged\nconfigmap/exports-env-gkb5bkh8ck unchanged\ningress.networking.k8s.io/zz-proxy configured\ningress.networking.k8s.io/zz-with- configured\nconfigmap/gateway-env-8k2b4fgbdk unchanged\ningress.networking.k8s.io/gateway-ingress configured\nconfigmap/projects-env-9mtm6552mh unchanged\nconfigmap/render-pipeline-env-d45g8gfgd9 unchanged\nconfigmap/service-env-fg6hdb454g unchanged\nrolebinding.rbac.authorization.k8s.io/another-ns-build-dev-user-allow-deploy configured\nrolebinding.rbac.authorization.k8s.io/another-ns-build-dev-user-allow-port-forward configured\nrolebinding.rbac.authorization.k8s.io/another-ns-build-dev-user-allow-run-jobs configured\ndeployment.apps/accounts-service unchanged\ncronjob.batch/accounts-verify unchanged\ndeployment.apps/assets-events unchanged\ndeployment.apps/assets-service unchanged\ncronjob.batch/assets-verify unchanged\ncronjob.batch/another-project1-deploy-to-preview unchanged\ncronjob.batch/another-project1-parse-configuration-jsons unchanged\ndeployment.apps/test-website-dev unchanged\ndeployment.apps/r-service unchanged\ncronjob.batch/r-verify unchanged\ndeployment.apps/exports-service unchanged\ncronjob.batch/exports-verify unchanged\ndeployment.apps/zz-service unchanged\ndeployment.apps/proxy unchanged\ndeployment.apps/projects-service unchanged\ncronjob.batch/projects-verify unchanged\ndeployment.apps/redis unchanged\ndeployment.apps/render-service unchanged\ndeployment.apps/render-service-worker unchanged\ncronjob.batch/render-verify unchanged\ndeployment.apps/web-gateway unchanged\nservicemonitor.monitoring.coreos.com/backend-service-monitor unchanged\nsealedsecret.bitnami.com/another-project1-secrets unchanged\nsealedsecret.bitnami.com/email-subscription-api-key unchanged\nservicemonitor.monitoring.coreos.com/gateway-monitor unchanged\nsealedsecret.bitnami.com/gateway-secrets unchanged\nsealedsecret.bitnami.com/leanplum-client-key unchanged\nsealedsecret.bitnami.com/-proxy unchanged"
ts=2021-08-01T22:04:26.684265027Z caller=warming.go:198 component=warmer info="refreshing image" image=xxxxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/another-ns/assets-test tag_count=40 to_update=3 of_which_refresh=3 of_which_missing=0
ts=2021-08-01T22:04:26.880048973Z caller=warming.go:206 component=warmer updated=xxxxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/another-ns/assets-test successful=3 attempted=3
ts=2021-08-01T22:04:29.680815209Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://git@github.com/tttttttttt/hhhhhhhhh.git branch=master HEAD=09721420fb012ecb6107df5c6473b29346e95785
ts=2021-08-01T22:05:06.598978513Z caller=warming.go:198 component=warmer info="refreshing image" image=justinbarrick/fluxcloud tag_count=109 to_update=109 of_which_refresh=0 of_which_missing=109
ts=2021-08-01T22:05:06.603256352Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/2018.5.1: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.603486881Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/2018.5.2: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.607162596Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-0245b5b8: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.607233369Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-656d868: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.607278919Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-21eb8c1: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.607554912Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-5b19970: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.607716899Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/dev-images-f698864: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.611706617Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-265bd9d: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.613887828Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-913b886: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.613955724Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/2ea327c84c65b38a27c80c66f077d88173b0d4d6474d65207585f5760e07fee7: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.614468381Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/24ff3c644aae673b8722e833ba87938ac47b0fac2680ecb83bfdb64bfe0900b0: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.614584514Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-663b6e5: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.614930558Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/4f17668020da3eef3fc0cbdcc31b162ba72ddd5fbb9842cf715dcaf351f400c8: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.615051259Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-5bf8cb3: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.614941063Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-6e62626d: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.615742258Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-89f5fec: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.616100287Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-302fddd: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.617193551Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/84fa6f6a6c42bc464849167dbfa006ee9010763b90bdb06353ceafd413cb61f4: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.617270493Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-039ea8a: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.617342228Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/9a80cea35721b8423a6e91df5acf5dbfa0db7c871749616e2d346dbe98fc1860: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.617685841Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-6d0ea46: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.617953351Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-12fe7ea: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.618174832Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/ci-453ec631: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.618869926Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/latest: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.618938901Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-e40e9df: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.619449434Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-2bc7cd4: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.620019252Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/49073066b5889a63292bc1984750eaca319558426fbe14ad1f2bf80526191009: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.620952343Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-f9ad303: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.621141808Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-06976b8: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.621637741Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/473acd3c677d5becd54eab55200430638c812d84b76ffb5f48b4eceb5afc53c5: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.621767488Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-a5ab296: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.621805702Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/flux-1.8.1-0a8dfd2: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.621833366Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-00b331c: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.621854223Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/set-timeouts-19048d3a: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.621909764Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-2cf057b: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.621924695Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/pr-7085a78: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.622085993Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/bff0048e80a71d6285060975e60b516873eb01566c945791a60db63d99680516: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.62271378Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/f18bc10d15a3523f6ecdac80f4f7e8c5a65856e2de9d18a96d09ff6299503a55: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.623228673Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-1dbbce9: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.623523006Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-f36d2a9: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.623872315Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/ci-8fcc9743: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.623871177Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/webhook-tests-42db71d: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.624109279Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/ci-9f6ea627: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.624362881Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/latchmihay-multi_exporters-7328a6dc: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.624649433Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/2ec27b57b547ee9b99c6fc87bcdcf3fda6721c01f167c0a803f1474a3fabc67f: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.624855082Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-05016ff: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.625077531Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-865687e: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.625263579Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-2f48c0c8: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.626014375Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-ea5f24f: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.626328308Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-3825324: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.626348949Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-4e11ba7: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.626607454Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-14799ba: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.626879079Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-72b4524: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.627091492Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/flux-1.8.1-93cfa5f: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.627188126Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-f20c456: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.628304175Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/be54e3d9fe9bee63076342c54fb3751e82732bc65f6f8e0e20fa33566d3dd6c2: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.628366141Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-1c1c129: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.628912679Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-008a6bf: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.629578563Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/set-timeouts-e69a5333: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.630075089Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-b0312e82: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.630141809Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/v0.3.0: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.630223572Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-d8881c8: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.630481558Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-d988719: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.630514475Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/v0.2.9: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.630683238Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-b560c2d: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.632463789Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/v0.3.5: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.632549699Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/v0.3.9-c2695df5: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.63282427Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/test-2: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.632906632Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/v0.2.2: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.633137321Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/v0.3.8: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.633287741Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/simba-sleep-add-formatter-templates-23f21d2: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.633656131Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/test-1: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.633773446Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/test-3: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.633979923Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/v0.3.6: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.634156887Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/v0.3.8-6e62626d: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.634624959Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/v0.3.9: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.635822562Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/configurable-port-878d362: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.636088341Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-fabe02c: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.636240616Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/raise-sync-errors-fbab6e4: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.637323253Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-f9f9f7a: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.63758637Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-a6757bd6: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.637937396Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-cbffdcc: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.638066039Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/v0.3.4: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.638308629Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-a8f98611: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.639511776Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-fb850fd: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.639631243Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/v0.2.11: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.640050532Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/v0.2.7: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.640756443Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/matrix-6237f11: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.642266476Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-17d5c1d: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.642475255Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-a6c7844: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.642461685Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-c2695df5: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.642755562Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/v0.2.1: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.642884634Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-afdccaa: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.642954016Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-eb7ae87: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.643134336Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/pr-0d00a43: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.643370774Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-27d710f: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.644975003Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-b3a5d0a: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.645073695Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/master-29e0013: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.644421544Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/pr-92ff541: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.645963577Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/v0.2.8: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.64603285Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/pr-f12b358: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.646046212Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/pr-7a58a68: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d
ts=2021-08-01T22:05:06.646508716Z caller=repocachemanager.go:226 component=warmer canonical_name=index.docker.io/justinbarrick/fluxcloud auth={map[]} err="Get https://index.docker.io/v2/justinbarrick/fluxcloud/manifests/v0.3.7: rate limited: rate: Wait(n=1) would exceed context deadline" ref=justinbarrick/fluxcloud:webhook-tests-42db71d

5) Yes, this a new flux instance I installed in the last few days and it didn't work from the beginning. I opened the ticket after I tried everything. The reason I installed v1 is that the team has needed a new namespace quickly for a new project with tight schedule. So we decided not to spend the time now and install the same version which runs on all the the other namespaces on the cluster. But very soon everything will be upgraded to v2. 6) I tried your suggestion with registry-rps=200 and registry-burst=125 and it didn't work. I also don't see anything new in the log .

kingdonb commented 3 years ago

Hey, First of all WOW huge thanks for the quick and detailed answer.

😄 This means a lot. But I don't want you to get the wrong idea that I spent all this time on research just for you... all of this information was mostly fresh in my head, and I've been getting the details together for docs updates, as well as a release I have planned that is forthcoming.

It's very helpful to have someone communicating the details of their issue. Especially in the middle of the release, when you might be affected by the problem we're hoping to solve. (I am not so sure, I think this is just both types of rate limiting.)

What is strange in theory about the problem with ECR is that only one image of my workloads is coming from ECR, The three other are not. And I still don't get data for all of them.

The limiting factor is based on the numbers of layers and tags, but also Flux is trying to avoid downgrading anything that has automation. Flux v1 will not apply any changes until it is sure that images which you have automated are going to be applied at the appropriate latest version. For example you might have upgraded them yourself, and then restarted flux before the scan completed successfully. It would be a bad outcome if Flux took what was in the git repo (the old image) and overwrote what was on your cluster, under those conditions. Flux regards the whole commit as one thing to apply, so ongoing scanning related to an unknown "latest" tag in one image will cause Flux to wait to apply the whole commit.

Flux (seemingly, based on my reading of the code at least) knows this is a possibility, at least in v1, so it balks and refuses to apply any changes before it knows 100% which images from the filtered list (or unfiltered list, if you haven't applied filters) are the latest tags.

You can reduce the number of tags to scan, by deleting tags, or by applying a more limiting filter with the regex or semver filter annotations. Flux will only pull metadata for images that match the automation filter, tags that could be a candidate.

Yes, this a new flux instance

If it is a new flux instance, then it will be starting the scanning over from scratch. If the clusters have the same public address, if they are behind the same NAT gateway for example, then yes they will potentially also be affecting each other and rate limited together, as to outside registries they will appear to be the same client. It is surprising you don't see the docker hub rate limits on the new cluster, but if the number of tags in ECR is very high, you might also simply not have got there yet.

The problem condition you're working through will start over from scratch, if and when memcached is restarted. The problem is worse if there are more tags that match the regex filter. If you can reduce the number of tags in any registries, by filtering or by deleting tags, this will (I think?) help mitigate the problem, without changing anything else.

But very soon everything will be upgraded to v2.

That's great news! You will not see these delays anymore. The whole story I just told applies only to Flux v1. It is a very common report for us.

I tried your suggestion with registry-rps=200 and registry-burst=125

Those are the default values I found in your original post. You would have to increase the limits to see any difference. As long as you don't restart memcached, the flux instance will continue making progress. In Flux v2, memcached is no longer involved and this precarious situation is a much better story for the Flux user experience.

Thanks for your very helpful response!

acarmel commented 3 years ago

Ok thank you! your help is much appreciated. I think that I will just upgrade to v2 as you recommend and not spend more time to cause v1 to work.

pierluigilenoci commented 3 years ago

@kingdonb is there any possibility of having a helm chart for installing v2? It is the only thing that stops us from migrating.

kingdonb commented 3 years ago

@pierluigilenoci I see you found:

https://github.com/fluxcd/flux2/discussions/431

There is also:

https://github.com/fluxcd/flux2/issues/1641

This request comes up a lot, but it is not likely as the Helm cli does not handle installing and upgrading CRDs well.

There are a number of supported ways to install Flux other than flux bootstrap, but this limitation of Helm is something we cannot easily work around. The bootstrap method is the primary/recommended way to install Flux v2.

Please follow up on fluxcd/flux2#1641 @pierluigilenoci if this is a blocker for you.

pierluigilenoci commented 3 years ago

We released a very basic pre-alpha version here: https://github.com/fluxcd-community/helm-charts For all feel to open issues & create PRs

fluxcd / flux

Flux doesn't pull images data #3523

3492