search

fluxcd / flux

Successor: https://github.com/fluxcd/flux2
https://fluxcd.io
Apache License 2.0
6.9k stars 1.08k forks source link

fluxctl list-images show up with timestamp of 01 Jan 70 00:00 UTC #1797

Closed christophercorn closed 5 years ago

christophercorn commented 5 years ago

While trying view images, all images end up with an impossible timestamp. I am using the flux docker image quay.io/weaveworks/flux:1.10.1 .

fluxctl list-images
apollo:deployment/app                                            app                           XXXXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app
                                                                                                 |   00bcaff1f                                                   01 Jan 70 00:00 UTC
                                                                                                 |   01db306c4                                                   01 Jan 70 00:00 UTC
                                                                                                 |   085bd202b                                                   01 Jan 70 00:00 UTC
                                                                                                 |   0a1a8bed1                                                   01 Jan 70 00:00 UTC
                                                                                                 |   13c3e828f                                                   01 Jan 70 00:00 UTC
                                                                                                 |   28375c7ae                                                   01 Jan 70 00:00 UTC
                                                                                                 |   2dcbe7693                                                   01 Jan 70 00:00 UTC
                                                                                                 |   2e8d80563                                                   01 Jan 70 00:00 UTC
                                                                                                 |   3f9610722                                                   01 Jan 70 00:00 UTC
                                                                                                 |   4434a8d7d                                                   01 Jan 70 00:00 UTC
                                                                                                 : (33 image(s) omitted)
                                                                                                 '-> d35ef3d96                                                   01 Jan 70 00:00 UTC
christophercorn commented 5 years ago

Here is the flux logs with registry trace logs enabled.

ts=2019-03-06T00:19:54.171961054Z caller=main.go:156 version=1.10.1
ts=2019-03-06T00:19:54.184013676Z caller=main.go:247 component=cluster identity=/etc/fluxd/ssh/identity
ts=2019-03-06T00:19:54.1840818Z caller=main.go:249 component=cluster host=https://172.20.0.1:443 version=kubernetes-v1.11.5-eks-6bad6d
ts=2019-03-06T00:19:54.184120067Z caller=main.go:261 component=cluster kubectl=/usr/local/bin/kubectl
ts=2019-03-06T00:19:54.1851302Z caller=main.go:269 component=cluster ping=true
ts=2019-03-06T00:19:54.18649656Z caller=aws.go:74 component=aws info="detected cluster region" region=us-west-2
ts=2019-03-06T00:19:54.186526551Z caller=aws.go:78 component=aws info="restricting ECR registry scans" regions=us-west-2 include-ids= exclude-ids=602401143452
ts=2019-03-06T00:19:54.187558446Z caller=main.go:417 url=git@github.com:XXXXXXXXXX/cluster-flux-configuration.git user="Weave Flux" email=support@weave.works sync-tag=flux-sync notes-ref=flux set-author=false
ts=2019-03-06T00:19:54.187589542Z caller=main.go:472 upstream="no upstream URL given"
ts=2019-03-06T00:19:54.188483922Z caller=main.go:493 addr=:3030
ts=2019-03-06T00:19:54.187956201Z caller=loop.go:92 component=sync-loop err="git repo not ready: git repo has not been cloned yet"
ts=2019-03-06T00:19:54.188568882Z caller=images.go:17 component=sync-loop msg="polling images"
ts=2019-03-06T00:19:54.188585023Z caller=images.go:27 component=sync-loop msg="no automated services"
ts=2019-03-06T00:19:54.420117487Z caller=aws.go:137 component=aws info="attempting to refresh auth tokens" region=us-west-2 account-ids=XXXXXXXXX
ts=2019-03-06T00:19:54.60245906Z caller=checkpoint.go:24 component=checkpoint msg="up to date" latest=1.9.0
ts=2019-03-06T00:19:56.114253186Z caller=client_factory.go:41 component=registry url=https://XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/v2/ status="401 Unauthorized"
ts=2019-03-06T00:19:56.114336672Z caller=client_factory.go:133 component=registry repo=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>" api=https://XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/v2/
ts=2019-03-06T00:19:56.195094547Z caller=client_factory.go:41 component=registry url=https://XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/v2/app/tags/list status="200 OK"
ts=2019-03-06T00:19:56.195575533Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:89a38859d last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.195770699Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:b22c8b947 last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.195934184Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:2dcbe7693 last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.196087491Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:5e9647061 last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.196227994Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:6cc707682 last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.196387693Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:d6e1a14b2 last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.196494323Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:77f0edfca last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.196614001Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:eacab9f24 last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.196733826Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:ef8e9e2c4 last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.196847883Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:ff16bed86 last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.196967135Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:80bee73e0 last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.197078715Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:5906e1900 last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.197196415Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:5a7157c7f last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.197318435Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:795fcfc3a last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.197433541Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:e7b3602d3 last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.197543965Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:c98448066 last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.197633008Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:085bd202b last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.197732411Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:ea233b812 last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.197826599Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:4434a8d7d last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.197969723Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:3f9610722 last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.198109875Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:500955cc4 last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.19822554Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:ccec25ff0 last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.198337019Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:56a2c49d3 last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.198448229Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:d9a79a0eb last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.198572834Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:48228bb65 last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.198698487Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:00bcaff1f last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.198812286Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:2e8d80563 last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.198947323Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:c1e2f0d46 last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.199094068Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:e40c2f3bb last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.199198836Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:e09393682 last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.19931632Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:f6d644de7 last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.199449564Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:fda3e3099 last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.199570701Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:28375c7ae last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.199679238Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:cec840c00 last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.199798458Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:cecad291d last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.199938194Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:ce09a361c last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.200074175Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:91e870fea last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.200189244Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:01db306c4 last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.200319597Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:932526b2a last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.20244371Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:e19592085 last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.20257598Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:0a1a8bed1 last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.202708167Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:d05fd69e8 last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.202845695Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:13c3e828f last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.202948907Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:aa249c72d last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.203072749Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:c51df5ec8 last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.203210292Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:55b829315 last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.203348575Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:d35ef3d96 last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.203502459Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:9b5e93d2e last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.203613391Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:fd8a89afb last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.203726853Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:4d6f76dd8 last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.203856091Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:52e9b0e9e last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.203958217Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:eb50f1ce2 last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.204056869Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:c534c091d last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.204165886Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:882d41188 last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.204297603Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:bbd376530 last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.20440828Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:9eeac15f6 last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.204509702Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:5418dea2c last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.204629756Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:edb8a1ff7 last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.204736373Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:92f4fc1ff last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
ts=2019-03-06T00:19:56.204850498Z caller=warming.go:237 component=warmer canonical_name=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app auth="{map[XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com:<registry creds for AWS@XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com, from AWS API>]}" trace="found cached manifest" ref=XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:1388e0875 last_fetched=2019-03-06T00:19:14Z deadline=2019-03-06T01:19:14Z
squaremo commented 5 years ago

Thanks for taking the time to log this issue, with extra detail @chriscorn-takt.

The timestamps are Unix time = 0, which makes me think the results are empty (structs are initialised with "zero" values in Go). But to rule out an alternative, would you confirm that the actual images have timestamps -- you can do this by inspecting one of them:

docker inspect XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:89a38859d

Assuming they are empty values, I wonder how they got there. I would expect that they can only be empty if the fetch resulted in an error -- and it should have recorded the fact of that error, and be reporting it back.

OK, so something to try: if you restart memcached, it will force fluxd to repopulate it with image metadata. Then we'll see more in the logs about the results it gets trying to fetch from the image registry.

christophercorn commented 5 years ago

Great catch @squaremo . The images do indeed boot however, and come back fine.

[
    {
        "Id": "sha256:5d90373eac03e1af4de1d9ffca2a026a9c70e4638d379121cf5cccaccae59d1e",
        "RepoTags": [
            "XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app:1388e0875"
        ],
        "RepoDigests": [
            "XXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/app@sha256:3bdfa9353949abe82db720a07fa9504a8f69502ca9c3ccbd177e28af361ab26f"
        ],
        "Parent": "",
        "Comment": "",
        "Created": "1970-01-01T00:00:01Z",
        "Container": "",
        "ContainerConfig": {
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": null,
            "Cmd": null,
            "Image": "",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": null
        },
        "DockerVersion": "",
        "Author": "",
        "Config": {
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "8080/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": null,
            "Cmd": [
                "/nix/store/0ly0k2bn1jwzparzrm4g459b6j1ymzvp-somatix-exe-app-api/bin/app-api",
                "--conf",
                "conf/app-api.conf",
                "+RTS",
                "-T",
                "-RTS"
            ],
            "Image": "",
            "Volumes": null,
            "WorkingDir": "/",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": null
        },
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 166204184,
        "VirtualSize": 166204184,
        "GraphDriver": {
            "Data": {
                "MergedDir": "/var/lib/docker/overlay2/58144d03b064697abd28496ca2f7fe53e7e0c72b5190e26b5c729b7bae30ed99/merged",
                "UpperDir": "/var/lib/docker/overlay2/58144d03b064697abd28496ca2f7fe53e7e0c72b5190e26b5c729b7bae30ed99/diff",
                "WorkDir": "/var/lib/docker/overlay2/58144d03b064697abd28496ca2f7fe53e7e0c72b5190e26b5c729b7bae30ed99/work"
            },
            "Name": "overlay2"
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:37518e82e45197422ec10cdf7c5467be7d799ff98947bd4e26dfe496147b9701"
            ]
        },
        "Metadata": {
            "LastTagTime": "0001-01-01T00:00:00Z"
        }
    }
]
squaremo commented 5 years ago

Oh wow, I did not expect the timestamps to genuinely be zero! How do you build them? Could it be that they are built in VMs that don't have the clock set, or something along those lines?

One thing we could do to help is let people override the build time with an image label (there are standard labels for that sort of thing).

christophercorn commented 5 years ago

It looks like we have an internal requirement for the docker image create time to be set to essentially 0. So, it is by design. Unfortunately, my docker image tags are simply a commit hash. so, image auto updating may not work for me. thanks for your assistance.