"image with zero created timestamp"

RickyHolland commented 2 years ago

Describe the bug

We are currently utlising the automated image update functionality of Flux and have had success with it in most use cases but we are having a problem with a particular deployment and its image tags.

For context, our container images in this paticular scenario are being tagged as "imageName: buildNo", so an example would be "imageName: 230" and, following code changes, a new image would be built and tagged as "imageName: 231". What we are finding though is that Flux is not able to automatically deploy new images when tagged as such. Although Flux detects that a new image is available it is failing to update the deployment with the following error.

ts=2022-02-15T10:47:59.063660596Z caller=images.go:106 component=sync-loop workload=prod:helmrelease/merchstack-vnext-prod container=backend repo=merchstack/merchstack-vnext pattern=regexp:[0-9]+ current=merchstack/merchstack-vnext warning="image with zero created timestamp" current="merchstack/merchstack-vnext (0001-01-01 00:00:00 +0000 UTC)" latest="merchstack/merchstack-vnext:240 (2022-02-14 20:19:27.888522263 +0000 UTC)" action="skip container"

We've done further research into the "image with zero created timestamp" error and have so far been unable to find anything of use. We have even tried ensuring a build date/time is tagged to the image via the following command, but still no luck.

We can validate that Flux is successfully detecting the image but so far has been unable to deploy it.

$ fluxctl --k8s-fwd-ns=flux list-images --namespace prod -w prod:helmrelease/merchstack-vnext-prod
WORKLOAD                                CONTAINER  IMAGE                     CREATED
prod:helmrelease/merchstack-vnext-prod  backend                              
                                                   |   240                   14 Feb 22 20:19 UTC
                                                   |   stage-91a94b0bd3      14 Feb 22 20:19 UTC
                                                   |   dev                   14 Feb 22 20:18 UTC
                                                   |   dev-fc8960dad6        14 Feb 22 20:18 UTC
                                                   |   237                   14 Feb 22 16:36 UTC
                                                   |   stage-bf03bc371b      14 Feb 22 16:36 UTC
                                                   |   dev-fbafaa9cfd        14 Feb 22 16:20 UTC
                                                   |   dev-ad64aef2b9        14 Feb 22 16:19 UTC
                                                   |   233                   11 Feb 22 16:08 UTC
                                                   |   latest                11 Feb 22 16:08 UTC
                                                   '-> (untagged)            ?
                                        ui         merchstack/merchstack-ui  
                                                   |   vnext                 02 Feb 22 21:17 UTC
                                                   '-> vnext-fd1f4037c0      02 Feb 22 21:17 UTC
                                                       dev                   02 Feb 22 20:17 UTC
                                                       dev-33c3d9e72c        02 Feb 22 20:17 UTC
                                                       vnext-994f7563a3      27 Jan 22 14:47 UTC
                                                       dev-dc7518668b        27 Jan 22 14:20 UTC
                                                       dev-7b828b34af        27 Jan 22 14:18 UTC
                                                       dev-6e8a4f0083        27 Jan 22 14:17 UTC
                                                       dev-43701e3169        27 Jan 22 14:16 UTC
                                                       dev-4e1da17f5c        27 Jan 22 14:16 UTC

We do have Flux's automated image update function working elsewhere, it just seems to be an issue with how we are tagging these particular images.

For context we are deploying our application via Helm using the Flux HelmOperator. We are using the following annotations to enable and control the automated image update function.

apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
metadata:
  name: merchstack-vnext-prod
  namespace: prod
  annotations:
    fluxcd.io/automated: "true"
    fluxcd.io/ignore: "false"
    filter.fluxcd.io/ui: "regex:^vnext-.{1,}"
    filter.fluxcd.io/backend: "regex:[0-9]+"

Steps to reproduce

Install Flux and the Flux HelmOperator. Create a release that uses the same image tag scheme as above.

Expected behavior

The new image should deploy automatically.

Kubernetes version / Distro / Cloud provider

AWS EKS - Kubernetes 1.21

Flux version

1.21

Git provider

GitHub

Container Registry provider

DockerHub

Additional context

No response

Maintenance Acknowledgement

[X] I am aware of Flux v1's maintenance status

Code of Conduct

[X] I agree to follow this project's Code of Conduct

kingdonb commented 2 years ago

How are you tagging the images? There are several related reports in this repo, under the same heading, see for more information:

2020
3417
2929

You are correct that the timestamp (or lack of timestamp) is the reason your automation is not progressing. Flux v1 hard-depends on those timestamps for ordering the images in your repository to determine which one is the latest, and if it finds any images matching the regex filter that do not have timestamps, then it cannot know if that one should be the latest. It balks and you get this error message.

The good news is Flux v2 no longer has this dependency on the build timestamps. If you are building images with a serial number tag, you are already following the guidance here: https://fluxcd.io/docs/guides/sortable-image-tags/ so an upgrade should be an easy quick win here (🤞)

I am honestly not sure how those images are produced without a timestamp, but if you have made using reproducible builds an intentional decision (or if anyone did) then it is likely on purpose, as builds that carry a timestamp are incompatible with reproducibility.

If there's any more information that I can provide, I'll be happy to assist you with finding migration docs to support your upgrade from Flux v1 to Flux v2.

RickyHolland commented 2 years ago

Thanks for the prompt reply @kingdonb.

We did some more troubleshooting on this issue today and seem to have resolved the issue by simply enclosing the tag in quotes.

      image:
        repository: merchstack/merchstack-vnext
        tag: '256'

Whereas previously it was unquoted.

      image:
        repository: merchstack/merchstack-vnext
        tag: 256

Following this change newly built images (for example with the tag 257) are automatically deployed by Flux and the manifest within the repo is also updated accordingly.

kingdonb commented 2 years ago

I have never seen that mode of failure before, but I'll be aware of it now. Thanks for reporting back and I think we can close this! 🎉

fluxcd / flux