Docker tag specified in yaml operator does not exist in dockerhub

[yogeek]

Hello

The image set in https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/operator.yaml is v1.21.0 does not seem to be in dockerhub :

docker pull jaegertracing/jaeger-operator:1.21.0               
Error response from daemon: manifest for jaegertracing/jaeger-operator:1.21.0 not found: manifest unknown: manifest unknown

docker pull jaegertracing/jaeger-operator:1.20.0               
ok

a problem with the CI that did not push the image for the last release maybe ?

[objectiser]

Yes the image push failed, so @rubenvp8510 will be releasing a new version shortly.

[objectiser]

Version 1.21.1 is now available.

[yogeek]

Thank you !

[yogeek]

@objectiser I have the same problem now with jaegertracing/jaeger-agent:1.21.1

[objectiser]

@yogeek Only the operator has been bumped to 1.21.1 - the operands (i.e. jaeger-agent) are still on 1.20.0.

The operator and operand versions are aligned for major and minor, but patch versions are independent.

[yogeek]

@objectiser ok but why my daemonset try to pull this image then...?

kubectl get ds jaeger-agent-daemonset -o yaml | grep -i "image:"
        image: jaegertracing/jaeger-agent:1.21.1

with this CR :

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: jaeger
spec:
  strategy: allInOne
  allInOne:
    image: jaegertracing/all-in-one:latest

[objectiser]

@yogeek This CR shouldn't have deployed a daemonset? That should only happen if the agent/strategy is explicitly set to DaemonSet: https://www.jaegertracing.io/docs/1.21/operator/#installing-the-agent-as-daemonset

[sylvainOL]

hello @objectiser, I've the same issue with a "production" deployment:

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: istio-metrics
  namespace: observability
spec:
  strategy: production
  collector:
    maxReplicas: 5
    resources:
      limits:
        cpu: 100m
        memory: 128Mi
  storage:
    type: elasticsearch
    options:
      es:
        server-urls: http://elasticsearch-coordinating-only:9200

Events:
  Type     Reason     Age                 From               Message
  ----     ------     ----                ----               -------
  Normal   Scheduled  107s                default-scheduler  Successfully assigned observability/istio-metrics-collector-68ffc69c6f-nklmh to compute04-k8s-internal
  Normal   Pulling    20s (x4 over 106s)  kubelet            Pulling image "jaegertracing/jaeger-collector:1.21.1"
  Warning  Failed     19s (x4 over 104s)  kubelet            Failed to pull image "jaegertracing/jaeger-collector:1.21.1": rpc error: code = Unknown desc = Error response from daemon: manifest for jaegertracing/jaeger-collector:1.21.1 not found: manifest unknown: manifest unknown
  Warning  Failed     19s (x4 over 104s)  kubelet            Error: ErrImagePull
  Normal   BackOff    8s (x6 over 104s)   kubelet            Back-off pulling image "jaegertracing/jaeger-collector:1.21.1"
  Warning  Failed     8s (x6 over 104s)   kubelet            Error: ImagePullBackOff

[yogeek]

@objectiser sorry I have not included all my CR, here it is :

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: jaeger
spec:
  strategy: allInOne
  allInOne:
    image: jaegertracing/all-in-one:latest
    # options:
    #   log-level: debug
  storage:
    type: memory
    # options:
    #   memory:
    #     max-traces: 100000
  ingress:
    enabled: false
  agent:
    strategy: DaemonSet

What about the daemonset image in this case ? it must be present in dockerhub isn't it ?

[objectiser]

@yogeek @sylvainOL Apologies, looks like a change in the release scripts has inadvertently bumped the operand version aswell - we will investigate and fix asap.

In the meantime, you can work around it by setting the images in the CR for each of the components.

[sylvainOL]

no problem :) I didn't managed to work around so I edited the deploy (it stills wants to pull 1.21.1 but as I have a running in 1.21.0, at least it works)

[yogeek]

@sylvainOL you can configure the image in the CustomResource :

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: jaeger
spec:
  strategy: allInOne
  allInOne:
    image: jaegertracing/all-in-one:latest
  storage:
    type: memory
  ingress:
    enabled: false
  agent:
    strategy: DaemonSet
    image: jaegertracing/jaeger-agent:1.21.0

@objectiser do we repopen the issue until this is fixed ?

[objectiser]

@yogeek Jaeger operator 1.21.2 has now been released, so should fix the problem. If there are further issues, then please reopen or create a new issue.

[yogeek]

@objectiser Sorry but I still do not see the tag 1.21.1 in docker hub... https://hub.docker.com/r/jaegertracing/jaeger-agent/tags

Do you mean we have to upgrade to operator 1.21.2 to fix our issue ?

[objectiser]

@yogeek That is correct, the Jaeger repo images (jaeger-agent, collector, query, etc) are still only on 1.21.0 - the issue was with the operator, so we have released the new version of the operator 1.21.2 which now correctly references the 1.21.0 operand images.

[yogeek]

@objectiser oh ok I thought the versions of the operator were aligned on the jaeger components versions. I will update then! Thank you for your help 👍

[sylvainOL]

@objectiser: upgrading to 1.21.2 solved the issue on my side! thanks!

[yogeek]

@objectiser same here ! 👍

Last question : is there a compatibility matrix between the operator versions and the deployment and agents versions ? Where can I find which docker images versions the operator is gonna deploy for these resources please ?

[objectiser]

@yogeek The version.txt file in the root folder provides those details: https://github.com/jaegertracing/jaeger-operator/blob/v1.21.2/versions.txt

So in this case the jaeger operaor 1.21.2 will use the jaeger operand (i.e. agent, collector ingester, etc) image versions 1.21.0.

[yogeek]

Thank you @objectiser 👍