Trace outputs are not reflected in Jaeger [Bug]:

[aytemuryakup]

What happened?

I'm doing the Jaeger operator deployment on kubernetes. In production mode, I stand up jaeger and deploy the hotrod application.

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: ob-jaeger
  namespace: observability
spec:
  strategy: production
  storage:
    type: elasticsearch
    options:
      es:
        server-urls: http://x.x.x.x:9200
        index-prefix: jaeger-collector
        version: 7
        create-index-templates: false
  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: nginx
    hosts:
     - tracing.local
    tls:
      - hosts:
         - tracing.local
        secretName: tls-secret
  agent:
    image: jaegertracing/jaeger-agent:1.39
    strategy: DaemonSet
    options:
      log-level: debug
  query:
    options:
      es:
        use-aliases: true
        tags-as-fields:
          all: true
    image: jaegertracing/jaeger-query:1.39
  collector:
    maxReplicas: 5
    resources:
      limits:
        cpu: 256m
        memory: 512Mi
    options:
      es:
        use-aliases: true
        tags-as-fields:
          all: true
    image: jaegertracing/jaeger-collector:1.39

Since there is no service for the agent in DaemonSet mode, I deploy its own service object as follows.

apiVersion: v1
kind: Service
metadata:
  labels:
    app: jaeger
    app.kubernetes.io/component: agent
    app.kubernetes.io/instance: ob-jaeger
    app.kubernetes.io/managed-by: jaeger-operator
    app.kubernetes.io/name: ob-jaeger-agent
    app.kubernetes.io/part-of: jaeger
  name: ob-jaeger-agent
  namespace: observability
spec:
  ports:
  - name: zk-compact-trft
    port: 5775
    protocol: UDP
    targetPort: 5775 

  - name: http-config-rest
    port: 5778
    protocol: TCP
    targetPort: 5778
  - name: jg-compact-trft
    port: 6831
    protocol: UDP
    targetPort: 6831
  - name: jg-binary-trft
    port: 6832
    protocol: UDP
    targetPort: 6832
  selector:
    app: jaeger
    app.kubernetes.io/component: agent
    app.kubernetes.io/instance: ob-jaeger
    app.kubernetes.io/managed-by: jaeger-operator
    app.kubernetes.io/name: ob-jaeger-agent
    app.kubernetes.io/part-of: jaeger
  sessionAffinity: None
  type: ClusterIP

When I start a trace through the Hotrod application, I see that the processes are coming to the jaeger-agent pods. But I can't see the services via query(jaeger-ui).

Steps to reproduce

1. Jaeger operator deploy.
2. Jaeger production deploy.
3. Jaeger agent k8s service deploy.
4. Jaeger hotrod demo app deploy.
5. Test

Expected behavior

Services should appear on the Jaeger UI screen and trace logs should be dropped.

Relevant log output

Agent DaemonSets trace log

{"level":"debug","ts":1668518240.153933,"caller":"processors/thrift_processor.go:120","msg":"Span(s) received by the agent","bytes-received":14735}
{"level":"debug","ts":1668518240.7519624,"caller":"processors/thrift_processor.go:120","msg":"Span(s) received by the agent","bytes-received":5749}
{"level":"debug","ts":1668518240.8075762,"caller":"processors/thrift_processor.go:120","msg":"Span(s) received by the agent","bytes-received":957}
{"level":"debug","ts":1668518241.7561247,"caller":"processors/thrift_processor.go:120","msg":"Span(s) received by the agent","bytes-received":17342}
{"level":"debug","ts":1668518241.807425,"caller":"processors/thrift_processor.go:120","msg":"Span(s) received by the agent","bytes-received":949}

Screenshot

Additional context

No response

Jaeger backend version

v1.39.0

SDK

No response

Pipeline

No response

Stogage backend

Elasticsearch

Operating system

Linux

Deployment model

Kubernetes

Deployment configs

No response

[v-sag]

I have the same issue. It seems the UI somehow is not able to read the data from jaeger backend. I have confirmed that the collector does store data in elastic search. Did you find a solution for this?

[aytemuryakup]

Hi @v-sag , We have determined that elasticsearch is problematic in 8. versions. We did our tests on versions above Elastic 7. and there were no problems.

[ChrisJBurns]

Having this problem also with the following

ElasticSearch Deployment via Operator

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch
  namespace: elastic-system
spec:
  version: 8.0.0
  nodeSets:
    - name: default
      config:
        node.roles:
          - master
          - data
        node.attr.attr_name: attr_value
        node.store.allow_mmap: false
      podTemplate:
        metadata:
          labels:
            foo: bar
        spec:
          containers:
            - name: elasticsearch
              resources:
                requests:
                  memory: 4Gi
                  cpu: 1
                limits:
                  memory: 4Gi
                  cpu: 2
      count: 3

Jaeger Deployment via Jaeger Operator

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: jaeger
  namespace: elastic-system
spec:
  annotations:
    sidecar.istio.io/inject: "true"
  strategy: production
  collector:
    options:
      es:
        use-aliases: true
        use-ilm: true
  query:
    options:
      es:
        use-aliases: true
        use-ilm: true
  storage:
    type: elasticsearch
    options:
      es:
        version: 7
        create-index-templates: false
        server-urls: https://elasticsearch-es-http:9200
        index-prefix: jaeger_
        tls: # <3>
          ca: /es/certificates/ca.crt
    secretName: jaeger-secret # <4>
  volumeMounts: # <5>
    - name: certificates
      mountPath: /es/certificates/
      readOnly: true
  volumes:
    - name: certificates
      secret:
        secretName: elasticsearch-es-http-certs-public

According to the collector, it is storing traces and spans in the storage backend (ElasticSearch), but the UI just isn't showing any of it.

[ChrisJBurns]

It's worth mentioning, I still can't get this to work on ElasticSearch 7.17.9. Same problem. No traces in the UI at all.

[v-sag]

Hi,

I was having the same issue and I solved it by not using the aliases and applying this workaround.

I found the issue by looking at the logs; I could see that Jaeger logs were showing errors that the aliases they were trying against Elastic Search were not present. Then I searched on documentation and found out that when you enable aliases on Jaeger, you have to manage the aliases separately/externally, so Jaeger will not try to create them for you. Let me know if you need my full yaml set-up.

[ChrisJBurns]

@v-sag Go for it, your YAML will help. Did you do the requests to the API manually? We'd like to avoid this if we can :(

[v-sag]

Yes, that workaround I mentioned above, has to be applied on the Elastic-Search. I did that through Kibana UI.

Here is my Jaeger

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: jaeger-main
  namespace: jaeger
spec:
  strategy: production
  collector:
    maxReplicas: 10
    resources:
      limits:
        cpu: 500m
        memory: 1024Mi
    options:
      es:
        log-level: debug
  query:
    options:
      es:
        log-level: debug
  storage:
    type: elasticsearch
    esIndexCleaner:
      enabled: true                                 # turn the cron job deployment on and off
      numberOfDays: 2                               # number of days to wait before deleting a record
      schedule: "55 23 * * 1-5"                       # cron expression for it to run
    options:
      es:
        version: 7
        create-index-templates: false
        server-urls: https://basic-es-http.elastic-system.svc:9200
        tls:
          ca: /es/certificates/ca.crt
    # The user has to be created first in elastic-search.
    # es credentials using: kubectl create secret generic jaeger-secret --from-literal=ES_PASSWORD=** --from-literal=ES_USERNAME=jaeger
    secretName: jaeger-secret
  volumeMounts:
    - name: certificates
      mountPath: /es/certificates/
      readOnly: true
  volumes:
    - name: certificates
      secret:
        # This secret is created by elastic-search
        secretName: basic-es-http-certs-public

[ChrisJBurns]

Ahh, we don't use Kibana for our Elasticsearch as we're just deploying Elasticsearch as part of the storage backend, not Kibana.