jaegertracing / helm-charts

Helm Charts for Jaeger backend
Apache License 2.0
269 stars 347 forks source link

[Bug]: esRollover configured via Helm chart not working correctly #567

Closed christopher-wong closed 7 months ago

christopher-wong commented 7 months ago

What happened?

I've deployed Jaeger with ElasticSearch via the latest Helm chart 3.0.4.

I've enabled esRollover with the following configuration:

esRollover:
  enabled: true
  extraEnv:
    - name: CONDITIONS
      value: '{"max_age": "1h", "max_size": "25gb"}'
  schedule: "10 * * * *"

If I inspect the ElasticSearch indices, I don't see new indices being created, just the existing daily ones.

➜  ~ curl 'localhost:9200/_cat/indices?v'
health status index                      uuid                   pri rep docs.count docs.deleted store.size pri.store.size dataset.size
green  open   jaeger-span-000001         C2MafU9kRZKZWPbv6FFJqg   5   1          0            0      2.4kb          1.2kb        1.2kb
green  open   jaeger-span-000002         rBWJ233fSy6BoUvsaPQCwQ   5   1          0            0      2.4kb          1.2kb        1.2kb
green  open   jaeger-span-000003         tnbkkgTjSrWrjET3TSV9rg   5   1          0            0      2.4kb          1.2kb        1.2kb
green  open   jaeger-service-2024-04-30  VHsseM4ZS_qP56kT3X2jhw   5   1        233           38    443.5kb        215.5kb      215.5kb
green  open   jaeger-span-2024-04-30     R21JY1q8RsO9hA6Fl-VSiQ   5   1   23908208            0      9.4gb            5gb          5gb
green  open   jaeger-span-2024-04-28     Q13pRsK_Rxq3X5qRFpdPWw   5   1  408810059            0     43.7gb         21.8gb       21.8gb
green  open   jaeger-span-2024-04-29     d4yKrCeyTIG1_-qp2E3rvw   5   1  382922115            0    125.3gb         62.7gb       62.7gb
green  open   jaeger-service-2024-04-28  YvQSmwAaQy-2WPwvFhA5OA   5   1        309           15    332.5kb        173.8kb      173.8kb
green  open   jaeger-service-2024-04-29  0pY2u_Z9Tra26kkoN4Jh1w   5   1        486           78    481.9kb        234.7kb      234.7kb
green  open   jaeger-dependencies-000001 At-Lil2mTxqavSsiv3kxtw   5   1          0            0      2.4kb          1.2kb        1.2kb
green  open   jaeger-dependencies-000002 BJB0ivqvSleaQJ59cx47vg   5   1          0            0      2.4kb          1.2kb        1.2kb
green  open   jaeger-dependencies-000003 OcHXSXayS76yukmGQHwHLw   5   1          0            0      2.4kb          1.2kb        1.2kb
green  open   jaeger-service-000001      J7cKacQrTTKY_vbibhYjag   5   1          0            0      2.4kb          1.2kb        1.2kb
green  open   jaeger-service-000002      bRDUPKA9RfakL8iYBTeH8A   5   1          0            0      2.4kb          1.2kb        1.2kb
green  open   jaeger-service-000003      0Z-EMb5CQS-0k5ldboh5Pw   5   1          0            0      2.4kb          1.2kb        1.2kb

Steps to reproduce

  1. Deploy Jaeger Helm Chart 3.0.4 with ElasticSearch and esRollover enabled
  2. Check to see if new indices are being created.

Expected behavior

I expect a new index to get created every hour or when maxSize is 25gb

Relevant log output

No response

Screenshot

No response

Additional context

No response

Jaeger backend version

v1.53.0

SDK

No response

Pipeline

No response

Stogage backend

No response

Operating system

No response

Deployment model

No response

Deployment configs

tag: "1.53.0"

storage:
  type: elasticsearch
  elasticsearch:
    host: otjaeger2-elasticsearch-master-hl
    extraEnv:
    - name: ES_TAGS_AS_FIELDS_ALL
      value: "true"

provisionDataStore:
  cassandra: false
  elasticsearch: true
  kafka: false

elasticsearch:
  global:
    kibanaEnabled: true
  master:
    replicaCount: 3
    resourcesPreset: "xlarge" 
    persistence:
      enabled: true
      storageClass: "ebs-sc"
      accessModes:
      - ReadWriteOnce
      size: 1000Gi
  data:
    replicaCount: 3
    resourcesPreset: "xlarge" 
    persistence:
      enabled: true
      storageClass: "ebs-sc"
      accessModes:
      - ReadWriteOnce
      size: 1000Gi
  ingest:
    replicaCount: 2
    resourcesPreset: "medium" 
  coordinating:
    replicaCount: 2
    resourcesPreset: "medium" 
  kibana:
    ingress:
      enabled: true
      ingressClassName: alb
      hostname: '*'
      path: '/*'
      annotations:
        alb.ingress.kubernetes.io/target-type: 'ip'
        alb.ingress.kubernetes.io/scheme: internal
  # - name: ES_JAVA_OPTS
  #   value: -Xmx14g -Xms14g

esIndexCleaner:
  enabled: true # must be false for initial deploy
  schedule: "55 23 * * *"
  numberOfDays: 7

esRollover:
  enabled: true # must be false for initial deploy
  extraEnv:
    - name: CONDITIONS
      value: '{"max_age": "1h", "max_size": "25gb"}'
  schedule: "10 * * * *"

collector:
  image:
    registry:  <REDACTED>
    repository: jaeger-collector
  replicaCount: 6
  serviceMonitor:
    enabled: true
    additionalLabels:
      release: base
  ingress:
    enabled: true
    ingressClassName: alb
    annotations:
      alb.ingress.kubernetes.io/target-type: 'ip'
      alb.ingress.kubernetes.io/scheme: internal
      alb.ingress.kubernetes.io/certificate-arn:  <REDACTED>
      alb.ingress.kubernetes.io/healthcheck-port: '14269'
      alb.ingress.kubernetes.io/healthcheck-path: /
    pathType: Prefix
    hosts:
      - host:  <REDACTED>
        servicePort: http
      - host:  <REDACTED>
        servicePort: otlp-http
  service:
    otlp:
      grpc:
        name: otlp-grpc
      http:
        name: otlp-http
  resources:
    limits:
      cpu: 2
      memory: 1Gi
    requests:
      cpu: 250m
      memory: 128Mi

query:
  image:
    registry:  <REDACTED>
    repository: jaeger-query
  replicaCount: 6
  serviceMonitor:
    enabled: true
    additionalLabels:
      release: base
  service:
    type: LoadBalancer
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-type: "external"
      service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"
      service.beta.kubernetes.io/aws-load-balancer-ssl-cert: <REDACTED>
      service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
    port: 443
  resources:
    limits:
      cpu: 2
      memory: 4Gi
    requests:
      cpu: 250m
      memory: 128Mi
  agentSidecar:
    enabled: false

agent:
  enabled: false

spark:
  enabled: false