search

jaegertracing / helm-charts

Helm Charts for Jaeger backend
Apache License 2.0
269 stars 347 forks source link

[JAEGER] Fill the `cmdlineParams` property of the `esIndexCleaner` to enable its functionality. #524

Closed scalalang2 closed 7 months ago

scalalang2 commented 1 year ago

What this PR does

I have encountered difficulties in determining the root cause of the unexpected behavior exhibited by esIndexCleaner.

{"level":"info","ts":1701136500.6844676,"caller":"es-index-cleaner/main.go:89","msg":"Indices before this date will be deleted","date":"2023-11-24T00:00:00Z"}
{"level":"info","ts":1701136500.6845164,"caller":"es-index-cleaner/main.go:98","msg":"Queried indices","indices":null}
{"level":"info","ts":1701136500.6846325,"caller":"es-index-cleaner/main.go:102","msg":"No indices to delete"}

The reason why I spent several hours to fix this issue is that the template.spec.containers includes enviornment value ES_SERVER_URLS and ES_INDEX_PREFIX. I didn't expect that this env vars are not related with the processor at all.

To enhance clarity, I think we should put index-prefix options into the values.yml file.

Checklist

pavelnikolov commented 12 months ago

A fix would be welcome.

Stevenpc3 commented 7 months ago

We use esIndexCleaner regularly with no issue out of the box. Like this:

jaeger:
  esIndexCleaner:
    enabled: true
    schedule: "55 23 * * *"
    resources:
      requests:
        cpu: 256m
        memory: 128Mi
    numberOfDays: 5

Jaeger naturally appends index patterns to it's data like this

"jaeger-span-*"
"jaeger-service-*"

I am not using this with any other data in elastic.

I am curious what the issue is described in here that requires the use of index prefix options.

I assume it is because there are overrides being used and those can be passed in via the correct service. For example. To set the number of replicas to 0 (which we required) you can pass

collector:
  cmdlineParams:
    es.num-replicas: "0"

This list of cmdline params are generated from "generate-help-output.sh" https://github.com/jaegertracing/jaeger/blob/a72dfc39ddf43cb703b1393c99f4ca3fe6b6c388/scripts/generate-help-output.sh#L3 which appears similar to this which is for the collector which I have shortened considerably by removing items...

2024/01/17 21:15:07 maxprocs: Leaving GOMAXPROCS=8: CPU quota undefined
2024/01/17 21:15:07 applicatio version: git-commit=10e82c07eb2aa7f9fafc65b8f37ece1257d25b7a, git-version=v1.51.0, build-date=2023-11-03T00:16:16Z
Error: unknown flag: --0
Usage:
jaeger-collector [flags]
jaeger-collector [command]
Available Commands:
completion Generate the autocompletion script for the specified shell
docs Generates documentation
env Help about environment variables.
help Help about any command
status Print the status.
version Print the version.
Flags:
--admin.http.host-port string The host:port (e.g. 127.0.0.1:14269 or :14269) for the admin server, including health check, /metrics, etc. (default ":14269")
...
--collector.grpc.tls.min-version string Minimum TLS version supported (Possible values: 1.0, 1.1, 1.2, 1.3)
--collector.http-server.host-port string The host:port (e.g. 127.0.0.1:12345 or :12345) of the collector's HTTP server (default ":14268")
--collector.http-server.idle-timeout duration See https://pkg.go.dev/net/http#Server (default 0s)
--collector.http-server.read-header-timeout duration See https://pkg.go.dev/net/http#Server (default 2s)
--collector.http-server.read-timeout duration See https://pkg.go.dev/net/http#Server (default 0s)
--collector.http.tls.cert string Path to a TLS Certificate file, used to identify this server to clients
--collector.http.tls.cipher-suites string Comma-separated list of cipher suites for the server, values are from tls package constants (https://golang.org/pkg/crypto/tls/#pkg-constants).
--collector.http.tls.client-ca string Path to a TLS CA (Certification Authority) file used to verify certificates presented by clients (if unset, all clients are permitted)
--collector.http.tls.enabled Enable TLS on the server
--collector.http.tls.key string Path to a TLS Private Key file, used to identify this server to clients
--collector.http.tls.max-version string Maximum TLS version supported (Possible values: 1.0, 1.1, 1.2, 1.3)
--collector.http.tls.min-version string Minimum TLS version supported (Possible values: 1.0, 1.1, 1.2, 1.3)
--collector.num-workers int The number of workers pulling items from the queue (default 50)
--collector.otlp.enabled Enables OpenTelemetry OTLP receiver on dedicated HTTP and gRPC ports (default true)
--collector.otlp.grpc.host-port string The host:port (e.g. 127.0.0.1:12345 or :12345) of the collector's gRPC server
...
--es-archive.create-index-templates Create index templates at application startup. Set to false when templates are installed manually. (default true)
--es-archive.enabled Enable extra storage
--es-archive.index-date-separator string Optional date separator of Jaeger indices. For example "." creates "jaeger-span-2020.11.20". (default "-")
--es-archive.index-prefix string Optional prefix of Jaeger indices. For example "production" creates "production-jaeger-*".
--es-archive.index-rollover-frequency-services string Rotates jaeger-service indices over the given period. For example "day" creates "jaeger-service-yyyy-MM-dd" every day after UTC 12AM. Valid options: [hour, day]. This does not delete old indices. For details on complete index management solutions supported by Jaeger, refer to: https://www.jaegertracing.io/docs/deployment/#elasticsearch-rollover (default "day")
--es-archive.index-rollover-frequency-spans string Rotates jaeger-span indices over the given period. For example "day" creates "jaeger-span-yyyy-MM-dd" every day after UTC 12AM. Valid options: [hour, day]. This does not delete old indices. For details on complete index management solutions supported by Jaeger, refer to: https://www.jaegertracing.io/docs/deployment/#elasticsearch-rollover (default "day")
...
--es-archive.timeout duration Timeout used for queries. A Timeout of zero means no timeout (default 0s)
--es-archive.tls.ca string Path to a TLS CA (Certification Authority) file used to verify the remote server(s) (by default will use the system truststore)
--es-archive.tls.cert string Path to a TLS Certificate file, used to identify this process to the remote server(s)
--es-archive.tls.enabled Enable TLS when talking to the remote server(s)
...
--es.bulk.workers int The number of workers that are able to receive bulk requests and eventually commit them to Elasticsearch (default 1)
--es.create-index-templates Create index templates at application startup. Set to false when templates are installed manually. (default true)
--es.index-date-separator string Optional date separator of Jaeger indices. For example "." creates "jaeger-span-2020.11.20". (default "-")
--es.index-prefix string Optional prefix of Jaeger indices. For example "production" creates "production-jaeger-*".
--es.index-rollover-frequency-services string Rotates jaeger-service indices over the given period. For example "day" creates "jaeger-service-yyyy-MM-dd" every day after UTC 12AM. Valid options: [hour, day]. This does not delete old indices. For details on complete index management solutions supported by Jaeger, refer to: https://www.jaegertracing.io/docs/deployment/#elasticsearch-rollover (default "day")
--es.index-rollover-frequency-spans string Rotates jaeger-span indices over the given period. For example "day" creates "jaeger-span-yyyy-MM-dd" every day after UTC 12AM. Valid options: [hour, day]. This does not delete old indices. For details on complete index management solutions supported by Jaeger, refer to: https://www.jaegertracing.io/docs/deployment/#elasticsearch-rollover (default "day")
--es.log-level string The Elasticsearch client log-level. Valid levels: [debug, info, error] (default "error")
--es.max-doc-count int The maximum document count to return from an Elasticsearch query. This will also apply to aggregations. (default 10000)
--es.max-span-age duration The maximum lookback for spans in Elasticsearch (default 72h0m0s)
--es.num-replicas int The number of replicas per index in Elasticsearch (default 1)
--es.num-shards int The number of shards per index in Elasticsearch (default 5)
--es.password string The password required by Elasticsearch
--es.password-file string Path to a file containing password. This file is watched for changes.
--es.remote-read-clusters string Comma-separated list of Elasticsearch remote cluster names for cross-cluster querying.See Elasticsearch remote clusters and cross-cluster query api.
--es.send-get-body-as string HTTP verb for requests that contain a body [GET, POST].
--es.server-urls string The comma-separated list of Elasticsearch servers, must be full url i.e. http://localhost:9200 (default "http://127.0.0.1:9200")
...
--es.use-ilm (experimental) Option to enable ILM for jaeger span & service indices. Use this option with es.use-aliases. It requires an external component to create aliases before startup and then performing its management. ILM policy must be manually created in ES before startup. Supported only for elasticsearch version 7+.
--es.username string The username required by Elasticsearch. The basic authentication also loads CA if it is specified.
--es.version uint The major Elasticsearch version. If not specified, the value will be auto-detected from Elasticsearch.
-h, --help help for jaeger-collector
--log-level string Minimal allowed log Level. For more levels see https://github.com/uber-go/zap (default "info")
--metrics-backend string Defines which metrics backend to use for metrics reporting: prometheus, none, or expvar (deprecated, will be removed after 2024-01-01 or in release v1.53.0, whichever is later) (default "prometheus")
--metrics-http-route string Defines the route of HTTP endpoint for metrics backends that support scraping (default "/metrics")
--multi-tenancy.enabled Enable tenancy header when receiving or querying
--multi-tenancy.header string HTTP header carrying tenant (default "x-tenant")
--multi-tenancy.tenants string comma-separated list of allowed values for --multi-tenancy.header header. (If not supplied, tenants are not restricted)
--sampling.strategies-file string The path for the sampling strategies file in JSON format. See sampling documentation to see format of the file
--sampling.strategies-reload-interval duration Reload interval to check and reload sampling strategies file. Zero value means no reloading (default 0s)
--span-storage.type string (deprecated) please use SPAN_STORAGE_TYPE environment variable. Run this binary with the 'env' command for help.
Use "jaeger-collector [command] --help" for more information about a command.

This can be generated in the cmdline via the --help argument.

I assume the issue here is that esIndexCleaner requires the proper cmdline args and I am not sure which those are that map to https://github.com/jaegertracing/jaeger/blob/main/cmd/es-index-cleaner/main.go#L38 and https://github.com/jaegertracing/jaeger/blob/main/cmd/es-index-cleaner/app/flags.go#L47 which could very well be 1 of or neither of:

es-archive.index-prefix
es.index-prefix

which are the only two to appear in the collector help. So perhaps test the following?

collector:
  cmdlineParams:
    es.index-prefix: "test"
esIndexCleaner:
  cmdlineParams:
    es.index-prefix: "test"