Elasticsearch 8.x support

[Tornhoof]

Requirement - what kind of business use case are you trying to solve?

Elastic recently released Elasticsearch 8.0, by now they've released 8.1.

Problem - what in Jaeger blocks you from solving the requirement?

The ES index can't be created anymore, I think the legacy template api part is now completely deprecated:

jaeger-collector    | {"level":"info","ts":1646824639.4679213,"caller":"config/config.go:205","msg":"Elasticsearch detected","version":8}
jaeger-collector    | {"level":"info","ts":1646824639.4697242,"caller":"zapgrpc/zapgrpc.go:129","msg":"Deprecation warning: 299 Elasticsearch-8.1.0-3700f7679f7d95e36da0b43762189bab189bc53a \"Legacy index templates are deprecated in favor of composable templates.\""}
jaeger-collector    | {"level":"fatal","ts":1646824639.4699457,"caller":"./main.go:84","msg":"Failed to create span writer","error":"elastic: Error 400 (Bad Request): unknown key [template] in the template  [type=parse_exception]","stacktrace":"main.main.func1\n\t./main.go:84\ngithub.com/spf13/cobra.(*Command).execute\n\tgithub.com/spf13/cobra@v1.3.0/command.go:856\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tgithub.com/spf13/cobra@v1.3.0/command.go:974\ngithub.com/spf13/cobra.(*Command).Execute\n\tgithub.com/spf13/cobra@v1.3.0/command.go:902\nmain.main\n\t./main.go:147\nruntime.main\n\truntime/proc.go:255"}

The https://www.elastic.co/guide/en/elasticsearch/reference/8.1/migrating-8.0.html lists now (somewhere in the middle)

The [create or update index template API](https://www.elastic.co/guide/en/elasticsearch/reference/8.1/indices-templates-v1.html)’s template parameter has been removed.

Details
In 6.0, we deprecated the template parameter in create or update index template requests in favor of using index_patterns. Support for the template parameter is now removed in 8.0.

Impact
Use the create or update index template API's index_patterns parameter. Requests that include the template parameter will return an error.

I think the code falls back to the unversioned mappings if it does not find a specific mapping

So, in my opinion the code should not fall back to the jaeger-span.json mapping but instead use the jaeger-span-7.json for newer versions, e.g. use the -7 jsons for 8 etc.

e.g. change line 42 from == 7 to >=7 https://github.com/jaegertracing/jaeger/blob/7872d1b07439c3f2d316065b1fd53e885b26a66f/plugin/storage/es/mappings/mapping.go#L40-L46

Workaround

Force the ES Version to 7 via --es.version=7

[yurishkuro]

Would you like to submit a PR? Besides the logic fix it would need to add v8 to the CI steps.

[Tornhoof]

Sure, I'll give it a try.

[Ashmita152]

I spent time investigating the effort in achieving it. I think this is going to be difficult now that we use same elasticsearch client olivere/elastic to support both of our elasticsearch and opensearch implementations.

Major challenge: Right now we are using v6 of olivere/elastic which doesn't have support for new index template. To support it we need to update to v7 of olivere/elastic specifically for https://github.com/olivere/elastic/pull/1458 and https://github.com/olivere/elastic/pull/1415 which inturn breaks the opensearch support.

Not sure how we should proceed.

cc @yurishkuro

[s7an-it]

Workaround

Force the ES Version to 7 via --es.version=7

I have tried latest helm operator installation and setting this parameter didn't result in anything better.

[bitfactory-sem-denbroeder]

Would it be possible for Jaeger to add a separate docker image/branch which maintains opensearch support and the main branch getting support for Elasticsearch 8.x ?

[smithyj]

hope to support elasticsearch 8.1.x

[pavolloffay]

We are working on bootstrapping Jaeger v2 (#3500) which will be based on OTEL collector. This will allow us to have ES exporter per ES version.

[DanielBruzualRW]

Is there any workaround currently available, e.g., creating the index manually?

I tried the OP's suggested workaround (with elastic 8.1.3) and I get the following error:

legacy template [jaeger-span] has index patterns [*jaeger-span-*] matching patterns from existing composable templates [.deprecation-indexing-template,.ml-anomalies-,.ml-state,.ml-stats,.monitoring-beats-mb,.monitoring-ent-search-mb,.monitoring-es-mb,.monitoring-kibana-mb,.monitoring-logstash-mb,.slm-history,.watch-history-16,ilm-history,logs,metrics,synthetics] with patterns (.deprecation-indexing-template => [.logs-deprecation.*],.ml-anomalies- => [.ml-anomalies-*],.ml-state => [.ml-state*],.ml-stats => [.ml-stats-*],.monitoring-beats-mb => [.monitoring-beats-8-*],.monitoring-ent-search-mb => [.monitoring-ent-search-8-*],.monitoring-es-mb => [.monitoring-es-8-*],.monitoring-kibana-mb => [.monitoring-kibana-8-*],.monitoring-logstash-mb => [.monitoring-logstash-8-*],.slm-history => [.slm-history-5*],.watch-history-16 => [.watcher-history-16*],ilm-history => [ilm-history-5*],logs => [logs-*-*],metrics => [metrics-*-*],synthetics => [synthetics-*-*]), use composable templates (/_index_template) instead

[kruczjak]

I would be thankful for any workaround too.

We need to upgrade our elasticsearch cluster but we are stuck because of this problem

[seilorjunior]

we also need this asap!!!!!!1

[foxracle]

we also need this asap! +1

[sergiomcalzada]

Hi,

Workaround

Force the ES Version to 7 via --es.version=7 and es.create-index-templates: false

Add add this two indexes. If some of them fails related to "index priority" just change it

PUT _index_template/jaeger-service
{
  "priority": 1,
  "template": {
    "settings": {
      "index": {
        "mapping": {
          "nested_fields": {
            "limit": "50"
          }
        },
        "requests": {
          "cache": {
            "enable": "true"
          }
        },
        "number_of_shards": "6",
        "number_of_replicas": "1"
      }
    },
    "mappings": {
      "_routing": {
        "required": false
      },
      "_size": {
        "enabled": false
      },
      "numeric_detection": false,
      "dynamic_date_formats": [
        "strict_date_optional_time",
        "yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"
      ],
      "_source": {
        "excludes": [],
        "includes": [],
        "enabled": true
      },
      "dynamic": true,
      "dynamic_templates": [
        {
          "span_tags_map": {
            "path_match": "tag.*",
            "mapping": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        {
          "process_tags_map": {
            "path_match": "process.tag.*",
            "mapping": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        }
      ],
      "date_detection": true,
      "properties": {
        "operationName": {
          "type": "keyword"
        },
        "serviceName": {
          "type": "keyword"
        }
      }
    }
  },
  "index_patterns": [
    "*jaeger-service-*"
  ],
  "composed_of": []
}

PUT _index_template/jaeger-span
{
  "priority": 2,
  "template": {
    "settings": {
      "index": {
        "mapping": {
          "nested_fields": {
            "limit": "50"
          }
        },
        "requests": {
          "cache": {
            "enable": "true"
          }
        },
        "number_of_shards": "6",
        "number_of_replicas": "1"
      }
    },
    "mappings": {
      "_routing": {
        "required": false
      },
      "_size": {
        "enabled": false
      },
      "numeric_detection": false,
      "dynamic_date_formats": [
        "strict_date_optional_time",
        "yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"
      ],
      "dynamic": true,
      "_source": {
        "excludes": [],
        "includes": [],
        "enabled": true
      },
      "dynamic_templates": [
        {
          "span_tags_map": {
            "path_match": "tag.*",
            "mapping": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        {
          "process_tags_map": {
            "path_match": "process.tag.*",
            "mapping": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        }
      ],
      "date_detection": true,
      "properties": {
        "traceID": {
          "type": "keyword"
        },
        "process": {
          "type": "object",
          "properties": {
            "tag": {
              "type": "object"
            },
            "serviceName": {
              "type": "keyword"
            },
            "tags": {
              "type": "nested",
              "properties": {
                "tagType": {
                  "type": "keyword"
                },
                "value": {
                  "type": "keyword"
                },
                "key": {
                  "type": "keyword"
                }
              }
            }
          }
        },
        "references": {
          "type": "nested",
          "properties": {
            "spanID": {
              "type": "keyword"
            },
            "traceID": {
              "type": "keyword"
            },
            "refType": {
              "type": "keyword"
            }
          }
        },
        "startTimeMillis": {
          "type": "date"
        },
        "flags": {
          "type": "integer"
        },
        "operationName": {
          "type": "keyword"
        },
        "parentSpanID": {
          "type": "keyword"
        },
        "tags": {
          "type": "nested",
          "properties": {
            "tagType": {
              "type": "keyword"
            },
            "value": {
              "type": "keyword"
            },
            "key": {
              "type": "keyword"
            }
          }
        },
        "duration": {
          "type": "long"
        },
        "spanID": {
          "type": "keyword"
        },
        "startTime": {
          "type": "long"
        },
        "tag": {
          "type": "object",
          "properties": {}
        },
        "logs": {
          "type": "nested",
          "properties": {
            "fields": {
              "type": "nested",
              "properties": {
                "tagType": {
                  "type": "keyword"
                },
                "value": {
                  "type": "keyword"
                },
                "key": {
                  "type": "keyword"
                }
              }
            },
            "timestamp": {
              "type": "long"
            }
          }
        }
      }
    }
  },
  "index_patterns": [
    "*jaeger-span-*"
  ],
  "composed_of": []
}

[aletor92]

Workaround

It Works ! thanks 👍

[Evesy]

As part of adding support for the newer index template API, it might be nice to have the option for Jaeger to create component templates instead of index templates.

With the current API we have a couple of other index templates targetting jaeger indices that all get merged together to perform the following modifications:

• Set custom default settings that apply to all our indices (Tier routing, refresh interval, translog settings etc.)
• Modify some field mappings (e.g. we set eager_global_ordinals on traceID for better search performance)

With the new index template API this won't be possible as only one index template wll be chosen (highest priority) and even if Jaeger supporting passing composed_of values to the template, which would allow us to pull in our default index settings, anything still explicitly set by Jaeger template (e.g. the traceID field mapping) would take precedence over our modifications.

I'm not sure if this is a common use case or not, otherwise we may choose to disable Jaeger index template creation and just managed them entirely ourselves, however pulling in any new field mappings etc. automatically when upgrading Jaeger is handy

[jasperjonker]

Thanks @sergiomcalzada that worked:

storage:
    type: elasticsearch
    options:
      es:
        server-urls: http://elasticsearch.elastic-db:9200
        index-prefix: jaeger_
        version: 7                                  # Necessary as it doesn't work with 8 yet
        create-index-templates: false               # Necessary as it doesn't work with 8 yet

[sylvainOL]

Fyi, https://github.com/jaegertracing/spark-dependencies is not working with es8.x also.

That means it cannot really be used in production IMHO

[billalley]

We are experiencing problems connecting Jaeger to our existing ES setup due to this issue. I've gone ahead and followed the work around instructions above, currently using the Helm chart for setup.

Ex:

   cmdlineParams:
      es.version: 7
      es.create-index-templates: false 

But no matter I still see this error relating to the template index patterns:

"caller":"./main.go:69","msg":"Failed to create span writer","error":"elastic: Error 400 (Bad Request): legacy template [jaeger-span] has index patterns [*jaeger-span-*] matching patterns from existing composable templates

I have also tried removing the leading * from the templates, but the error message stayed the same which leads me to believe this is still on the Jaeger end of things.

Anyone have further workaround info/updates on ES8 support?

[billalley]

Looks like this was some yaml mangling - No matter what was specified, checking the config state on the ingester showed

spec:
  containers:
  - args:
    - --es.create-index-templates
    - --es.version=7

Solution: quote the false

   cmdlineParams:
      es.version: 7
      es.create-index-templates: "false"

Which becomes:

spec:
  containers:
  - args:
    - --es.create-index-templates=false
    - --es.version=7

[yurishkuro]

@Ashmita152 I came across https://github.com/olivere/elastic/issues/1533, where the author of the library unequivocally states:

Elastic v8 won't happen. I will polish v7 if I have the time and energy, but v8 won't happen. I'm sorry.

So there seems to be little point in upgrading to v7 of the library as it won't support v8 of ES.

Unlike our Cassandra implementation that uses an internal abstraction over gocql driver, the ES implementation has a large exposure to many different types from olivere/elastic driver. This complicates things a lot if we want to support both ESv8 and OpenSearch using different drivers.

ESv7 will be EOL-ed by 2023-08-01.

[ajax-osadchuk-r]

@yurishkuro Could you please clarify whether Jaeger is going to support ElasticSearch v8.+? At least, if there are any plans to do so in near times? This issue blocks our team and forces either to use outdated infrastructure or switch to another distributed tracing system.

[ajax-osadchuk-r]

@yurishkuro @pavolloffay Any thoughts about my previous comment?

[luanguimaraesla]

Thank you @sergiomcalzada. It works. We're also using ILM/aliases, which changes the configuration a little bit.

Script to apply all manifests

Make sure you named the files correctly

#! /bin/bash

function apply_ilm() {
    policy_name=$1
    file_name=$2
    curl -Ss -XPUT -H 'Content-Type: application/json' \
        -d @"${file_name}" \
        "http://localhost:9200/_ilm/policy/${policy_name}"
}

function apply_template() {
    template_name=$1
    file_name=$2
    curl -Ss -XPUT -H 'Content-Type: application/json' \
        -d @"${file_name}" \
        "http://localhost:9200/_index_template/${template_name}"
}

function apply_index() {
    index_name=$1
    file_name=$2
    curl -Ss -XPUT -H 'Content-Type: application/json' \
        -d @"${file_name}" \
        "http://localhost:9200/${index_name}"
}

apply_ilm "jaeger" jaeger.ilm.json
apply_template "jaeger-service" jaeger-service.tpl.json
apply_template "jaeger-span" jaeger-span.tpl.json
apply_index "jaeger-service-000001" jaeger-service-000001.idx.json
apply_index "jaeger-span-000001" jaeger-span-000001.idx.json

ILM manfest (jaeger.ilm.json)

{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_age": "4h"
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "delete": {
        "min_age": "2d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Spans Composable Template (jaeger-span.tpl.json)

{
  "version": 3,
  "index_patterns": [
    "*jaeger-span-*"
  ],
  "priority": 101,
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "jaeger",
          "rollover_alias": "jaeger-span-write"
        },
        "mapping": {
          "nested_fields": {
            "limit": "50"
          }
        },
        "requests": {
          "cache": {
            "enable": "true"
          }
        },
        "number_of_shards": 5
      }
    },
    "aliases": {
      "jaeger-span-read": {}
    },
    "mappings": {
      "_routing": {
        "required": false
      },
      "numeric_detection": false,
      "dynamic_date_formats": [
        "strict_date_optional_time",
        "yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"
      ],
      "dynamic": true,
      "_source": {
        "excludes": [],
        "includes": [],
        "enabled": true
      },
      "dynamic_templates": [
        {
          "span_tags_map": {
            "path_match": "tag.*",
            "mapping": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        {
          "process_tags_map": {
            "path_match": "process.tag.*",
            "mapping": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        }
      ],
      "date_detection": true,
      "properties": {
        "traceID": {
          "type": "keyword"
        },
        "process": {
          "type": "object",
          "properties": {
            "tag": {
              "type": "object"
            },
            "serviceName": {
              "type": "keyword"
            },
            "tags": {
              "type": "nested",
              "properties": {
                "tagType": {
                  "type": "keyword"
                },
                "value": {
                  "type": "keyword"
                },
                "key": {
                  "type": "keyword"
                }
              }
            }
          }
        },
        "references": {
          "type": "nested",
          "properties": {
            "spanID": {
              "type": "keyword"
            },
            "traceID": {
              "type": "keyword"
            },
            "refType": {
              "type": "keyword"
            }
          }
        },
        "startTimeMillis": {
          "type": "date"
        },
        "flags": {
          "type": "integer"
        },
        "operationName": {
          "type": "keyword"
        },
        "parentSpanID": {
          "type": "keyword"
        },
        "tags": {
          "type": "nested",
          "properties": {
            "tagType": {
              "type": "keyword"
            },
            "value": {
              "type": "keyword"
            },
            "key": {
              "type": "keyword"
            }
          }
        },
        "duration": {
          "type": "long"
        },
        "spanID": {
          "type": "keyword"
        },
        "startTime": {
          "type": "long"
        },
        "tag": {
          "type": "object",
          "properties": {}
        },
        "logs": {
          "type": "nested",
          "properties": {
            "fields": {
              "type": "nested",
              "properties": {
                "tagType": {
                  "type": "keyword"
                },
                "value": {
                  "type": "keyword"
                },
                "key": {
                  "type": "keyword"
                }
              }
            },
            "timestamp": {
              "type": "long"
            }
          }
        }
      }
    }
  },
  "composed_of": []
}

Services Composable Template (jaeger-service.tpl.json)

{
  "version": 3,
  "index_patterns": [
    "*jaeger-service-*"
  ],
  "priority": 102,
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "jaeger",
          "rollover_alias": "jaeger-service-write"
        },
        "mapping": {
          "nested_fields": {
            "limit": "50"
          }
        },
        "requests": {
          "cache": {
            "enable": "true"
          }
        },
        "number_of_shards": 5
      }
    },
    "aliases": {
      "jaeger-service-read": {}
    },
    "mappings": {
      "_routing": {
        "required": false
      },
      "numeric_detection": false,
      "dynamic_date_formats": [
        "strict_date_optional_time",
        "yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"
      ],
      "dynamic": true,
      "_source": {
        "excludes": [],
        "includes": [],
        "enabled": true
      },
      "dynamic_templates": [
        {
          "span_tags_map": {
            "path_match": "tag.*",
            "mapping": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        {
          "process_tags_map": {
            "path_match": "process.tag.*",
            "mapping": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        }
      ],
      "date_detection": true,
      "properties": {
        "operationName": {
          "type": "keyword"
        },
        "serviceName": {
          "type": "keyword"
        }
      }
    }
  },
  "composed_of": []
}

Spans rollover origin index (jaeger-span-000001.idx.json)

Since it's not possible to add rollover writable aliases using templates, you'll have to deploy the origin index manually.

{
    "aliases": {
        "jaeger-span-write": {
            "is_write_index": true
        }
    }
}

Services rollover origin index (jaeger-service-000001.idx.json)

{
    "aliases": {
        "jaeger-service-write": {
            "is_write_index": true
        }
    }
}

K8s Jaeger Operator

If you're using the k8s operator to deploy your Jaeger stack, the all-in-one distribution won't work since it'll try to create the rollover job because of the necessary flag use-aliases: true. You'll have to use the version of the separate components. This is a basic setup using Jaeger CRD manifest:

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: jaeger
spec:
  collector:
    options:
      es:
        use-aliases: true
        use-ilm: true
  query:
    options:
      es:
        use-aliases: true
        use-ilm: true
  storage:
    type: elasticsearch
    options:
      es:
        version: 7
        create-index-templates: false
        server-urls: http://elasticsearch:9200

  # Necessary for elasticsearch integration to work properly
  # read more: https://github.com/jaegertracing/jaeger-operator/issues/1187
  strategy: production

[jicki]

hope to support elasticsearch 8.2.x

[nonamef]

Elasticsearch is up to 8.5.0 and still waiting on an update? Using out of date infrastructure in production is becoming a burden.

[jkowall]

Opensearch is a good alternative. Not sure there is interest in building support for 8.x+. There are also other backends which are efficient as options.

[PetrusZ]

We are experiencing problems connecting Jaeger to our existing ES setup due to this issue. I've gone ahead and followed the work around instructions above, currently using the Helm chart for setup.

Ex:

   cmdlineParams:
      es.version: 7
      es.create-index-templates: false 

But no matter I still see this error relating to the template index patterns:

"caller":"./main.go:69","msg":"Failed to create span writer","error":"elastic: Error 400 (Bad Request): legacy template [jaeger-span] has index patterns [*jaeger-span-*] matching patterns from existing composable templates

I have also tried removing the leading * from the templates, but the error message stayed the same which leads me to believe this is still on the Jaeger end of things.

Anyone have further workaround info/updates on ES8 support?

try to use env as workaroud:

    extraEnv:
      - name: ES_VERSION
        value: "7"
      - name: ES_CREATE_INDEX_TEMPLATES
        value: "false"

[guitcastro]

mangli

This is a bug in the helm char: False property are treat as true.

https://github.com/jaegertracing/helm-charts/blob/main/charts/jaeger/templates/collector-deploy.yaml#L53

[rajims07]

which value we need to modify to work properly

[guitcastro]

which value we need to modify to work properly

As workaround? Just use "false" instead of false.

To fix the chart, check if values are empty or null instead of false:

The bugged code:

          {{- if $value }}
          - --{{ $key }}={{ $value }}
          {{- else }}
          - --{{ $key }}

[sheng-jie]

Any progress?

[stevenkitter]

anybody can tell me, now it support to 8.x??

[Renatas-M]

Facing issues after upgrading to ES8. Jaeger Collector startup errors can be fixed with workaround by adding --es.version=7 and --es.create-index-templates=false command parameters. But then Jaeger Query is erroring with _Error 400 (Bad Request): all shards failed [type=search_phase_executionexception]. Thanks to sergiomcalzada, the suggested manual index template creation is helping to restore Jaeger Query functionality. I am applying suggested templates using Kibana->Dev Tools->Console.

[sunxuan11]

Is there any workaround currently available, e.g., creating the index manually?

I tried the OP's suggested workaround (with elastic 8.1.3) and I get the following error:

legacy template [jaeger-span] has index patterns [*jaeger-span-*] matching patterns from existing composable templates [.deprecation-indexing-template,.ml-anomalies-,.ml-state,.ml-stats,.monitoring-beats-mb,.monitoring-ent-search-mb,.monitoring-es-mb,.monitoring-kibana-mb,.monitoring-logstash-mb,.slm-history,.watch-history-16,ilm-history,logs,metrics,synthetics] with patterns (.deprecation-indexing-template => [.logs-deprecation.*],.ml-anomalies- => [.ml-anomalies-*],.ml-state => [.ml-state*],.ml-stats => [.ml-stats-*],.monitoring-beats-mb => [.monitoring-beats-8-*],.monitoring-ent-search-mb => [.monitoring-ent-search-8-*],.monitoring-es-mb => [.monitoring-es-8-*],.monitoring-kibana-mb => [.monitoring-kibana-8-*],.monitoring-logstash-mb => [.monitoring-logstash-8-*],.slm-history => [.slm-history-5*],.watch-history-16 => [.watcher-history-16*],ilm-history => [ilm-history-5*],logs => [logs-*-*],metrics => [metrics-*-*],synthetics => [synthetics-*-*]), use composable templates (/_index_template) instead

[sunxuan11]

Have you solved it? How did you solve it?

[sunxuan11]

Is there any workaround currently available, e.g., creating the index manually?

I tried the OP's suggested workaround (with elastic 8.1.3) and I get the following error:

legacy template [jaeger-span] has index patterns [*jaeger-span-*] matching patterns from existing composable templates [.deprecation-indexing-template,.ml-anomalies-,.ml-state,.ml-stats,.monitoring-beats-mb,.monitoring-ent-search-mb,.monitoring-es-mb,.monitoring-kibana-mb,.monitoring-logstash-mb,.slm-history,.watch-history-16,ilm-history,logs,metrics,synthetics] with patterns (.deprecation-indexing-template => [.logs-deprecation.*],.ml-anomalies- => [.ml-anomalies-*],.ml-state => [.ml-state*],.ml-stats => [.ml-stats-*],.monitoring-beats-mb => [.monitoring-beats-8-*],.monitoring-ent-search-mb => [.monitoring-ent-search-8-*],.monitoring-es-mb => [.monitoring-es-8-*],.monitoring-kibana-mb => [.monitoring-kibana-8-*],.monitoring-logstash-mb => [.monitoring-logstash-8-*],.slm-history => [.slm-history-5*],.watch-history-16 => [.watcher-history-16*],ilm-history => [ilm-history-5*],logs => [logs-*-*],metrics => [metrics-*-*],synthetics => [synthetics-*-*]), use composable templates (/_index_template) instead

I have the same problem with you

[pie-r]

I think the index-template workaround, could be implemented in the initContainers of the collector deployment. I've created the MR https://github.com/jaegertracing/helm-charts/issues/452. If you are using the jaeger operator helm chart, you may get adavantage from this.

[mikednight]

for the most part our system is working, however the init phase of rollover still returns: Error: failed to create template: jaeger-span, request failed, status code: 400, body: {"error":{"root_cause":[{"type":"parse_exception","reason":"unknown key [template] in the template "}],"type":"parse_exception","reason":"unknown key [template] in the template "},"status":400}

the actual rollover returns: Error: failed to create rollover target: jaeger-span-write, request failed, status code: 404, body: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [jaeger-span-write]","resource.type":"index_or_alias","resource.id":"jaeger-span-write","index_uuid":"_na_","index":"jaeger-span-write"}],"type":"index_not_found_exception","reason":"no such index [jaeger-span-write]","resource.type":"index_or_alias","resource.id":"jaeger-span-write","index_uuid":"_na_","index":"jaeger-span-write"},"status":404}

is there a known workaround for this?

Also, is there a timeline on when actual support will be ready for elasticsearch 8?

thank you.

[MephistoLynn]

Thanks @sergiomcalzada that worked:

storage:
    type: elasticsearch
    options:
      es:
        server-urls: http://elasticsearch.elastic-db:9200
        index-prefix: jaeger_
        version: 7                                  # Necessary as it doesn't work with 8 yet
        create-index-templates: false               # Necessary as it doesn't work with 8 yet

Thanks @sergiomcalzada that worked:

storage:
    type: elasticsearch
    options:
      es:
        server-urls: http://elasticsearch.elastic-db:9200
        index-prefix: jaeger_
        version: 7                                  # Necessary as it doesn't work with 8 yet
        create-index-templates: false               # Necessary as it doesn't work with 8 yet

Thanks. It works.感谢方法有效。

[s7an-it]

@sergiomcalzada, on 8.7 I had to delete from both templates, should I replace it with something else? "_size": { "enabled": false }, Thanks it started working like that, I had to restart all (jaeger and eck operator components) and then I had to do delete all old indexes from Jaeger and wait a bit.

[jpuskar]

Opensearch is a good alternative. Not sure there is interest in building support for 8.x+. There are also other backends which are efficient as options.

Unfortunately, none of the others have operators as robust as ECK. E.g. Sure OpenSearch has a helm chart, but it's missing a ton of functionality.

[fmassot]

@jpuskar what are your requirements in terms of features/dataset size?

[paulc0te]

Hi. This topic is still relevant. Is there any plan to support Elasticsearch 8? The problem is exacerbated by the fact that there are currently no supported helm charts Elasticsearch 7.

[henrygas2018]

The _template refers to the legacy template, while the _index_template is a new version of the template known as composable templates. In ES 7.x, both _template and _index_template are supported. However, users should manage their index_patterns to be split. As such, you may need to modify the _template created by the jaeger-collector and remove the prefix * in index_patterns.

First, execute the following command:

GET /_template/jaeger-span

Next, copy the response content. Change *jaeger-span-* to jaeger-span-*, and keep the rest of the content unchanged.

Finally, execute the following command in ES:

PUT /_template/jaeger-span
{
    ...
    "index_patterns": [
        "jaeger-span-*"
    ],
    ...
}

This will update the template accordingly, notice the response warning.

[odinsy]

@henrygas2018 or you can just don't set prefix and everything will be work (if you created ILM policy before jaeger start)

storage:
  type: elasticsearch
  elasticsearch:
    # indexPrefix: jaeger
    ...
    cmdlineParams:
      es.version: 7
      es.create-index-templates: "false"
      # es.index-prefix: jaeger

collector:
  enabled: true
  cmdlineParams:
    es.use-aliases: true
    es.use-ilm: true

query:
  enabled: true
  cmdlineParams:
    es.use-aliases: true
    es.use-ilm: true

[fatihdogmus]

Elastic v7 has ben EOL as of 1st of August. If someone wants to use jaeger properly, they would have to install an EOL version.

Is there at least a timeline on when can we expect support for Elasticsearch 8?

[haanhvu]

@fatihdogmus This has been proposed as a project for LFX mentorship: https://github.com/jaegertracing/jaeger/issues/4600 So someone's going to start working on it in Sep.

[stmlange]

Just in case someone bumps into this: With ES_VERSION=7 (or --es.version=7 and ES_CREATE_INDEX_TEMPLATES="false" (or es.create-index-templates="false") as suggested above one still might get the following cryptic error message:

"Failed to init storage factory","error":"failed to create primary Elasticsearch client: health check timeout: no Elasticsearch node available"

Any suggestions?

[sergeykad]

Is there a way to create all the configurations in a fresh installation on Elasticsearch 8 or the only option is to install with ES7 and then upgrade to 8 using the workarounds mentioned above?

[sergiomcalzada]

You can apply all the workarounds in a fresh v8 (tested). You will need to create the index by your self, you can find the code for the indexes in a previous comment from me

[sergeykad]

You can apply all the workarounds in a fresh v8 (tested). You will need to create the index by your self, you can find the code for the indexes in a previous comment from me

I did this, but there are missing aliases and ILM related configurations.