Elasticsearch 7.8 is not supported

[benqs]

Tried to setup zipkin with elasticsearch 7.8 as a storage. Got following errors, while trying to view traces in ui.

`2020-07-03 11:33:51.103 WARN 1 --- [king-tasks-1-32] z.s.i.BodyIsExceptionMessage : Unexpected error handling request.

java.lang.RuntimeException: Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [traceId] in order to load field data by uninverting the inverted index. Note that this can use significant memory. at zipkin2.elasticsearch.internal.client.HttpCall.lambda$parseResponse$4(HttpCall.java:272) ~[zipkin-storage-elasticsearch-2.21.4.jar:?] at zipkin2.elasticsearch.internal.client.HttpCall.parseResponse(HttpCall.java:282) ~[zipkin-storage-elasticsearch-2.21.4.jar:?] at zipkin2.elasticsearch.internal.client.HttpCall.doExecute(HttpCall.java:173) ~[zipkin-storage-elasticsearch-2.21.4.jar:?] at zipkin2.Call$Base.execute(Call.java:380) ~[zipkin-2.21.4.jar:?] at zipkin2.Call$FlatMapping.doExecute(Call.java:276) ~[zipkin-2.21.4.jar:?] at zipkin2.Call$Base.execute(Call.java:380) ~[zipkin-2.21.4.jar:?] at zipkin2.Call$Mapping.doExecute(Call.java:237) ~[zipkin-2.21.4.jar:?] at zipkin2.Call$Base.execute(Call.java:380) ~[zipkin-2.21.4.jar:?] at zipkin2.Call$Mapping.doExecute(Call.java:237) ~[zipkin-2.21.4.jar:?] at zipkin2.Call$Base.execute(Call.java:380) ~[zipkin-2.21.4.jar:?] at zipkin2.server.internal.ZipkinQueryApiV2.getTraces(ZipkinQueryApiV2.java:147) ~[classes/:?] at jdk.internal.reflect.GeneratedMethodAccessor61.invoke(Unknown Source) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?] at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?] at com.linecorp.armeria.internal.server.annotation.AnnotatedService.invoke(AnnotatedService.java:294) ~[armeria-0.99.6.jar:?] at com.linecorp.armeria.internal.server.annotation.AnnotatedService.lambda$serve0$8(AnnotatedService.java:276) ~[armeria-0.99.6.jar:?] at java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$Completion.run(Unknown Source) ~[?:?] at com.linecorp.armeria.common.RequestContext.lambda$makeContextAware$3(RequestContext.java:489) ~[armeria-0.99.6.jar:?] at io.micrometer.core.instrument.internal.TimedRunnable.run(TimedRunnable.java:44) [micrometer-core-1.5.1.jar:1.5.1] at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?] at java.util.concurrent.FutureTask.run(Unknown Source) [?:?] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.50.Final.jar:4.1.50.Final] at java.lang.Thread.run(Unknown Source) [?:?] `

`2020-07-03 11:37:23.709 WARN 1 --- [king-tasks-1-33] z.s.i.BodyIsExceptionMessage : Unexpected error handling request.

java.lang.NullPointerException: null at zipkin2.elasticsearch.internal.JsonSerializers.parseSpan(JsonSerializers.java:67) ~[zipkin-storage-elasticsearch-2.21.4.jar:?] at zipkin2.elasticsearch.internal.client.SearchResultConverter.convert(SearchResultConverter.java:46) ~[zipkin-storage-elasticsearch-2.21.4.jar:?] at zipkin2.elasticsearch.internal.client.SearchResultConverter.convert(SearchResultConverter.java:27) ~[zipkin-storage-elasticsearch-2.21.4.jar:?] at zipkin2.elasticsearch.internal.client.HttpCall.parseResponse(HttpCall.java:282) ~[zipkin-storage-elasticsearch-2.21.4.jar:?] at zipkin2.elasticsearch.internal.client.HttpCall.doExecute(HttpCall.java:173) ~[zipkin-storage-elasticsearch-2.21.4.jar:?] at zipkin2.Call$Base.execute(Call.java:380) ~[zipkin-2.21.4.jar:?] at zipkin2.server.internal.ZipkinQueryApiV2.getTrace(ZipkinQueryApiV2.java:156) ~[classes/:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?] at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?] at com.linecorp.armeria.internal.server.annotation.AnnotatedService.invoke(AnnotatedService.java:294) ~[armeria-0.99.6.jar:?] at com.linecorp.armeria.internal.server.annotation.AnnotatedService.lambda$serve0$8(AnnotatedService.java:276) ~[armeria-0.99.6.jar:?] at java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$Completion.run(Unknown Source) ~[?:?] at com.linecorp.armeria.common.RequestContext.lambda$makeContextAware$3(RequestContext.java:489) ~[armeria-0.99.6.jar:?] at io.micrometer.core.instrument.internal.TimedRunnable.run(TimedRunnable.java:44) [micrometer-core-1.5.1.jar:1.5.1] at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?] at java.util.concurrent.FutureTask.run(Unknown Source) [?:?] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.50.Final.jar:4.1.50.Final] at java.lang.Thread.run(Unknown Source) [?:?] `

[codefromthecrypt]

We haven't tried 7.8 yet.. did you try 7.7? I just tried that a couple days ago..

On Fri, Jul 3, 2020 at 7:41 PM benqs notifications@github.com wrote:

Tried to setup zipkin with elasticsearch 7.8 as a storage. Got following errors, while trying to view traces in ui.

`2020-07-03 11:33:51.103 WARN 1 --- [king-tasks-1-32] z.s.i.BodyIsExceptionMessage : Unexpected error handling request.

java.lang.RuntimeException: Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [traceId] in order to load field data by uninverting the inverted index. Note that this can use significant memory. at zipkin2.elasticsearch.internal.client.HttpCall.lambda$parseResponse$4(HttpCall.java:272) ~[zipkin-storage-elasticsearch-2.21.4.jar:?] at zipkin2.elasticsearch.internal.client.HttpCall.parseResponse(HttpCall.java:282) ~[zipkin-storage-elasticsearch-2.21.4.jar:?] at zipkin2.elasticsearch.internal.client.HttpCall.doExecute(HttpCall.java:173) ~[zipkin-storage-elasticsearch-2.21.4.jar:?] at zipkin2.Call$Base.execute(Call.java:380) ~[zipkin-2.21.4.jar:?] at zipkin2.Call$FlatMapping.doExecute(Call.java:276) ~[zipkin-2.21.4.jar:?] at zipkin2.Call$Base.execute(Call.java:380) ~[zipkin-2.21.4.jar:?] at zipkin2.Call$Mapping.doExecute(Call.java:237) ~[zipkin-2.21.4.jar:?] at zipkin2.Call$Base.execute(Call.java:380) ~[zipkin-2.21.4.jar:?] at zipkin2.Call$Mapping.doExecute(Call.java:237) ~[zipkin-2.21.4.jar:?] at zipkin2.Call$Base.execute(Call.java:380) ~[zipkin-2.21.4.jar:?] at zipkin2.server.internal.ZipkinQueryApiV2.getTraces(ZipkinQueryApiV2.java:147) ~[classes/:?] at jdk.internal.reflect.GeneratedMethodAccessor61.invoke(Unknown Source) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?] at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?] at com.linecorp.armeria.internal.server.annotation.AnnotatedService.invoke(AnnotatedService.java:294) ~[armeria-0.99.6.jar:?] at com.linecorp.armeria.internal.server.annotation.AnnotatedService.lambda$serve0$8(AnnotatedService.java:276) ~[armeria-0.99.6.jar:?] at java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$Completion.run(Unknown Source) ~[?:?] at com.linecorp.armeria.common.RequestContext.lambda$makeContextAware$3(RequestContext.java:489) ~[armeria-0.99.6.jar:?] at io.micrometer.core.instrument.internal.TimedRunnable.run(TimedRunnable.java:44) [micrometer-core-1.5.1.jar:1.5.1] at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?] at java.util.concurrent.FutureTask.run(Unknown Source) [?:?] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.50.Final.jar:4.1.50.Final] at java.lang.Thread.run(Unknown Source) [?:?] `

`2020-07-03 11:37:23.709 WARN 1 --- [king-tasks-1-33] z.s.i.BodyIsExceptionMessage : Unexpected error handling request.

java.lang.NullPointerException: null at zipkin2.elasticsearch.internal.JsonSerializers.parseSpan(JsonSerializers.java:67) ~[zipkin-storage-elasticsearch-2.21.4.jar:?] at zipkin2.elasticsearch.internal.client.SearchResultConverter.convert(SearchResultConverter.java:46) ~[zipkin-storage-elasticsearch-2.21.4.jar:?] at zipkin2.elasticsearch.internal.client.SearchResultConverter.convert(SearchResultConverter.java:27) ~[zipkin-storage-elasticsearch-2.21.4.jar:?] at zipkin2.elasticsearch.internal.client.HttpCall.parseResponse(HttpCall.java:282) ~[zipkin-storage-elasticsearch-2.21.4.jar:?] at zipkin2.elasticsearch.internal.client.HttpCall.doExecute(HttpCall.java:173) ~[zipkin-storage-elasticsearch-2.21.4.jar:?] at zipkin2.Call$Base.execute(Call.java:380) ~[zipkin-2.21.4.jar:?] at zipkin2.server.internal.ZipkinQueryApiV2.getTrace(ZipkinQueryApiV2.java:156) ~[classes/:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?] at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?] at com.linecorp.armeria.internal.server.annotation.AnnotatedService.invoke(AnnotatedService.java:294) ~[armeria-0.99.6.jar:?] at com.linecorp.armeria.internal.server.annotation.AnnotatedService.lambda$serve0$8(AnnotatedService.java:276) ~[armeria-0.99.6.jar:?] at java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$Completion.run(Unknown Source) ~[?:?] at com.linecorp.armeria.common.RequestContext.lambda$makeContextAware$3(RequestContext.java:489) ~[armeria-0.99.6.jar:?] at io.micrometer.core.instrument.internal.TimedRunnable.run(TimedRunnable.java:44) [micrometer-core-1.5.1.jar:1.5.1] at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?] at java.util.concurrent.FutureTask.run(Unknown Source) [?:?] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.50.Final.jar:4.1.50.Final] at java.lang.Thread.run(Unknown Source) [?:?] `

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/openzipkin/zipkin/issues/3132, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAPVV7PDBDOIQHCNWD47YLRZW7VJANCNFSM4OPXRKBA .

[benqs]

Indeed, works without problem on 7.7, but since I'm upgrading all clusters to 7.8 this one is a blocker.

[codefromthecrypt]

the index template isn't installed or is inconsistent in your cluster. Zipkin will install an index template upon startup by default. you can set ES_HTTP_LOGGING to see why your index template isn't installing as it should by default.

https://github.com/openzipkin/zipkin/tree/master/zipkin-server#elasticsearch-storage

I just checked with the last published version so I don't think this is a problem at the moment. I'd check to see why index templates aren't installed.

$ curl -s localhost:9411/health
{
  "status" : "UP",
  "zipkin" : {
    "status" : "UP",
    "details" : {
      "ElasticsearchStorage{initialEndpoints=http://localhost:9200, index=zipkin}" : {
        "status" : "UP"
      }
    }
  }
}
$ curl -s localhost:9411/info
{"zipkin":{"version":"2.21.4","commit":"0863da0"}}

$ curl -s localhost:9200
{
  "name" : "MacBook-Pro.local",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "4BFXk9zVTYSpOuxegPRzuA",
  "version" : {
    "number" : "7.8.0",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "757314695644ea9a1dc2fecd26d1a43856725e65",
    "build_date" : "2020-06-14T19:35:50.234439Z",
    "build_snapshot" : false,
    "lucene_version" : "8.5.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

[codefromthecrypt]

FYI just did a patch release updating our demo image of ES to 7.8.0 just to make sure and things look ok. Good luck!

https://github.com/openzipkin/zipkin/tree/master/docker/examples#elasticsearch

[codefromthecrypt]

another tool maybe helpful is https://github.com/tetratelabs/zipkin-es-templater from @basvanbeek and other tetrands

[benqs]

It turned out that problem was my _index_template (introduced in 7.8 https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html) with pattern "*", which served some default values for all indices. Since _index_template took priority over zipkin templates in _template, mappings provided by those templates were ignored.

[codefromthecrypt]

eek.. wonder if we can do anything about this? maybe look conditionally for composable ones.. or use a composeable for our templates when ES 7.8+?

[benqs]

Composeable templates are prefered in ES 7.8+, since documentation is refering to _template as legacy ones. Another issue it composeable template priority, either it would have to determine automatically what value should be set or be parameterizable through env variable.

[mostafabayat]

When I upgrade my elasticsearch cluster to 7.16.2, I faced this problem too. I create composable template manually, based on legacy ones, and everything became working!

[jcchavezs]

Do you think there is something we should improve in the docs?

On Mon, 19 Sept 2022, 03:59 Mostafa Bayat, @.***> wrote:

When I upgrade my elasticsearch cluster to 7.16.2, I faced this problem too. I create composable template manually, based on legacy ones, and everything became working!

— Reply to this email directly, view it on GitHub https://github.com/openzipkin/zipkin-support/issues/23#issuecomment-1250749063, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAXOYARJGM2XT63YF5BBCULV7ATNVANCNFSM4OPZZMMA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[mostafabayat]

Do you think there is something we should improve in the docs?

Yes, I think we can add a note in this section, mentioning that if your elasticsearch version is 7.8+, you should create three composable index templates, zipkin-autocomplete_template ,zipkin-dependency_template, and zipkin-span_template.
But I personally think that it's the responsibility of zipkin itself, not its users. It means this problem should address in the program!

[jcchavezs]

Zipkin is a volunteering Open Source project, meaning that it is responsibility of the community and the community is mainly its users. Are you up to add that note in the readme?

On Tue, 20 Sept 2022, 07:28 Mostafa Bayat, @.***> wrote:

Do you think there is something we should improve in the docs?

Yes, I think we can add a note in this section https://github.com/openzipkin/zipkin/tree/master/zipkin-server#elasticsearch-storage, mentioning that if your elasticsearch version is 7.8+, you should create three composable index templates, zipkin-autocomplete_template , zipkin-dependency_template, and zipkin-span_template. But I personally think that it's the responsibility of zipkin itself, not its users. It means this problem should address in the program!

— Reply to this email directly, view it on GitHub https://github.com/openzipkin/zipkin-support/issues/23#issuecomment-1252283223, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAXOYAT62JOZMIX2Y547ZHLV7GUXDANCNFSM4OPZZMMA . You are receiving this because you commented.Message ID: @.***>