zipkin 2.14.0 can not run on the elasticsearch 7.0.0

[lushan01]

Describe the Bug

My environment is zipkin version: 2.14.0, elasticsearch version: 7.0.0

when opening zipkin web, browser prints "ERROR: cannot load span names: server error (Internal Server Error), ERROR: cannot load service names: server error (Internal Server Error)", zipkin‘s error log is as follows:

2019-05-24 09:04:15.279  WARN 1 --- [king-tasks-1-37] c.l.a.i.a.DefaultExceptionHandler        : [id: 0x909d4eb0, L:/172.30.23.132:9411 - R:/172.30.23.136:50786][h1c://zipkin-d955589cf-9zlbb:9411/zipkin/api/v2/services#GET] Unhandled exception from an annotated service:

java.lang.IllegalStateException: over capacity
    at zipkin2.elasticsearch.internal.client.HttpCall.doExecute(HttpCall.java:79) ~[zipkin-storage-elasticsearch-2.14.0.jar!/:2.14.0]
    at zipkin2.Call$Base.execute(Call.java:382) ~[zipkin-2.14.0.jar!/:2.14.0]
    at zipkin2.server.internal.brave.TracedCall.execute(TracedCall.java:40) ~[classes/:?]
    at zipkin2.server.internal.ZipkinQueryApiV2.getServiceNames(ZipkinQueryApiV2.java:86) ~[classes/:?]
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
    at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
    at com.linecorp.armeria.internal.annotation.AnnotatedHttpService.invoke(AnnotatedHttpService.java:258) ~[armeria-0.85.0.jar!/:?]
    at com.linecorp.armeria.internal.annotation.AnnotatedHttpService.lambda$serve0$3(AnnotatedHttpService.java:245) ~[armeria-0.85.0.jar!/:?]
    at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:642) ~[?:?]
    at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478) ~[?:?]
    at com.linecorp.armeria.common.AbstractRequestContext.lambda$makeContextAware$1(AbstractRequestContext.java:69) ~[armeria-0.85.0.jar!/:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.34.Final.jar!/:4.1.34.Final]
    at java.lang.Thread.run(Thread.java:834) [?:?]

[shakuzen]

The error message indicates that you are sending more load to your Elasticsearch cluster than it can handle. You may need to do some tuning of the configuration of your Elasticsearch cluster, scale the cluster, or sample less tracing data in your applications so there is less load.

[lushan01]

@shakuzen My environment is dev, less tracing data, so elasticsearch is very free.

[codefromthecrypt]

we don't do troubleshooting in github issues. please use https://gitter.im/openzipkin/zipkin for this.

Please look at the release notes as there's also a throttled option from @Logic-32 which might help you. https://github.com/apache/incubator-zipkin/releases