Migrates AWS Elasticsearch Storage to armeria/jackson

[codefromthecrypt]

Obviates #134 which I've lost write access to :P

[codefromthecrypt]

self tracing is a little bit messed up, but I don't think that's related to this. I think main thing was sorting out the contents of the jar.

While this works, we shouldn't merge until we are on a final version. There are a few dents to knock out upstream such as logging the health check. Plus we shouldn't release zipkin 2.16 until the next armeria release I think, as that could cleanup some code here also.

[codefromthecrypt]

by self-tracing messed up I mean that there are no child spans of the "accept-spans" call. There should be a bulk http POST request

[codefromthecrypt]

@anuraaga I noticed that we made some utilities for moshi which aren't relevant when using jackson. This helped cleanup the code a good bit I think. Thanks for the other feedback as well. PTAL

[codefromthecrypt]

ps you have karma so feel free to update the branch. I have to look at other things a bit

[codefromthecrypt]

Maybe not worth it - IIUC, this is always going to be client-initiated and not sampled.

actually I think on first request it would be server initiated.. we can check.

[codefromthecrypt]

oops..

2019-07-29 15:12:42.278  WARN 78315 --- [-worker-nio-4-2] .a.c.b.RequestContextCurrentTraceContext : Attempted to propagate trace context, but no request context available. Did you forget to use RequestContext.contextAwareExecutor() or RequestContext.makeContextAware()?

com.linecorp.armeria.common.brave.RequestContextCurrentTraceContext$LogRequestContextWarningOnce$NoRequestContextException: null
    at com.linecorp.armeria.common.brave.RequestContextCurrentTraceContext$LogRequestContextWarningOnce$ClassLoaderHack.<clinit>(RequestContextCurrentTraceContext.java:266) ~[armeria-brave-0.89.0.jar!/:?]
    at com.linecorp.armeria.common.brave.RequestContextCurrentTraceContext$LogRequestContextWarningOnce.get(RequestContextCurrentTraceContext.java:254) ~[armeria-brave-0.89.0.jar!/:?]
    at com.linecorp.armeria.common.brave.RequestContextCurrentTraceContext$LogRequestContextWarningOnce.get(RequestContextCurrentTraceContext.java:247) ~[armeria-brave-0.89.0.jar!/:?]
    at com.linecorp.armeria.common.RequestContext.mapCurrent(RequestContext.java:108) ~[armeria-0.89.0.jar!/:?]
    at com.linecorp.armeria.common.brave.RequestContextCurrentTraceContext.getRequestContextOrWarnOnce(RequestContextCurrentTraceContext.java:232) ~[armeria-brave-0.89.0.jar!/:?]
    at com.linecorp.armeria.common.brave.RequestContextCurrentTraceContext.get(RequestContextCurrentTraceContext.java:143) ~[armeria-brave-0.89.0.jar!/:?]
    at brave.Tracer.nextSpan(Tracer.java:472) ~[brave-5.6.9.jar!/:?]
    at brave.http.HttpClientHandler.nextSpan(HttpClientHandler.java:127) ~[brave-instrumentation-http-5.6.9.jar!/:?]
    at brave.http.HttpClientHandler.handleSend(HttpClientHandler.java:89) ~[brave-instrumentation-http-5.6.9.jar!/:?]
    at com.linecorp.armeria.client.brave.BraveClient.execute(BraveClient.java:121) ~[armeria-brave-0.89.0.jar!/:?]
    at com.linecorp.armeria.client.brave.BraveClient.execute(BraveClient.java:56) ~[armeria-brave-0.89.0.jar!/:?]
    at zipkin.autoconfigure.storage.elasticsearch.aws.AWSSignatureVersion4.lambda$execute$2(AWSSignatureVersion4.java:98) ~[elasticsearch-aws.jar!/:?]
    at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1072) ~[?:?]
    at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
    at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) ~[?:?]
    at com.linecorp.armeria.common.HttpMessageAggregator.onComplete(HttpMessageAggregator.java:98) ~[armeria-0.89.0.jar!/:?]
    at com.linecorp.armeria.common.stream.AbstractStreamMessage$CloseEvent.notifySubscriber(AbstractStreamMessage.java:363) ~[armeria-0.89.0.jar!/:?]
    at com.linecorp.armeria.common.stream.FixedStreamMessage.notifySubscriberOfCloseEvent(FixedStreamMessage.java:126) ~[armeria-0.89.0.jar!/:?]
    at com.linecorp.armeria.common.stream.EmptyFixedStreamMessage.doRequest(EmptyFixedStreamMessage.java:32) ~[armeria-0.89.0.jar!/:?]
    at com.linecorp.armeria.common.stream.FixedStreamMessage.request(FixedStreamMessage.java:88) ~[armeria-0.89.0.jar!/:?]
    at com.linecorp.armeria.common.stream.AbstractStreamMessage$SubscriptionImpl.request(AbstractStreamMessage.java:309) ~[armeria-0.89.0.jar!/:?]
    at com.linecorp.armeria.common.HttpMessageAggregator.onSubscribe(HttpMessageAggregator.java:55) ~[armeria-0.89.0.jar!/:?]
    at com.linecorp.armeria.common.stream.FixedStreamMessage.lambda$subscribe$1(FixedStreamMessage.java:117) ~[armeria-0.89.0.jar!/:?]
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [netty-common-4.1.38.Final.jar!/:4.1.38.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:416) [netty-common-4.1.38.Final.jar!/:4.1.38.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:515) [netty-transport-4.1.38.Final.jar!/:4.1.38.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) [netty-common-4.1.38.Final.jar!/:4.1.38.Final]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.38.Final.jar!/:4.1.38.Final]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.38.Final.jar!/:4.1.38.Final]
    at java.lang.Thread.run(Thread.java:834) [?:?]

[codefromthecrypt]

I didn't mention here, but basically we are at functional level if not using implicit url lookup. As these requests need to be signed I need to do some more refactoring upstream first. https://github.com/openzipkin/zipkin/pull/2711

[codefromthecrypt]

@anuraaga I did a manual check and put a unit test in with the data, which checks out.

Wild guess is there is something different about how things go on the wire from what's executed in the DomainEndpoint api vs our other requests. Even if self-tracing is disabled the domain endpoint requests fail when they don't at the same time on my workstation.

If you know a way to see the real request (ex https proxy), it might be a start... or I suppose we could make it possible to use http plain text to see whats actually getting onto the wire.

here's a bad request from this commit.

2019-07-30 22:15:32.915  INFO 21647 --- [-worker-nio-2-2] c.l.a.c.l.LoggingClient                  : Request: {startTime=2019-07-30T14:15:32.909Z(1564496132909667), length=0B, duration=3335µs(3335369ns), scheme=none+h1, headers=[:method=GET, :path=/2015-01-01/es/domain/zipkin, :authority=es.ap-southeast-1.amazonaws.com, :scheme=https, x-amz-date=20190730T141532Z, host=es.ap-southeast-1.amazonaws.com, authorization=AWS4-HMAC-SHA256 Credential=AKIAJGY6SK4B32OMPXUQ/20190730/ap-southeast-1/es/aws4_request, SignedHeaders=host;x-amz-date, Signature=3605843acfad95d9aff88f6a0352a9808e48611060df2ec563341e24f3db690d, accept-encoding=gzip,deflate, user-agent=armeria/0.89.0]}

here's a good one from the same time using boto

<AWSPreparedRequest stream_output=False, method=GET, url=https://es.ap-southeast-1.amazonaws.com/2015-01-01/es/domain/zipkin, headers={'User-Agent': b'aws-cli/1.16.190 Python/3.7.4 Darwin/18.7.0 botocore/1.12.180', 'X-Amz-Date': b'20190730T141529Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAJGY6SK4B32OMPXUQ/20190730/ap-southeast-1/es/aws4_request, SignedHeaders=host;x-amz-date, Signature=b51df7a627ea8e8495ca93575fb354bb5adf213df443d4012edd9c6297d661d9'}

[anuraaga]

Thanks for adding the test. I think It's possible to log all traffic (literally the bytes) by enabling TRACE level logging of the logger com.linecorp.armeria.logging.traffic.client in the log4j config.

https://github.com/line/armeria/blob/master/core/src/main/java/com/linecorp/armeria/internal/TrafficLoggingHandler.java#L34

[codefromthecrypt]

Thanks.. here's a failed request

         +-------------------------------------------------+
         |  0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f |
+--------+-------------------------------------------------+----------------+
|00000000| 47 45 54 20 2f 32 30 31 35 2d 30 31 2d 30 31 2f |GET /2015-01-01/|
|00000010| 65 73 2f 64 6f 6d 61 69 6e 2f 7a 69 70 6b 69 6e |es/domain/zipkin|
|00000020| 20 48 54 54 50 2f 31 2e 31 0d 0a 68 6f 73 74 3a | HTTP/1.1..host:|
|00000030| 20 65 73 2e 61 70 2d 73 6f 75 74 68 65 61 73 74 | es.ap-southeast|
|00000040| 2d 31 2e 61 6d 61 7a 6f 6e 61 77 73 2e 63 6f 6d |-1.amazonaws.com|
|00000050| 0d 0a 68 6f 73 74 3a 20 65 73 2e 61 70 2d 73 6f |..host: es.ap-so|
|00000060| 75 74 68 65 61 73 74 2d 31 2e 61 6d 61 7a 6f 6e |utheast-1.amazon|
|00000070| 61 77 73 2e 63 6f 6d 0d 0a 78 2d 61 6d 7a 2d 64 |aws.com..x-amz-d|
|00000080| 61 74 65 3a 20 32 30 31 39 30 37 33 30 54 31 34 |ate: 20190730T14|
|00000090| 35 39 35 34 5a 0d 0a 61 75 74 68 6f 72 69 7a 61 |5954Z..authoriza|
|000000a0| 74 69 6f 6e 3a 20 41 57 53 34 2d 48 4d 41 43 2d |tion: AWS4-HMAC-|
|000000b0| 53 48 41 32 35 36 20 43 72 65 64 65 6e 74 69 61 |SHA256 Credentia|
|000000c0| 6c 3d 41 4b 49 41 34 49 50 48 4c 43 45 4b 49 51 |l=AKIA4IPHLCEKIQ|
|000000d0| 41 54 42 4f 55 4e 2f 32 30 31 39 30 37 33 30 2f |ATBOUN/20190730/|
|000000e0| 61 70 2d 73 6f 75 74 68 65 61 73 74 2d 31 2f 65 |ap-southeast-1/e|
|000000f0| 73 2f 61 77 73 34 5f 72 65 71 75 65 73 74 2c 20 |s/aws4_request, |
|00000100| 53 69 67 6e 65 64 48 65 61 64 65 72 73 3d 68 6f |SignedHeaders=ho|
|00000110| 73 74 3b 78 2d 61 6d 7a 2d 64 61 74 65 2c 20 53 |st;x-amz-date, S|
|00000120| 69 67 6e 61 74 75 72 65 3d 35 31 62 61 63 36 62 |ignature=51bac6b|
|00000130| 31 65 34 62 39 61 61 65 38 63 39 34 62 66 30 30 |1e4b9aae8c94bf00|
|00000140| 64 63 32 35 34 61 66 63 33 39 34 30 37 65 33 34 |dc254afc39407e34|
|00000150| 32 65 30 36 35 62 37 35 38 32 61 30 35 30 66 30 |2e065b7582a050f0|
|00000160| 32 33 65 33 66 36 64 33 38 0d 0a 61 63 63 65 70 |23e3f6d38..accep|
|00000170| 74 2d 65 6e 63 6f 64 69 6e 67 3a 20 67 7a 69 70 |t-encoding: gzip|
|00000180| 2c 64 65 66 6c 61 74 65 0d 0a 75 73 65 72 2d 61 |,deflate..user-a|
|00000190| 67 65 6e 74 3a 20 61 72 6d 65 72 69 61 2f 30 2e |gent: armeria/0.|
|000001a0| 38 39 2e 30 0d 0a 0d 0a                         |89.0....        |
+--------+-------------------------------------------------+----------------+

Note this is http/1 and the actual requests to ES itself are http/2. that's another difference

[codefromthecrypt]

fyi turning in for real now. ttul8r

[codefromthecrypt]

wait.. it looks like the host header is repeated. I wonder if that's what's up. maybe we have to add the host header on http2 but not 1.1?

[anuraaga]

From reading the docs, it looked like http/1 uses host header and http/2 uses authority header. I guess they might be using the repeated host header on http/1 and the non-repeated authority header for http/2 - while we use a non-repeated value for host when computing the signature. The repeated host header looks like it could be a very silly bug but seems promising in theory.

[anuraaga]

Found the Armeria bug, will send a PR there - I think our work around would be to delete the HOST header after signing, before delegating.

[codefromthecrypt]

stil didn't work but I can't even read the screen anymore. good time to stop. thanks for chasing up

[codefromthecrypt]

actually it did work, just it got an exception.. com.linecorp.armeria.client.endpoint.EndpointGroupException: com.linecorp.armeria.client.endpoint.dns.DnsAddressEndpointGroup@686c4360 is empty

I need to move the other fix here about the slacker resolution :P

[codefromthecrypt]

finally!

ok this works, so we can now do cleanups etc. as long as we super-verify that this continues to work I'm also ok with re-enabling the health check stuff. thx again for the epic support @anuraaga

[codefromthecrypt]

./mvnw clean install -DskipTests
mv ./autoconfigure/storage-elasticsearch-aws/target/zipkin-autoconfigure-storage-elasticsearch-aws-0.17.3-SNAPSHOT-module.jar elasticsearch-aws.jar
TAG=master-SNAPSHOT
curl -sSL https://jitpack.io/com/github/openzipkin/zipkin/zipkin-server/${TAG}/zipkin-server-${TAG}-exec.jar > zipkin.jar
SELF_TRACING_ENABLED=true ES_AWS_DOMAIN=zipkin-ha ES_HTTP_LOGGING=BODY STORAGE_TYPE=elasticsearch    java     -Dloader.path='elasticsearch-aws.jar,elasticsearch-aws.jar!/lib'     -Dspring.profiles.active=elasticsearch-aws     -cp zipkin.jar     org.springframework.boot.loader.PropertiesLauncher

[codefromthecrypt]

running wrk and trying to do health check I get 503..

 2019-08-09 13:28:43 ⌚  MacBook-Pro-7 in ~/oss/zipkin-aws
± |elasticsearch-armeria {1} ?:4 ✗| → curl -s localhost:9411/actuator/prometheus|grep .0
503 Service Unavailable

[codefromthecrypt]

sorry this is health check and also prometheus. I'll try to figure out what's up..

[codefromthecrypt]

with logging on, you can see the health request go back and forth, but the health check itself is impatient I guess?

request/response for domain endpoint

2019-08-09 13:32:35.579  INFO 9762 --- [-worker-nio-2-2] c.l.a.c.l.LoggingClient                  : Request: {startTime=2019-08-09T05:32:35.578Z(1565328755578079), length=0B, duration=754µs(754310ns), scheme=none+h1, headers=[:method=GET, :path=/2015-01-01/es/domain/zipkin, :authority=es.ap-southeast-1.amazonaws.com, :scheme=https, x-amz-date=20190809T053235Z, authorization=AWS4-HMAC-SHA256 Credential=AKIA4IPHLCEKIQATBOUN/20190809/ap-southeast-1/es/aws4_request, SignedHeaders=host;x-amz-date, Signature=1558887a5267a800b3bdfbb9790842d4b77c51ac8624e369eb7cae49063ac446, b3=6d9dda115a2c437b-61db88a71f38f649-0-6d9dda115a2c437b, accept-encoding=gzip,deflate, user-agent=armeria/unknown]}
2019-08-09 13:32:35.741  INFO 9762 --- [-worker-nio-2-2] c.l.a.c.l.LoggingClient                  : Response: {startTime=2019-08-09T05:32:35.740Z(1565328755740153), length=1540B, duration=917µs(917564ns), headers=[:status=200, x-amzn-requestid=0d8ea2ae-ba67-11e9-9296-479f22947f13, content-type=application/json, content-length=1540, date=Fri, 09 Aug 2019 05:32:18 GMT], content={"DomainStatus":{"ARN":"arn:aws:es:ap-southeast-1:842835628308:domain/zipkin","AccessPolicies":"{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":{\"AWS\":\"*\"},\"Action\":\"es:*\",\"Resource\":\"arn:aws:es:ap-southeast-1:842835628308:domain/zipkin/*\",\"Condition\":{\"IpAddress\":{\"aws:SourceIp\":\"36.2.165.225\"}}}]}","AdvancedOptions":{"rest.action.multi.allow_explicit_index":"true"},"CognitoOptions":{"Enabled":false,"IdentityPoolId":null,"RoleArn":null,"UserPoolId":null},"Created":true,"Deleted":false,"DomainId":"842835628308/zipkin","DomainName":"zipkin","EBSOptions":{"EBSEnabled":true,"Iops":null,"VolumeSize":10,"VolumeType":"gp2"},"ElasticsearchClusterConfig":{"DedicatedMasterCount":null,"DedicatedMasterEnabled":false,"DedicatedMasterType":null,"InstanceCount":1,"InstanceType":"t2.small.elasticsearch","ZoneAwarenessConfig":null,"ZoneAwarenessEnabled":false},"ElasticsearchVersion":"6.7","EncryptionAtRestOptions":{"Enabled":false,"KmsKeyId":null},"Endpoint":"search-zipkin-2rlyh66ibw43ftlk4342ceeewu.ap-southeast-1.es.amazonaws.com","Endpoints":null,"LogPublishingOptions":null,"NodeToNodeEncryptionOptions":{"Enabled":false},"Processing":false,"ServiceSoftwareOptions":{"AutomatedUpdateDate":1.56709052E9,"Cancellable":false,"CurrentVersion":"R20190724","Description":"There is no software update available for this domain.","NewVersion":"","UpdateAvailable":false,"UpdateStatus":"COMPLETED"},"SnapshotOptions":{"AutomatedSnapshotStartHour":0},"UpgradeProcessing":false,"VPCOptions":null}}}

health check relating to the above call

curl -s localhost:9411/health
{"status":{"status":"DOWN"},"zipkin":"Timed out computing health status. This often means your storage backend is unreachable."}

[codefromthecrypt]

hmm.. trying a different domain.. at last the prometheus endpoint can show the endpoints, which I can hit, but of course these are signed urls..

$ curl -s localhost:9411/actuator/prometheus|grep health
# HELP armeria_client_endpointGroup_healthy  
# TYPE armeria_client_endpointGroup_healthy gauge
armeria_client_endpointGroup_healthy{authority="search-zipkin-2rlyh66ibw43ftlk4342ceeewu.ap-southeast-1.es.amazonaws.com:443",ip="52.77.100.49",name="elasticsearch",} 0.0
armeria_client_endpointGroup_healthy{authority="search-zipkin-2rlyh66ibw43ftlk4342ceeewu.ap-southeast-1.es.amazonaws.com:443",ip="54.251.169.105",name="elasticsearch",} 0.0
armeria_client_endpointGroup_count{name="elasticsearch",state="unhealthy",} 2.0
armeria_client_endpointGroup_count{name="elasticsearch",state="healthy",} 0.0
$ curl -s https://search-zipkin-2rlyh66ibw43ftlk4342ceeewu.ap-southeast-1.es.amazonaws.com/_cluster/health
{"Message":"User: anonymous is not authorized to perform: es:ESHttpGet"}

[codefromthecrypt]

if I disable client-side health-check, it is back..

run

ES_AWS_DOMAIN=zipkin ES_HTTP_LOGGING=BODY  STORAGE_TYPE=elasticsearch java     -Dloader.path='elasticsearch-aws.jar,elasticsearch-aws.jar!/lib'     -Dspring.profiles.active=elasticsearch-aws -cp zipkin.jar     org.springframework.boot.loader.PropertiesLauncher --zipkin.storage.elasticsearch.health-check.enabled=false

check

$ curl -s localhost:9411/health
{"status":"UP","zipkin":{"status":"UP","details":{"ElasticsearchStorage{initialEndpoints=aws://ap-southeast-1/zipkin, index=zipkin}":{"status":"UP"}}}}

[codefromthecrypt]

works, though on a bad network a ton of timeout messages and an interesting one about http/2 reset. I think we should consider adding com.linecorp.armeria.client.ResponseTimeoutException to the list of exceptions that indicate StorageComponent.isOverCapacity though it is debateable..

com.linecorp.armeria.client.ResponseTimeoutException: null

2019-08-09 17:48:19.459  WARN 24215 --- [worker-nio-2-13] c.l.a.c.HttpResponseDecoder              : Unexpected exception:

com.linecorp.armeria.client.ResponseTimeoutException: null

2019-08-09 17:48:19.466  WARN 24215 --- [-worker-nio-2-2] c.l.a.c.HttpResponseDecoder              : Unexpected exception:

com.linecorp.armeria.client.ResponseTimeoutException: null

2019-08-09 17:48:19.467  WARN 24215 --- [-worker-nio-2-4] c.l.a.c.HttpResponseDecoder              : Unexpected exception:

com.linecorp.armeria.client.ResponseTimeoutException: null

2019-08-09 17:48:19.469  WARN 24215 --- [-worker-nio-2-4] c.l.a.c.HttpRequestSubscriber            : [id: 0xfa0a1a64, L:/192.168.1.108:64213 - R:search-zipkin-2rlyh66ibw43ftlk4342ceeewu.ap-southeast-1.es.amazonaws.com/54.251.169.105:443][h2] Unexpected exception:

io.netty.handler.codec.http2.Http2Exception$StreamException: Maximum active streams violated for this endpoint.
    at io.netty.handler.codec.http2.Http2Exception.streamError(Http2Exception.java:147) ~[netty-codec-http2-4.1.38.Final.jar!/:4.1.38.Final]
    at io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultEndpoint.checkNewStreamAllowed(DefaultHttp2Connection.java:896) ~[netty-codec-http2-4.1.38.Final.jar!/:4.1.38.Final]
    at io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultEndpoint.createStream(DefaultHttp2Connection.java:748) ~[netty-codec-http2-4.1.38.Final.jar!/:4.1.38.Final]
    at io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultEndpoint.createStream(DefaultHttp2Connection.java:668) ~[netty-codec-http2-4.1.38.Final.jar!/:4.1.38.Final]
    at io.netty.handler.codec.http2.DefaultHttp2ConnectionEncoder.writeHeaders(DefaultHttp2ConnectionEncoder.java:176) ~[netty-codec-http2-4.1.38.Final.jar!/:4.1.38.Final]
    at io.netty.handler.codec.http2.DefaultHttp2ConnectionEncoder.writeHeaders(DefaultHttp2ConnectionEncoder.java:151) ~[netty-codec-http2-4.1.38.Final.jar!/:4.1.38.Final]
    at com.linecorp.armeria.internal.Http2ObjectEncoder.doWriteHeaders(Http2ObjectEncoder.java:73) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
    at com.linecorp.armeria.internal.HttpObjectEncoder.writeHeaders(HttpObjectEncoder.java:56) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
    at com.linecorp.armeria.client.HttpRequestSubscriber.write0(HttpRequestSubscriber.java:279) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
    at com.linecorp.armeria.client.HttpRequestSubscriber.writeFirstHeader(HttpRequestSubscriber.java:173) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
    at com.linecorp.armeria.client.HttpRequestSubscriber.onSubscribe(HttpRequestSubscriber.java:151) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
    at com.linecorp.armeria.common.stream.FixedStreamMessage.subscribe(FixedStreamMessage.java:115) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
    at com.linecorp.armeria.common.stream.AbstractStreamMessage.subscribe(AbstractStreamMessage.java:93) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
    at com.linecorp.armeria.common.stream.AbstractStreamMessage.subscribe(AbstractStreamMessage.java:84) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
    at com.linecorp.armeria.common.HeaderOverridingHttpRequest.subscribe(HeaderOverridingHttpRequest.java:113) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
    at com.linecorp.armeria.client.HttpSessionHandler.invoke(HttpSessionHandler.java:157) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
    at com.linecorp.armeria.client.HttpClientDelegate.doExecute(HttpClientDelegate.java:234) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
    at com.linecorp.armeria.client.HttpClientDelegate.acquireConnectionAndExecute(HttpClientDelegate.java:136) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
    at com.linecorp.armeria.client.HttpClientDelegate.execute(HttpClientDelegate.java:83) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
    at com.linecorp.armeria.client.HttpClientDelegate.execute(HttpClientDelegate.java:42) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
    at com.linecorp.armeria.client.encoding.HttpDecodingClient.execute(HttpDecodingClient.java:92) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
    at com.linecorp.armeria.client.encoding.HttpDecodingClient.execute(HttpDecodingClient.java:39) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
    at zipkin2.server.internal.elasticsearch.ZipkinElasticsearchStorageConfiguration.lambda$esTracing$1(ZipkinElasticsearchStorageConfiguration.java:144) ~[classes!/:?]
    at com.linecorp.armeria.client.FunctionalDecoratingClient.execute(FunctionalDecoratingClient.java:47) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
    at com.linecorp.armeria.client.brave.BraveClient.execute(BraveClient.java:131) ~[armeria-brave-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
    at com.linecorp.armeria.client.brave.BraveClient.execute(BraveClient.java:56) ~[armeria-brave-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
    at zipkin.autoconfigure.storage.elasticsearch.aws.AWSSignatureVersion4.lambda$execute$2(AWSSignatureVersion4.java:93) ~[elasticsearch-aws.jar!/:?]
    at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:642) ~[?:?]
    at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
    at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) ~[?:?]
    at com.linecorp.armeria.common.HttpMessageAggregator.onComplete(HttpMessageAggregator.java:98) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
    at com.linecorp.armeria.common.stream.AbstractStreamMessage$CloseEvent.notifySubscriber(AbstractStreamMessage.java:363) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
    at com.linecorp.armeria.common.stream.FixedStreamMessage.notifySubscriberOfCloseEvent(FixedStreamMessage.java:126) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
    at com.linecorp.armeria.common.stream.OneElementFixedStreamMessage.doNotify(OneElementFixedStreamMessage.java:70) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
    at com.linecorp.armeria.common.stream.OneElementFixedStreamMessage.doRequest(OneElementFixedStreamMessage.java:55) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
    at com.linecorp.armeria.common.stream.FixedStreamMessage.request(FixedStreamMessage.java:88) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
    at com.linecorp.armeria.common.stream.AbstractStreamMessage$SubscriptionImpl.request(AbstractStreamMessage.java:309) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
    at com.linecorp.armeria.common.HttpMessageAggregator.onSubscribe(HttpMessageAggregator.java:55) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
    at com.linecorp.armeria.common.stream.FixedStreamMessage.lambda$subscribe$1(FixedStreamMessage.java:117) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
    at com.linecorp.armeria.common.AbstractRequestContext.lambda$makeContextAware$1(AbstractRequestContext.java:69) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [netty-common-4.1.38.Final.jar!/:4.1.38.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:416) [netty-common-4.1.38.Final.jar!/:4.1.38.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:515) [netty-transport-4.1.38.Final.jar!/:4.1.38.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) [netty-common-4.1.38.Final.jar!/:4.1.38.Final]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.38.Final.jar!/:4.1.38.Final]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.38.Final.jar!/:4.1.38.Final]
    at java.lang.Thread.run(Thread.java:834) [?:?]

[codefromthecrypt]

retested against released binaries. still works

[codefromthecrypt]

another epic thanks @anuraaga