search

openzipkin / zipkin-aws

Reporters and collectors for use in Amazon's cloud
Apache License 2.0
69 stars 34 forks source link

zipkin aws storage elasticsearch not working with aws elsticsearch version 7+ #196

Closed ax-korneenkov closed 9 months ago

ax-korneenkov commented 2 years ago

I am using elasticsearch 7.10 image on aws. When I connect to elasticsearch I use the openzipkin/zipkin-aws:0.23 image. and i get an error: 

DEBUG 1 --- [orker-epoll-2-2] z.s.i.ZipkinHttpCollector                : Cannot decode spans due to IllegalArgumentException(Expected value at line 1 column 1 path $ reading List<Span> from json)

java.lang.IllegalArgumentException: Expected value at line 1 column 1 path $ reading List<Span> from json
    at zipkin2.internal.JsonCodec.exceptionReading(JsonCodec.java:230) ~[zipkin-2.21.7.jar:?]
    at zipkin2.internal.JsonCodec.readList(JsonCodec.java:151) ~[zipkin-2.21.7.jar:?]
    at zipkin2.codec.SpanBytesDecoder$3.decodeList(SpanBytesDecoder.java:127) ~[zipkin-2.21.7.jar:?]
    at zipkin2.codec.SpanBytesDecoder.doDecodeList(SpanBytesDecoder.java:209) ~[zipkin-2.21.7.jar:?]
    at zipkin2.codec.SpanBytesDecoder$3.decodeList(SpanBytesDecoder.java:143) ~[zipkin-2.21.7.jar:?]
    at zipkin2.collector.Collector.acceptSpans(Collector.java:149) ~[zipkin-collector-2.21.7.jar:?]
    at zipkin2.server.internal.ZipkinHttpCollector.lambda$validateAndStoreSpans$0(ZipkinHttpCollector.java:154) ~[classes/:?]
    at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) [?:?]
    at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) [?:?]
    at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) [?:?]
    at java.util.concurrent.CompletableFuture.complete(Unknown Source) [?:?]
    at com.linecorp.armeria.common.HttpMessageAggregator.onComplete(HttpMessageAggregator.java:92) [armeria-0.99.9.jar:?]
    at com.linecorp.armeria.common.stream.AbstractStreamMessage$CloseEvent.notifySubscriber(AbstractStreamMessage.java:269) [armeria-0.99.9.jar:?]
    at com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriberOfCloseEvent0(DefaultStreamMessage.java:264) [armeria-0.99.9.jar:?]
    at com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriberOfCloseEvent(DefaultStreamMessage.java:256) [armeria-0.99.9.jar:?]
    at com.linecorp.armeria.common.stream.DefaultStreamMessage.handleCloseEvent(DefaultStreamMessage.java:436) [armeria-0.99.9.jar:?]
    at com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriber0(DefaultStreamMessage.java:368) [armeria-0.99.9.jar:?]
    at com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriber(DefaultStreamMessage.java:315) [armeria-0.99.9.jar:?]
    at com.linecorp.armeria.common.stream.DefaultStreamMessage.addObjectOrEvent(DefaultStreamMessage.java:301) [armeria-0.99.9.jar:?]
    at com.linecorp.armeria.common.stream.DefaultStreamMessage.close(DefaultStreamMessage.java:443) [armeria-0.99.9.jar:?]
    at com.linecorp.armeria.server.Http1RequestDecoder.channelRead(Http1RequestDecoder.java:253) [armeria-0.99.9.jar:?]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.51.Final.jar:4.1.51.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
    at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [netty-codec-4.1.51.Final.jar:4.1.51.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) [netty-codec-4.1.51.Final.jar:4.1.51.Final]
    at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
    at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.51.Final.jar:4.1.51.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
    at io.netty.handler.flush.FlushConsolidationHandler.channelRead(FlushConsolidationHandler.java:152) [netty-handler-4.1.51.Final.jar:4.1.51.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
    at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) [netty-transport-native-epoll-4.1.51.Final-linux-x86_64.jar:4.1.51.Final]
    at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) [netty-transport-native-epoll-4.1.51.Final-linux-x86_64.jar:4.1.51.Final]
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) [netty-transport-native-epoll-4.1.51.Final-linux-x86_64.jar:4.1.51.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.51.Final.jar:4.1.51.Final]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.51.Final.jar:4.1.51.Final]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.51.Final.jar:4.1.51.Final]
    at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: zipkin2.internal.gson.stream.MalformedJsonException: Expected value at line 1 column 1 path $
    at zipkin2.internal.gson.stream.JsonReader.syntaxError(JsonReader.java:1564) ~[zipkin-2.21.7.jar:?]
    at zipkin2.internal.gson.stream.JsonReader.doPeek(JsonReader.java:591) ~[zipkin-2.21.7.jar:?]
    at zipkin2.internal.gson.stream.JsonReader.beginArray(JsonReader.java:344) ~[zipkin-2.21.7.jar:?]
    at zipkin2.internal.JsonCodec$JsonReader.beginArray(JsonCodec.java:55) ~[zipkin-2.21.7.jar:?]
    at zipkin2.internal.JsonCodec.readList(JsonCodec.java:145) ~[zipkin-2.21.7.jar:?]

Such problems were not noticed with the use of elasticsearch aws version 6.8

I suspect that this may be due to the fact that in elasticsearch 7+ Indices prior to 7.0 could contain a colon (:), but that’s been deprecated and won’t be supported in 7.0+ https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html

jcchavezs commented 2 years ago

Ping @xeraa

José Carlos Chávez

fre. 21. jan. 2022 kl. 07:35 skrev ax-korneenkov @.***>:

I am using elasticsearch 7.10 image on aws. When I connect to elasticsearch I use the openzipkin/zipkin-aws:0.23 image. and i get an error:

DEBUG 1 --- [orker-epoll-2-2] z.s.i.ZipkinHttpCollector : Cannot decode spans due to IllegalArgumentException(Expected value at line 1 column 1 path $ reading List from json)

java.lang.IllegalArgumentException: Expected value at line 1 column 1 path $ reading List from json

at zipkin2.internal.JsonCodec.exceptionReading(JsonCodec.java:230) ~[zipkin-2.21.7.jar:?]

at zipkin2.internal.JsonCodec.readList(JsonCodec.java:151) ~[zipkin-2.21.7.jar:?]

at zipkin2.codec.SpanBytesDecoder$3.decodeList(SpanBytesDecoder.java:127) ~[zipkin-2.21.7.jar:?]

at zipkin2.codec.SpanBytesDecoder.doDecodeList(SpanBytesDecoder.java:209) ~[zipkin-2.21.7.jar:?]

at zipkin2.codec.SpanBytesDecoder$3.decodeList(SpanBytesDecoder.java:143) ~[zipkin-2.21.7.jar:?]

at zipkin2.collector.Collector.acceptSpans(Collector.java:149) ~[zipkin-collector-2.21.7.jar:?]

at zipkin2.server.internal.ZipkinHttpCollector.lambda$validateAndStoreSpans$0(ZipkinHttpCollector.java:154) ~[classes/:?]

at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) [?:?]

at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) [?:?]

at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) [?:?]

at java.util.concurrent.CompletableFuture.complete(Unknown Source) [?:?]

at com.linecorp.armeria.common.HttpMessageAggregator.onComplete(HttpMessageAggregator.java:92) [armeria-0.99.9.jar:?]

at com.linecorp.armeria.common.stream.AbstractStreamMessage$CloseEvent.notifySubscriber(AbstractStreamMessage.java:269) [armeria-0.99.9.jar:?]

at com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriberOfCloseEvent0(DefaultStreamMessage.java:264) [armeria-0.99.9.jar:?]

at com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriberOfCloseEvent(DefaultStreamMessage.java:256) [armeria-0.99.9.jar:?]

at com.linecorp.armeria.common.stream.DefaultStreamMessage.handleCloseEvent(DefaultStreamMessage.java:436) [armeria-0.99.9.jar:?]

at com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriber0(DefaultStreamMessage.java:368) [armeria-0.99.9.jar:?]

at com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriber(DefaultStreamMessage.java:315) [armeria-0.99.9.jar:?]

at com.linecorp.armeria.common.stream.DefaultStreamMessage.addObjectOrEvent(DefaultStreamMessage.java:301) [armeria-0.99.9.jar:?]

at com.linecorp.armeria.common.stream.DefaultStreamMessage.close(DefaultStreamMessage.java:443) [armeria-0.99.9.jar:?]

at com.linecorp.armeria.server.Http1RequestDecoder.channelRead(Http1RequestDecoder.java:253) [armeria-0.99.9.jar:?]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.51.Final.jar:4.1.51.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.51.Final.jar:4.1.51.Final]

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.51.Final.jar:4.1.51.Final]

at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.51.Final.jar:4.1.51.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.51.Final.jar:4.1.51.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.51.Final.jar:4.1.51.Final]

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.51.Final.jar:4.1.51.Final]

at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) [netty-transport-4.1.51.Final.jar:4.1.51.Final]

at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [netty-codec-4.1.51.Final.jar:4.1.51.Final]

at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) [netty-codec-4.1.51.Final.jar:4.1.51.Final]

at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) [netty-transport-4.1.51.Final.jar:4.1.51.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.51.Final.jar:4.1.51.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.51.Final.jar:4.1.51.Final]

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.51.Final.jar:4.1.51.Final]

at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.51.Final.jar:4.1.51.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.51.Final.jar:4.1.51.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.51.Final.jar:4.1.51.Final]

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.51.Final.jar:4.1.51.Final]

at io.netty.handler.flush.FlushConsolidationHandler.channelRead(FlushConsolidationHandler.java:152) [netty-handler-4.1.51.Final.jar:4.1.51.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.51.Final.jar:4.1.51.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.51.Final.jar:4.1.51.Final]

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.51.Final.jar:4.1.51.Final]

at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.51.Final.jar:4.1.51.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.51.Final.jar:4.1.51.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.51.Final.jar:4.1.51.Final]

at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.51.Final.jar:4.1.51.Final]

at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) [netty-transport-native-epoll-4.1.51.Final-linux-x86_64.jar:4.1.51.Final]

at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) [netty-transport-native-epoll-4.1.51.Final-linux-x86_64.jar:4.1.51.Final]

at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) [netty-transport-native-epoll-4.1.51.Final-linux-x86_64.jar:4.1.51.Final]

at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.51.Final.jar:4.1.51.Final]

at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.51.Final.jar:4.1.51.Final]

at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.51.Final.jar:4.1.51.Final]

at java.lang.Thread.run(Unknown Source) [?:?]

Caused by: zipkin2.internal.gson.stream.MalformedJsonException: Expected value at line 1 column 1 path $

at zipkin2.internal.gson.stream.JsonReader.syntaxError(JsonReader.java:1564) ~[zipkin-2.21.7.jar:?]

at zipkin2.internal.gson.stream.JsonReader.doPeek(JsonReader.java:591) ~[zipkin-2.21.7.jar:?]

at zipkin2.internal.gson.stream.JsonReader.beginArray(JsonReader.java:344) ~[zipkin-2.21.7.jar:?]

at zipkin2.internal.JsonCodec$JsonReader.beginArray(JsonCodec.java:55) ~[zipkin-2.21.7.jar:?]

at zipkin2.internal.JsonCodec.readList(JsonCodec.java:145) ~[zipkin-2.21.7.jar:?]

Such problems were not noticed with the use of elasticsearch aws version 6.8

I suspect that this may be due to the fact that in elasticsearch 7+ Indices prior to 7.0 could contain a colon (:), but that’s been deprecated and won’t be supported in 7.0+

https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html

— Reply to this email directly, view it on GitHub https://github.com/openzipkin/zipkin-aws/issues/196, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAXOYATIESIM5JUO6ECL6VTUXD5E3ANCNFSM5MOWFOSA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you are subscribed to this thread.Message ID: @.***>

xeraa commented 2 years ago
  1. Yes, you cannot create indices with a colon in 7.0+ any more but you can still read from it.
  2. There was an earlier discussion on it in https://github.com/openzipkin/zipkin/issues/2219 and I thought this was generally fixed? Also 7.0 came out like 2 years ago, so there must be something special / uncommon going on here.
xf0e commented 1 year ago

Hello, we can't upgrade our ELK-Cluter from version 6 to 7 due to this ":"-restriction. Is there any chance to receive an update for this issue. Idealy with a fix, so zipkin cat create an Index in Elastic >= 7.0?

shakuzen commented 1 year ago

@xf0e As @xeraa mentioned, it's not a limitation in Zipkin; it is a restriction in Elasticsearch that an index cannot be created with : from Elasticsearch 7. You will have to use a different delimiter. As far as I remember from the outcome of the issue @xeraa linked, Zipkin should use a different delimiter when writing to Elasticsearch >= 7.0.

shakuzen commented 1 year ago

@ax-korneenkov that stacktrace is showing that an exception is being thrown by the collector, not the storage layer. It is failing to decode spans sent to the Zipkin collector. Make sure that spans are being sent in the correct format to the correct endpoint.

codefromthecrypt commented 9 months ago

I think @shakuzen was correct. maybe this is the wrong input data. in any case, if this were an issue it should be on the zipkin repo, which has been significantly updated recently