search

pinterest / secor

Secor is a service implementing Kafka log persistence
Apache License 2.0
1.84k stars 542 forks source link

Get exception when using micrometer #1732

Open richiesgr opened 3 years ago

richiesgr commented 3 years ago

Hi when I switch using com.pinterest.secor.monitoring.MicroMeterMetricCollector instead of com.pinterest.secor.monitoring.OstrichMetricCollector I get thousands of exception WARN Error while discarding collection, stopping java.util.ConcurrentModificationException at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909) at java.util.ArrayList$Itr.next(ArrayList.java:859) at io.micrometer.shaded.reactor.core.publisher.Operators.onDiscardMultiple(Operators.java:565) at io.micrometer.shaded.reactor.core.publisher.FluxBufferPredicate$BufferPredicateSubscriber.cancel(FluxBufferPredicate.java:174) at io.micrometer.shaded.reactor.core.publisher.FluxMap$MapSubscriber.cancel(FluxMap.java:160) at io.micrometer.shaded.reactor.core.publisher.FluxOnBackpressureLatest$LatestSubscriber.cancel(FluxOnBackpressureLatest.java:99) at io.micrometer.shaded.reactor.core.publisher.FluxMap$MapSubscriber.cancel(FluxMap.java:160) at io.micrometer.shaded.reactor.core.publisher.Operators.terminate(Operators.java:1222) at io.micrometer.shaded.reactor.netty.channel.MonoSendMany$SendManyInner.tryFailure(MonoSendMany.java:577) at io.micrometer.shaded.io.netty.util.internal.PromiseNotificationUtil.tryFailure(PromiseNotificationUtil.java:64) at io.micrometer.shaded.io.netty.channel.ChannelOutboundBuffer.safeFail(ChannelOutboundBuffer.java:723) at io.micrometer.shaded.io.netty.channel.ChannelOutboundBuffer.remove0(ChannelOutboundBuffer.java:308) at io.micrometer.shaded.io.netty.channel.ChannelOutboundBuffer.remove(ChannelOutboundBuffer.java:288) at io.micrometer.shaded.io.netty.channel.nio.AbstractNioMessageChannel.doWrite(AbstractNioMessageChannel.java:159) at io.micrometer.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:930) at io.micrometer.shaded.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:354) at io.micrometer.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:897) at io.micrometer.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1372) at io.micrometer.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750) at io.micrometer.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:742) at io.micrometer.shaded.io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:728) at io.micrometer.shaded.reactor.netty.channel.MonoSendMany$SendManyInner$AsyncFlush.run(MonoSendMany.java:758) at io.micrometer.shaded.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at io.micrometer.shaded.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at io.micrometer.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) at io.micrometer.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.micrometer.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.micrometer.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)

my config are secor.monitoring.metrics.collector.micrometer.jmx.enabled=true secor.monitoring.metrics.collector.micrometer.statsd.enabled=true secor.monitoring.metrics.collector.micrometer.prometheus.enabled=true

pdambrauskas commented 3 years ago

Looks like https://github.com/micrometer-metrics/micrometer/issues/2238

It is a bug in micrometer version < 1.5.5. Latest secor release uses 1.5.2. You can try to fix this issue by building secor from master (dependencies are updated there). Or disabling statsd and waiting for new release.

HenryCaiHaiying commented 3 years ago

The latest secor is using micrometer 1.6.1

On Tue, Dec 1, 2020 at 5:44 AM Paulius notifications@github.com wrote:

Looks like micrometer-metrics/micrometer#2238 https://github.com/micrometer-metrics/micrometer/issues/2238

It is a bug in micrometer version < 1.5.5. Latest secor release uses 1.5.2. You can try to fix this issue by building secor from master (dependencies are updated there). Or disabling statsd and waiting for new release.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/pinterest/secor/issues/1732#issuecomment-736560042, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABYJP73RPVKJKKEQWHY7PDDSSTXLXANCNFSM4UJAQLSA .

pdambrauskas commented 3 years ago

The latest secor is using micrometer 1.6.1

But release tagged with 0.29 uses 1.5.2 :) https://github.com/pinterest/secor/blob/v0.29/pom.xml#L381

HenryCaiHaiying commented 3 years ago

We can create a new tag as well if the code at head fixes the problem.

On Wed, Dec 2, 2020 at 12:45 AM Paulius notifications@github.com wrote:

The latest secor is using micrometer 1.6.1

But release tagged with 0.29 uses 1.5.2 :) https://github.com/pinterest/secor/blob/v0.29/pom.xml#L381

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/pinterest/secor/issues/1732#issuecomment-737082902, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABYJP72KYHGBXQLWFO3SFN3SSX5BJANCNFSM4UJAQLSA .