search

streamnative / kop

Kafka-on-Pulsar - A protocol handler that brings native Kafka protocol to Apache Pulsar
https://streamnative.io/docs/kop
Apache License 2.0
452 stars 137 forks source link

[BUG] KOP NoClassDefFoundError #954

Open xiaotongwang1 opened 2 years ago

xiaotongwang1 commented 2 years ago

ERROR LOG

2021-12-04 04:15:34.634 [pulsar-io-4-4] ERROR i.s.pulsar.handlers.kop.KafkaRequestHandler - Caught error in handler, closing channel java.lang.NoClassDefFoundError: io/streamnative/pulsar/handlers/kop/utils/delayed/DelayedOperationKey$GroupKey at io.streamnative.pulsar.handlers.kop.coordinator.group.GroupCoordinator.unsafeJoinGroup(GroupCoordinator.java:431) at io.streamnative.pulsar.handlers.kop.coordinator.group.GroupCoordinator.lambda$doJoinGroup$2(GroupCoordinator.java:283) at io.streamnative.pulsar.handlers.kop.utils.CoreUtils.inLock(CoreUtils.java:35) at io.streamnative.pulsar.handlers.kop.coordinator.group.GroupMetadata.inLock(GroupMetadata.java:214) at io.streamnative.pulsar.handlers.kop.coordinator.group.GroupCoordinator.doJoinGroup(GroupCoordinator.java:283) at io.streamnative.pulsar.handlers.kop.coordinator.group.GroupCoordinator.lambda$handleJoinGroup$0(GroupCoordinator.java:241) at java.util.Optional.map(Optional.java:215)

Jmap

jmap -histo:live 21832|head -n 350

num #instances #bytes class name

1: 5810429 558743592 [C 2: 2273772 202961352 [B 3: 5809544 139429056 java.lang.String 4: 1163294 103237968 [Ljava.util.HashMap$Node; 5: 1131206 81446832 io.streamnative.pulsar.handlers.kop.coordinator.group.MemberMetadata 6: 2295001 73440032 java.util.HashMap$Node 7: 1151242 55259616 java.util.HashMap 8: 1137261 27294264 java.util.concurrent.CompletableFuture 9: 293279 23570160 [J 10: 286409 18330176 io.netty.buffer.PoolSubpage 11: 1138567 18217072 java.util.HashMap$KeySet 12: 66827 10474960 [Ljava.lang.Object; 13: 83393 3335720 java.util.LinkedHashMap$Entry 14: 2294 2920288 [Lio.netty.util.Recycler$DefaultHandle; 15: 79600 2547200 java.util.concurrent.atomic.LongAdder 16: 8295 2226304 [Z 17: 2640 1774080 io.netty.util.internal.shaded.org.jctools.queues.MpscArrayQueue 18: 10714 1714240 org.apache.bookkeeper.mledger.impl.NonDurableCursorImpl 19: 15791 1663488 [I 20: 14810 1640248 java.lang.Class 21: 43026 1376832 java.util.concurrent.ConcurrentHashMap$Node 22: 14328 1260864 java.lang.reflect.Method 23: 176 1251840 [Lio.netty.buffer.PoolSubpage; 24: 16038 1026432 java.nio.DirectByteBuffer 25: 17852 999712 java.util.LinkedHashMap 26: 35681 856344 java.util.ArrayList 27: 33911 813864 java.util.concurrent.ConcurrentSkipListMap$Node 28: 12288 786432 org.apache.bookkeeper.util.collections.ConcurrentOpenHashMap$Section 29: 5828 752832 [Ljava.util.concurrent.ConcurrentHashMap$Node; 30: 10714 685696 com.google.common.util.concurrent.SmoothRateLimiter$SmoothBursty 31: 20946 670272 org.apache.bookkeeper.mledger.impl.PositionImpl 32: 12615 605520 java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync 33: 22872 548928 org.apache.bookkeeper.mledger.impl.ManagedCursorContainer$Item 34: 22134 531216 org.glassfish.jersey.internal.routing.CombinedMediaType$EffectiveMediaType

JVM config

-Xms2636M -Xmx2636M -XX:MaxDirectMemorySize=5272M

BewareMyPower commented 2 years ago

Did you download the binary from release page directly? I think you can decompress the NAR file and see if the class exists.

xiaotongwang1 commented 2 years ago

yes ,binary file

some ip :/tmp/pulsar-nar/pulsar-protocol-handler-kafka-2.8.1.2.nar-unpacked/wDKNcYYlZF_hUVFMDgMQpg/io/streamnative/pulsar/handlers/kop/utils/delayed

total 12 -rw------- 1 dmq dmq 1651 Nov 22 19:54 'DelayedOperationKey$GroupKey.class' -rw------- 1 dmq dmq 2147 Nov 22 19:54 'DelayedOperationKey$MemberKey.class' -rw------- 1 dmq dmq 1641 Nov 22 19:54 'DelayedOperationKey$TopicKey.class'

some ip : total 52 -rw------- 1 dmq dmq 2926 Dec 4 14:13 DelayedOperation.class -rw------- 1 dmq dmq 1651 Dec 4 14:13 'DelayedOperationKey$GroupKey.class' -rw------- 1 dmq dmq 2147 Dec 4 14:13 'DelayedOperationKey$MemberKey.class' -rw------- 1 dmq dmq 1641 Dec 4 14:13 'DelayedOperationKey$TopicKey.class' -rw------- 1 dmq dmq 2205 Dec 4 14:13 'DelayedOperationKey$TopicPartitionOperationKey.class' -rw------- 1 dmq dmq 684 Dec 4 14:13 DelayedOperationKey.class -rw------- 1 dmq dmq 1144 Dec 4 14:13 'DelayedOperationPurgatory$1.class' -rw------- 1 dmq dmq 3767 Dec 4 14:13 'DelayedOperationPurgatory$Builder.class' -rw------- 1 dmq dmq 3327 Dec 4 14:13 'DelayedOperationPurgatory$Watchers.class' -rw------- 1 dmq dmq 11822 Dec 4 14:13 DelayedOperationPurgatory.class -rw------- 1 dmq dmq 155 Dec 4 14:13 package-info.class

sikangliu commented 2 years ago

@xiaotongwang1 I also appeared a similar problem, please comment, you appear Java. Lang. NoClassDefFoundError error is how to solve? The error message is as follows:

2022-05-17 10:44:52.0421 [group-coordinator-executor-OrderedScheduler-0-0] ERROR io.streamnative.pulsar.handlers.kop.coordinator.group.GroupMetadataManager - Unknown exception caught when processing the received metadata message from topic public/kafka/consumer_offsets-partition-40 java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: io/streamnative/pulsar/handlers/kop/coordinator/group/GroupMetadata$CommitRecordMetadataAndOffset at io.streamnative.pulsar.handlers.kop.coordinator.group.GroupMetadataManager.processLoadedAndRemovedGroups(GroupMetadataManager.java:993) ~[USZiZHPdx5ceO9OkdZPoyQ/:?] at io.streamnative.pulsar.handlers.kop.coordinator.group.GroupMetadataManager.lambda$unsafeLoadNextMetadataMessage$54(GroupMetadataManager.java:839) ~[USZiZHPdx5ceO9OkdZPoyQ/:?] at io.streamnative.pulsar.handlers.kop.coordinator.group.GroupMetadataManager.lambda$unsafeLoadNextMetadataMessage$56(GroupMetadataManager.java:971) ~[USZiZHPdx5ceO9OkdZPoyQ/:?] at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) [?:1.8.0_192] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) [?:1.8.0_192] at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442) [?:1.8.0_192] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_192] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_192] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_192] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_192] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_192] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_192] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_192] Caused by: java.lang.NoClassDefFoundError: io/streamnative/pulsar/handlers/kop/coordinator/group/GroupMetadata$CommitRecordMetadataAndOffset ... 14 more Caused by: java.lang.ClassNotFoundException: io.streamnative.pulsar.handlers.kop.coordinator.group.GroupMetadata$CommitRecordMetadataAndOffset at java.net.URLClassLoader.findClass(URLClassLoader.java:382) ~[?:1.8.0_192] at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_192] at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_192]

sikangliu commented 2 years ago

Kop program automatically decompressed after the file： ip：/tmp/pulsar-nar/pulsar-protocol-handler-kafka-2.9.2.15.nar-unpacked/USZiZHPdx5ceO9OkdZPoyQ/io/streamnative/pulsar/handlers/kop/coordinator/group

ll

total 220 -rw-r--r-- 1 root root 2931 Apr 29 15:07 DelayedHeartbeat.class -rw-r--r-- 1 root root 2491 Apr 29 15:07 DelayedJoin.class -rw-r--r-- 1 root root 1681 Apr 29 15:07 GroupConfig.class -rw-r--r-- 1 root root 64371 Apr 29 15:07 GroupCoordinator.class -rw-r--r-- 1 root root 34741 Apr 29 15:07 GroupMetadata.class -rw-r--r-- 1 root root 3393 Apr 29 15:07 GroupMetadata$GroupSummary.class -rw-r--r-- 1 root root 85315 Apr 29 15:07 GroupMetadataManager.class -rw-r--r-- 1 root root 1385 Apr 29 15:07 GroupState.class -rw-r--r-- 1 root root 3767 Apr 29 15:07 InitialDelayedJoin.class -rw-r--r-- 1 root root 5478 Apr 29 15:07 OffsetConfig.class -rw-r--r-- 1 root root 3450 Apr 29 15:07 OffsetConfig$OffsetConfigBuilder.class

sikangliu commented 2 years ago

Solution: Check whether related classes are missing in the /tmp directory. If yes, delete pulsar-nar in /tmp. Then restart the broker and the nar package is decompressed again.