search

apache / pulsar

Apache Pulsar - distributed pub-sub messaging system
https://pulsar.apache.org/
Apache License 2.0
14.13k stars 3.57k forks source link

[Bug] Connector cannot access methods from pulsar-client-original #20869

Open BewareMyPower opened 1 year ago

BewareMyPower commented 1 year ago

Search before asking

Version

Minimal reproduce step

See https://github.com/BewareMyPower/pulsar-sink-demo

What did you expect to see?

It should succeed.

What did you see instead?

ClassCastException

Anything else?

From https://github.com/apache/pulsar/pull/17835, it seems that we need to use reflection to access methods and classes in the module. But it harms the user experience.

The root problem is that connectors should not bundle the Pulsar client impl because it's provided at runtime by the Java Functions framework.

From the PR description, it seems to be related to how the Java Functions framework loads the connector. Ideally, it should use the default classpath of Pulsar broker.

Are you willing to submit a PR?

tisonkun commented 1 year ago

Can reproduce on my local env. This is a X is not X class loading issue -

java.lang.ClassCastException: class org.apache.pulsar.client.impl.MessageIdImpl cannot be cast to class org.apache.pulsar.client.impl.MessageIdImpl (org.apache.pulsar.client.impl.MessageIdImpl is in unnamed module of loader java.net.URLClassLoader @148080bb; org.apache.pulsar.client.impl.MessageIdImpl is in unnamed module of loader org.apache.pulsar.common.nar.NarClassLoader @2416a51)
    at org.example.DemoSink.lambda$write$0(DemoSink.java:25) ~[OjpVVcIuC05Ba5ELqz3aLA/:?]
    at java.util.Optional.ifPresent(Optional.java:178) ~[?:?]
    at org.example.DemoSink.write(DemoSink.java:22) ~[OjpVVcIuC05Ba5ELqz3aLA/:?]
    at org.apache.pulsar.functions.instance.JavaInstanceRunnable.sendOutputMessage(JavaInstanceRunnable.java:439) ~[?:?]
    at org.apache.pulsar.functions.instance.JavaInstanceRunnable.handleResult(JavaInstanceRunnable.java:401) ~[?:?]
    at org.apache.pulsar.functions.instance.JavaInstanceRunnable.run(JavaInstanceRunnable.java:341) ~[?:?]
    at java.lang.Thread.run(Thread.java:833) ~[?:?]

I'll try to investigate it.

tisonkun commented 1 year ago

Logging how NarClassLoader load classes. It loads -

- loading org.example.DemoSink
- loading org.apache.pulsar.client.impl.MessageIdImpl
- loading org.apache.pulsar.client.api.MessageIdAdv

But not MessageId. So the instance is of MessageId loaded by the URLClassLoader while we try to cast it to MessageIdImpl loaded by NarClassLoader.

We set the NarClassLoader as context class loader before handling result -

                Thread.currentThread().setContextClassLoader(instanceClassLoader);

                // register end time
                stats.processTimeEnd();

                if (result != null) {
                    // process the synchronous results
                    handleResult(currentRecord, result);
                }
tisonkun commented 1 year ago

Workaround with a hack internal -

diff --git a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/JavaInstanceRunnable.java b/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/JavaInstanceRunnable.java
index c3f36f754d..cc4bd4d7d2 100644
--- a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/JavaInstanceRunnable.java
+++ b/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/JavaInstanceRunnable.java
@@ -211,6 +211,8 @@ public class JavaInstanceRunnable implements AutoCloseable, Runnable {
         this.collectorRegistry = collectorRegistry;

         this.instanceClassLoader = Thread.currentThread().getContextClassLoader();
+        log.info("componentClassLoader: {}", this.componentClassLoader);
+        log.info("instanceClassLoader: {}", this.instanceClassLoader);
     }

     /**
diff --git a/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/JavaInstanceStarter.java b/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/JavaInstanceStarter.java
index 89281a2f55..c47c2bf9fa 100644
--- a/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/JavaInstanceStarter.java
+++ b/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/JavaInstanceStarter.java
@@ -195,7 +195,7 @@ public class JavaInstanceStarter implements AutoCloseable {
             functionDetailsJsonString = functionDetailsJsonString.substring(0, functionDetailsJsonString.length() - 1);
         }
         JsonFormat.parser().merge(functionDetailsJsonString, functionDetailsBuilder);
-        FunctionCacheManager fnCache = new FunctionCacheManagerImpl(rootClassLoader);
+        FunctionCacheManager fnCache = new FunctionCacheManagerImpl(functionInstanceClassLoader);
         ClassLoader functionClassLoader = ThreadRuntime.loadJars(jarFile, instanceConfig, functionId,
                 functionDetailsBuilder.getName(), narExtractionDirectory, fnCache);
         inferringMissingTypeClassName(functionDetailsBuilder, functionClassLoader);

The root cause is that the Record passed to sink.write is loaded by functionInstanceClassLoader and thus its internal MessageIdImpl. The class loader is URLClassLoader loading pulsar.functions.instance.classpath.

... while the sink class is loaded by NarClassLoader whose parent is "root classloader" a.k.a. the default AppClassLoader.

For more details,

functionInstanceClassLoader =

classpath /Users/tison/Brittani/pulsar/conf:/Users/tison/Brittani/pulsar/pulsar-broker/target/pulsar-functions-api-examples.jar::/Users/tison/Brittani/pulsar/pulsar-broker/target/pulsar-broker.jar:/Users/tison/.m2/repository/commons-codec/commons-codec/1.15/commons-codec-1.15.jar:/Users/tison/.m2/repository/org/apache/commons/commons-collections4/4.4/commons-collections4-4.4.jar:/Users/tison/.m2/repository/org/apache/commons/commons-lang3/3.11/commons-lang3-3.11.jar:/Users/tison/.m2/repository/org/slf4j/slf4j-api/1.7.32/slf4j-api-1.7.32.jar:/Users/tison/.m2/repository/io/netty/netty-transport/4.1.94.Final/netty-transport-4.1.94.Final.jar:/Users/tison/.m2/repository/com/google/protobuf/protobuf-java/3.19.6/protobuf-java-3.19.6.jar:/Users/tison/Brittani/pulsar/pulsar-client/target/pulsar-client-original.jar:/Users/tison/Brittani/pulsar/pulsar-client-api/target/pulsar-client-api.jar:/Users/tison/.m2/repository/io/netty/netty-codec-socks/4.1.94.Final/netty-codec-socks-4.1.94.Final.jar:/Users/tison/.m2/repository/io/netty/netty-resolver-dns-native-macos/4.1.94.Final/netty-resolver-dns-native-macos-4.1.94.Final-osx-aarch_64.jar:/Users/tison/.m2/repository/io/netty/netty-resolver-dns-classes-macos/4.1.94.Final/netty-resolver-dns-classes-macos-4.1.94.Final.jar:/Users/tison/.m2/repository/io/netty/netty-resolver-dns-native-macos/4.1.94.Final/netty-resolver-dns-native-macos-4.1.94.Final-osx-x86_64.jar:/Users/tison/.m2/repository/org/apache/avro/avro-protobuf/1.10.2/avro-protobuf-1.10.2.jar:/Users/tison/.m2/repository/net/jcip/jcip-annotations/1.0/jcip-annotations-1.0.jar:/Users/tison/Brittani/pulsar/pulsar-websocket/target/pulsar-websocket.jar:/Users/tison/.m2/repository/org/eclipse/jetty/websocket/websocket-api/9.4.51.v20230217/websocket-api-9.4.51.v20230217.jar:/Users/tison/.m2/repository/org/eclipse/jetty/websocket/websocket-server/9.4.51.v20230217/websocket-server-9.4.51.v20230217.jar:/Users/tison/.m2/repository/org/eclipse/jetty/websocket/websocket-common/9.4.51.v20230217/websocket-common-9.4.51.v20230217.jar:/Users/tison/.m2/repository/org/eclipse/jetty/websocket/websocket-servlet/9.4.51.v20230217/websocket-servlet-9.4.51.v20230217.jar:/Users/tison/Brittani/pulsar/pulsar-client-admin/target/pulsar-client-admin-original.jar:/Users/tison/Brittani/pulsar/managed-ledger/target/managed-ledger.jar:/Users/tison/.m2/repository/org/apache/bookkeeper/stats/codahale-metrics-provider/4.16.2/codahale-metrics-provider-4.16.2.jar:/Users/tison/.m2/repository/io/dropwizard/metrics/metrics-jmx/4.1.12.1/metrics-jmx-4.1.12.1.jar:/Users/tison/Brittani/pulsar/pulsar-metadata/target/pulsar-metadata.jar:/Users/tison/.m2/repository/io/etcd/jetcd-core/0.7.5/jetcd-core-0.7.5.jar:/Users/tison/.m2/repository/io/etcd/jetcd-grpc/0.7.5/jetcd-grpc-0.7.5.jar:/Users/tison/.m2/repository/io/vertx/vertx-grpc/4.3.5/vertx-grpc-4.3.5.jar:/Users/tison/.m2/repository/io/etcd/jetcd-api/0.7.5/jetcd-api-0.7.5.jar:/Users/tison/.m2/repository/io/etcd/jetcd-common/0.7.5/jetcd-common-0.7.5.jar:/Users/tison/.m2/repository/net/jodah/failsafe/2.4.4/failsafe-2.4.4.jar:/Users/tison/.m2/repository/org/apache/curator/curator-recipes/5.1.0/curator-recipes-5.1.0.jar:/Users/tison/.m2/repository/org/apache/curator/curator-framework/5.1.0/curator-framework-5.1.0.jar:/Users/tison/.m2/repository/org/apache/curator/curator-client/5.1.0/curator-client-5.1.0.jar:/Users/tison/.m2/repository/org/apache/bookkeeper/stream-storage-server/4.16.2/stream-storage-server-4.16.2.jar:/Users/tison/.m2/repository/org/apache/bookkeeper/stream-storage-service-impl/4.16.2/stream-storage-service-impl-4.16.2.jar:/Users/tison/.m2/repository/org/apache/bookkeeper/stream-storage-service-api/4.16.2/stream-storage-service-api-4.16.2.jar:/Users/tison/.m2/repository/org/apache/bookkeeper/stream-storage-proto/4.16.2/stream-storage-proto-4.16.2.jar:/Users/tison/.m2/repository/org/apache/bookkeeper/stream-storage-java-client-base/4.16.2/stream-storage-java-client-base-4.16.2.jar:/Users/tison/.m2/repository/org/apache/bookkeeper/stream-storage-api/4.16.2/stream-storage-api-4.16.2.jar:/Users/tison/.m2/repository/org/apache/bookkeeper/statelib/4.16.2/statelib-4.16.2.jar:/Users/tison/.m2/repository/org/apache/bookkeeper/stream-storage-common/4.16.2/stream-storage-common-4.16.2.jar:/Users/tison/.m2/repository/org/apache/distributedlog/distributedlog-core/4.16.2/distributedlog-core-4.16.2-tests.jar:/Users/tison/.m2/repository/org/apache/bookkeeper/stream-storage-java-client/4.16.2/stream-storage-java-client-4.16.2.jar:/Users/tison/.m2/repository/com/google/auth/google-auth-library-oauth2-http/1.4.0/google-auth-library-oauth2-http-1.4.0.jar:/Users/tison/.m2/repository/com/google/http-client/google-http-client/1.41.0/google-http-client-1.41.0.jar:/Users/tison/.m2/repository/io/opencensus/opencensus-api/0.28.0/opencensus-api-0.28.0.jar:/Users/tison/.m2/repository/io/opencensus/opencensus-contrib-http-util/0.28.0/opencensus-contrib-http-util-0.28.0.jar:/Users/tison/.m2/repository/com/google/http-client/google-http-client-gson/1.41.0/google-http-client-gson-1.41.0.jar:/Users/tison/.m2/repository/org/apache/bookkeeper/bookkeeper-tools-framework/4.16.2/bookkeeper-tools-framework-4.16.2.jar:/Users/tison/Brittani/pulsar/pulsar-broker-common/target/pulsar-broker-common.jar:/Users/tison/.m2/repository/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar:/Users/tison/.m2/repository/io/jsonwebtoken/jjwt-impl/0.11.1/jjwt-impl-0.11.1.jar:/Users/tison/.m2/repository/io/jsonwebtoken/jjwt-api/0.11.1/jjwt-api-0.11.1.jar:/Users/tison/.m2/repository/io/jsonwebtoken/jjwt-jackson/0.11.1/jjwt-jackson-0.11.1.jar:/Users/tison/Brittani/pulsar/pulsar-transaction/common/target/pulsar-transaction-common.jar:/Users/tison/Brittani/pulsar/pulsar-client-messagecrypto-bc/target/pulsar-client-messagecrypto-bc.jar:/Users/tison/.m2/repository/org/eclipse/jetty/jetty-server/9.4.51.v20230217/jetty-server-9.4.51.v20230217.jar:/Users/tison/.m2/repository/org/eclipse/jetty/jetty-http/9.4.51.v20230217/jetty-http-9.4.51.v20230217.jar:/Users/tison/.m2/repository/org/eclipse/jetty/jetty-io/9.4.51.v20230217/jetty-io-9.4.51.v20230217.jar:/Users/tison/.m2/repository/org/eclipse/jetty/jetty-alpn-conscrypt-server/9.4.51.v20230217/jetty-alpn-conscrypt-server-9.4.51.v20230217.jar:/Users/tison/.m2/repository/org/eclipse/jetty/jetty-alpn-server/9.4.51.v20230217/jetty-alpn-server-9.4.51.v20230217.jar:/Users/tison/.m2/repository/org/eclipse/jetty/jetty-servlet/9.4.51.v20230217/jetty-servlet-9.4.51.v20230217.jar:/Users/tison/.m2/repository/org/eclipse/jetty/jetty-security/9.4.51.v20230217/jetty-security-9.4.51.v20230217.jar:/Users/tison/.m2/repository/org/eclipse/jetty/jetty-util-ajax/9.4.51.v20230217/jetty-util-ajax-9.4.51.v20230217.jar:/Users/tison/.m2/repository/org/eclipse/jetty/jetty-servlets/9.4.51.v20230217/jetty-servlets-9.4.51.v20230217.jar:/Users/tison/.m2/repository/org/eclipse/jetty/jetty-continuation/9.4.51.v20230217/jetty-continuation-9.4.51.v20230217.jar:/Users/tison/.m2/repository/org/eclipse/jetty/jetty-util/9.4.51.v20230217/jetty-util-9.4.51.v20230217.jar:/Users/tison/.m2/repository/org/glassfish/jersey/core/jersey-server/2.34/jersey-server-2.34.jar:/Users/tison/.m2/repository/org/glassfish/jersey/core/jersey-common/2.34/jersey-common-2.34.jar:/Users/tison/.m2/repository/org/glassfish/hk2/osgi-resource-locator/1.0.3/osgi-resource-locator-1.0.3.jar:/Users/tison/.m2/repository/jakarta/ws/rs/jakarta.ws.rs-api/2.1.6/jakarta.ws.rs-api-2.1.6.jar:/Users/tison/.m2/repository/jakarta/annotation/jakarta.annotation-api/1.3.5/jakarta.annotation-api-1.3.5.jar:/Users/tison/.m2/repository/org/glassfish/hk2/external/jakarta.inject/2.6.1/jakarta.inject-2.6.1.jar:/Users/tison/.m2/repository/jakarta/validation/jakarta.validation-api/2.0.2/jakarta.validation-api-2.0.2.jar:/Users/tison/.m2/repository/org/glassfish/jersey/containers/jersey-container-servlet-core/2.34/jersey-container-servlet-core-2.34.jar:/Users/tison/.m2/repository/org/glassfish/jersey/containers/jersey-container-servlet/2.34/jersey-container-servlet-2.34.jar:/Users/tison/.m2/repository/org/glassfish/jersey/media/jersey-media-json-jackson/2.34/jersey-media-json-jackson-2.34.jar:/Users/tison/.m2/repository/org/glassfish/jersey/ext/jersey-entity-filtering/2.34/jersey-entity-filtering-2.34.jar:/Users/tison/.m2/repository/com/fasterxml/jackson/core/jackson-annotations/2.14.2/jackson-annotations-2.14.2.jar:/Users/tison/.m2/repository/com/fasterxml/jackson/module/jackson-module-jaxb-annotations/2.14.2/jackson-module-jaxb-annotations-2.14.2.jar:/Users/tison/.m2/repository/jakarta/xml/bind/jakarta.xml.bind-api/2.3.3/jakarta.xml.bind-api-2.3.3.jar:/Users/tison/.m2/repository/jakarta/activation/jakarta.activation-api/1.2.2/jakarta.activation-api-1.2.2.jar:/Users/tison/.m2/repository/com/fasterxml/jackson/jaxrs/jackson-jaxrs-json-provider/2.14.2/jackson-jaxrs-json-provider-2.14.2.jar:/Users/tison/.m2/repository/com/fasterxml/jackson/jaxrs/jackson-jaxrs-base/2.14.2/jackson-jaxrs-base-2.14.2.jar:/Users/tison/.m2/repository/org/glassfish/jersey/inject/jersey-hk2/2.34/jersey-hk2-2.34.jar:/Users/tison/.m2/repository/org/glassfish/hk2/hk2-locator/2.6.1/hk2-locator-2.6.1.jar:/Users/tison/.m2/repository/org/glassfish/hk2/external/aopalliance-repackaged/2.6.1/aopalliance-repackaged-2.6.1.jar:/Users/tison/.m2/repository/org/glassfish/hk2/hk2-api/2.6.1/hk2-api-2.6.1.jar:/Users/tison/.m2/repository/org/glassfish/hk2/hk2-utils/2.6.1/hk2-utils-2.6.1.jar:/Users/tison/.m2/repository/com/fasterxml/jackson/module/jackson-module-jsonSchema/2.14.2/jackson-module-jsonSchema-2.14.2.jar:/Users/tison/.m2/repository/javax/validation/validation-api/1.1.0.Final/validation-api-1.1.0.Final.jar:/Users/tison/.m2/repository/org/slf4j/jcl-over-slf4j/1.7.32/jcl-over-slf4j-1.7.32.jar:/Users/tison/.m2/repository/com/google/guava/guava/32.1.1-jre/guava-32.1.1-jre.jar:/Users/tison/.m2/repository/com/google/guava/failureaccess/1.0.1/failureaccess-1.0.1.jar:/Users/tison/.m2/repository/com/google/guava/listenablefuture/9999.0-empty-to-avoid-conflict-with-guava/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/Users/tison/.m2/repository/org/checkerframework/checker-qual/3.33.0/checker-qual-3.33.0.jar:/Users/tison/.m2/repository/com/google/errorprone/error_prone_annotations/2.5.1/error_prone_annotations-2.5.1.jar:/Users/tison/.m2/repository/com/google/j2objc/j2objc-annotations/1.3/j2objc-annotations-1.3.jar:/Users/tison/.m2/repository/com/beust/jcommander/1.82/jcommander-1.82.jar:/Users/tison/.m2/repository/io/prometheus/simpleclient/0.16.0/simpleclient-0.16.0.jar:/Users/tison/.m2/repository/io/prometheus/simpleclient_tracer_otel/0.16.0/simpleclient_tracer_otel-0.16.0.jar:/Users/tison/.m2/repository/io/prometheus/simpleclient_tracer_common/0.16.0/simpleclient_tracer_common-0.16.0.jar:/Users/tison/.m2/repository/io/prometheus/simpleclient_tracer_otel_agent/0.16.0/simpleclient_tracer_otel_agent-0.16.0.jar:/Users/tison/.m2/repository/io/prometheus/simpleclient_jetty/0.16.0/simpleclient_jetty-0.16.0.jar:/Users/tison/.m2/repository/io/prometheus/simpleclient_hotspot/0.16.0/simpleclient_hotspot-0.16.0.jar:/Users/tison/.m2/repository/io/prometheus/simpleclient_caffeine/0.16.0/simpleclient_caffeine-0.16.0.jar:/Users/tison/.m2/repository/org/hdrhistogram/HdrHistogram/2.1.9/HdrHistogram-2.1.9.jar:/Users/tison/.m2/repository/com/google/code/gson/gson/2.8.9/gson-2.8.9.jar:/Users/tison/.m2/repository/com/github/zafarkhaja/java-semver/0.9.0/java-semver-0.9.0.jar:/Users/tison/.m2/repository/org/apache/avro/avro/1.10.2/avro-1.10.2.jar:/Users/tison/.m2/repository/org/apache/commons/commons-compress/1.21/commons-compress-1.21.jar:/Users/tison/.m2/repository/com/carrotsearch/hppc/0.9.1/hppc-0.9.1.jar:/Users/tison/.m2/repository/org/roaringbitmap/RoaringBitmap/0.9.44/RoaringBitmap-0.9.44.jar:/Users/tison/.m2/repository/org/roaringbitmap/shims/0.9.44/shims-0.9.44.jar:/Users/tison/.m2/repository/com/github/oshi/oshi-core-java11/6.4.0/oshi-core-java11-6.4.0.jar:/Users/tison/.m2/repository/net/java/dev/jna/jna-jpms/5.12.1/jna-jpms-5.12.1.jar:/Users/tison/.m2/repository/net/java/dev/jna/jna-platform-jpms/5.12.1/jna-platform-jpms-5.12.1.jar:/Users/tison/.m2/repository/javax/xml/bind/jaxb-api/2.3.1/jaxb-api-2.3.1.jar:/Users/tison/.m2/repository/com/sun/activation/javax.activation/1.2.0/javax.activation-1.2.0.jar:/Users/tison/Brittani/pulsar/pulsar-transaction/coordinator/target/pulsar-transaction-coordinator.jar:/Users/tison/Brittani/pulsar/pulsar-package-management/core/target/pulsar-package-core.jar:/Users/tison/Brittani/pulsar/pulsar-docs-tools/target/pulsar-docs-tools.jar:/Users/tison/.m2/repository/io/swagger/swagger-annotations/1.6.2/swagger-annotations-1.6.2.jar:/Users/tison/.m2/repository/io/swagger/swagger-core/1.6.2/swagger-core-1.6.2.jar:/Users/tison/.m2/repository/io/swagger/swagger-models/1.6.2/swagger-models-1.6.2.jar:/Users/tison/Brittani/pulsar/pulsar-proxy/target/pulsar-proxy.jar:/Users/tison/Brittani/pulsar/pulsar-common/target/pulsar-common.jar:/Users/tison/.m2/repository/com/fasterxml/jackson/module/jackson-module-parameter-names/2.14.2/jackson-module-parameter-names-2.14.2.jar:/Users/tison/.m2/repository/com/fasterxml/jackson/datatype/jackson-datatype-jsr310/2.14.2/jackson-datatype-jsr310-2.14.2.jar:/Users/tison/.m2/repository/com/fasterxml/jackson/datatype/jackson-datatype-jdk8/2.14.2/jackson-datatype-jdk8-2.14.2.jar:/Users/tison/.m2/repository/io/netty/netty-transport-native-unix-common/4.1.94.Final/netty-transport-native-unix-common-4.1.94.Final-linux-x86_64.jar:/Users/tison/.m2/repository/org/apache/bookkeeper/cpu-affinity/4.16.2/cpu-affinity-4.16.2.jar:/Users/tison/.m2/repository/io/airlift/aircompressor/0.20/aircompressor-0.20.jar:/Users/tison/.m2/repository/io/netty/incubator/netty-incubator-transport-classes-io_uring/0.0.21.Final/netty-incubator-transport-classes-io_uring-0.0.21.Final.jar:/Users/tison/.m2/repository/io/netty/incubator/netty-incubator-transport-native-io_uring/0.0.21.Final/netty-incubator-transport-native-io_uring-0.0.21.Final-linux-x86_64.jar:/Users/tison/.m2/repository/io/netty/incubator/netty-incubator-transport-native-io_uring/0.0.21.Final/netty-incubator-transport-native-io_uring-0.0.21.Final-linux-aarch_64.jar:/Users/tison/.m2/repository/io/netty/netty-codec-haproxy/4.1.94.Final/netty-codec-haproxy-4.1.94.Final.jar:/Users/tison/.m2/repository/org/eclipse/jetty/jetty-proxy/9.4.51.v20230217/jetty-proxy-9.4.51.v20230217.jar:/Users/tison/.m2/repository/org/eclipse/jetty/jetty-client/9.4.51.v20230217/jetty-client-9.4.51.v20230217.jar:/Users/tison/.m2/repository/io/prometheus/simpleclient_servlet/0.16.0/simpleclient_servlet-0.16.0.jar:/Users/tison/.m2/repository/io/prometheus/simpleclient_common/0.16.0/simpleclient_common-0.16.0.jar:/Users/tison/.m2/repository/io/prometheus/simpleclient_servlet_common/0.16.0/simpleclient_servlet_common-0.16.0.jar:/Users/tison/.m2/repository/com/github/seancfoley/ipaddress/5.3.3/ipaddress-5.3.3.jar:/Users/tison/Brittani/pulsar/pulsar-broker-auth-oidc/target/pulsar-broker-auth-oidc.jar:/Users/tison/.m2/repository/com/auth0/java-jwt/4.3.0/java-jwt-4.3.0.jar:/Users/tison/.m2/repository/com/auth0/jwks-rsa/0.22.0/jwks-rsa-0.22.0.jar:/Users/tison/.m2/repository/com/github/ben-manes/caffeine/caffeine/2.9.1/caffeine-2.9.1.jar:/Users/tison/.m2/repository/org/asynchttpclient/async-http-client/2.12.1/async-http-client-2.12.1.jar:/Users/tison/.m2/repository/org/asynchttpclient/async-http-client-netty-utils/2.12.1/async-http-client-netty-utils-2.12.1.jar:/Users/tison/.m2/repository/org/reactivestreams/reactive-streams/1.0.3/reactive-streams-1.0.3.jar:/Users/tison/.m2/repository/io/kubernetes/client-java/18.0.0/client-java-18.0.0.jar:/Users/tison/.m2/repository/io/kubernetes/client-java-api/18.0.0/client-java-api-18.0.0.jar:/Users/tison/.m2/repository/com/squareup/okhttp3/okhttp/4.9.3/okhttp-4.9.3.jar:/Users/tison/.m2/repository/com/squareup/okio/okio/2.8.0/okio-2.8.0.jar:/Users/tison/.m2/repository/org/jetbrains/kotlin/kotlin-stdlib-common/1.8.20/kotlin-stdlib-common-1.8.20.jar:/Users/tison/.m2/repository/org/jetbrains/kotlin/kotlin-stdlib/1.8.20/kotlin-stdlib-1.8.20.jar:/Users/tison/.m2/repository/org/jetbrains/annotations/13.0/annotations-13.0.jar:/Users/tison/.m2/repository/com/squareup/okhttp3/logging-interceptor/4.9.3/logging-interceptor-4.9.3.jar:/Users/tison/.m2/repository/org/jetbrains/kotlin/kotlin-stdlib-jdk8/1.8.20/kotlin-stdlib-jdk8-1.8.20.jar:/Users/tison/.m2/repository/org/jetbrains/kotlin/kotlin-stdlib-jdk7/1.8.20/kotlin-stdlib-jdk7-1.8.20.jar:/Users/tison/.m2/repository/io/gsonfire/gson-fire/1.8.5/gson-fire-1.8.5.jar:/Users/tison/.m2/repository/io/kubernetes/client-java-proto/18.0.0/client-java-proto-18.0.0.jar:/Users/tison/.m2/repository/org/bitbucket/b_c/jose4j/0.9.3/jose4j-0.9.3.jar:/Users/tison/Brittani/pulsar/pulsar-broker-auth-sasl/target/pulsar-broker-auth-sasl.jar:/Users/tison/Brittani/pulsar/pulsar-client-auth-sasl/target/pulsar-client-auth-sasl.jar:/Users/tison/.m2/repository/javax/ws/rs/javax.ws.rs-api/2.1/javax.ws.rs-api-2.1.jar:/Users/tison/.m2/repository/org/glassfish/jersey/core/jersey-client/2.34/jersey-client-2.34.jar:/Users/tison/.m2/repository/jline/jline/2.14.6/jline-2.14.6.jar:/Users/tison/.m2/repository/org/apache/zookeeper/zookeeper-prometheus-metrics/3.8.1/zookeeper-prometheus-metrics-3.8.1.jar:/Users/tison/.m2/repository/org/apache/zookeeper/zookeeper/3.8.1/zookeeper-3.8.1.jar:/Users/tison/.m2/repository/org/apache/zookeeper/zookeeper-jute/3.8.1/zookeeper-jute-3.8.1.jar:/Users/tison/.m2/repository/org/apache/yetus/audience-annotations/0.12.0/audience-annotations-0.12.0.jar:/Users/tison/.m2/repository/io/netty/netty-transport-native-epoll/4.1.94.Final/netty-transport-native-epoll-4.1.94.Final.jar:/Users/tison/Brittani/pulsar/pulsar-package-management/bookkeeper-storage/target/pulsar-package-bookkeeper-storage.jar:/Users/tison/.m2/repository/org/apache/distributedlog/distributedlog-core/4.16.2/distributedlog-core-4.16.2.jar:/Users/tison/.m2/repository/org/apache/distributedlog/distributedlog-protocol/4.16.2/distributedlog-protocol-4.16.2.jar:/Users/tison/.m2/repository/org/apache/distributedlog/distributedlog-common/4.16.2/distributedlog-common-4.16.2.jar:/Users/tison/.m2/repository/org/apache/thrift/libthrift/0.14.2/libthrift-0.14.2.jar:/Users/tison/Brittani/pulsar/testmocks/target/testmocks.jar:/Users/tison/Brittani/pulsar/pulsar-package-management/filesystem-storage/target/pulsar-package-filesystem-storage.jar:/Users/tison/Brittani/pulsar/pulsar-client-tools/target/pulsar-client-tools.jar:/Users/tison/Brittani/pulsar/pulsar-client-admin-api/target/pulsar-client-admin-api.jar:/Users/tison/Brittani/pulsar/pulsar-client-tools-api/target/pulsar-client-tools-api.jar:/Users/tison/.m2/repository/commons-io/commons-io/2.8.0/commons-io-2.8.0.jar:/Users/tison/.m2/repository/com/typesafe/netty/netty-reactive-streams/2.0.6/netty-reactive-streams-2.0.6.jar:/Users/tison/.m2/repository/org/conscrypt/conscrypt-openjdk-uber/2.5.2/conscrypt-openjdk-uber-2.5.2.jar:/Users/tison/.m2/repository/org/javassist/javassist/3.25.0-GA/javassist-3.25.0-GA.jar:/Users/tison/.m2/repository/org/eclipse/jetty/websocket/javax-websocket-client-impl/9.4.51.v20230217/javax-websocket-client-impl-9.4.51.v20230217.jar:/Users/tison/.m2/repository/org/eclipse/jetty/websocket/websocket-client/9.4.51.v20230217/websocket-client-9.4.51.v20230217.jar:/Users/tison/.m2/repository/javax/websocket/javax.websocket-client-api/1.0/javax.websocket-client-api-1.0.jar:/Users/tison/.m2/repository/org/jline/jline/3.21.0/jline-3.21.0.jar:/Users/tison/.m2/repository/net/java/dev/jna/jna/5.12.1/jna-5.12.1.jar:/Users/tison/.m2/repository/org/apache/commons/commons-text/1.10.0/commons-text-1.10.0.jar:/Users/tison/Brittani/pulsar/pulsar-testclient/target/pulsar-testclient.jar:/Users/tison/.m2/repository/commons-configuration/commons-configuration/1.10/commons-configuration-1.10.jar:/Users/tison/.m2/repository/commons-lang/commons-lang/2.6/commons-lang-2.6.jar:/Users/tison/.m2/repository/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar:/Users/tison/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.14.2/jackson-databind-2.14.2.jar:/Users/tison/.m2/repository/org/apache/logging/log4j/log4j-api/2.18.0/log4j-api-2.18.0.jar:/Users/tison/.m2/repository/org/apache/logging/log4j/log4j-core/2.18.0/log4j-core-2.18.0.jar:/Users/tison/.m2/repository/org/apache/logging/log4j/log4j-web/2.18.0/log4j-web-2.18.0.jar:/Users/tison/.m2/repository/io/dropwizard/metrics/metrics-core/4.1.12.1/metrics-core-4.1.12.1.jar:/Users/tison/.m2/repository/io/dropwizard/metrics/metrics-graphite/4.1.12.1/metrics-graphite-4.1.12.1.jar:/Users/tison/.m2/repository/com/rabbitmq/amqp-client/5.5.3/amqp-client-5.5.3.jar:/Users/tison/.m2/repository/io/dropwizard/metrics/metrics-jvm/4.1.12.1/metrics-jvm-4.1.12.1.jar:/Users/tison/.m2/repository/org/xerial/snappy/snappy-java/1.1.10.1/snappy-java-1.1.10.1.jar:/Users/tison/.m2/repository/com/fasterxml/jackson/dataformat/jackson-dataformat-yaml/2.14.2/jackson-dataformat-yaml-2.14.2.jar:/Users/tison/.m2/repository/org/yaml/snakeyaml/2.0/snakeyaml-2.0.jar:/Users/tison/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.14.2/jackson-core-2.14.2.jar:/Users/tison/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.18.0/log4j-slf4j-impl-2.18.0.jar:/Users/tison/.m2/repository/org/apache/bookkeeper/stats/prometheus-metrics-provider/4.16.2/prometheus-metrics-provider-4.16.2.jar:/Users/tison/.m2/repository/org/apache/bookkeeper/stats/bookkeeper-stats-api/4.16.2/bookkeeper-stats-api-4.16.2.jar:/Users/tison/.m2/repository/io/netty/netty-common/4.1.94.Final/netty-common-4.1.94.Final.jar:/Users/tison/.m2/repository/com/yahoo/datasketches/sketches-core/0.8.3/sketches-core-0.8.3.jar:/Users/tison/.m2/repository/com/yahoo/datasketches/memory/0.8.3/memory-0.8.3.jar:/Users/tison/.m2/repository/io/prometheus/simpleclient_log4j2/0.16.0/simpleclient_log4j2-0.16.0.jar:/Users/tison/Brittani/pulsar/bouncy-castle/bc/target/bouncy-castle-bc-3.1.0-SNAPSHOT-pkg.jar:/Users/tison/.m2/repository/org/bouncycastle/bcprov-ext-jdk18on/1.75/bcprov-ext-jdk18on-1.75.jar:/Users/tison/Brittani/pulsar/pulsar-functions/worker/target/pulsar-functions-worker.jar:/Users/tison/Brittani/pulsar/pulsar-functions/runtime/target/pulsar-functions-runtime.jar:/Users/tison/Brittani/pulsar/pulsar-functions/instance/target/pulsar-functions-instance.jar:/Users/tison/Brittani/pulsar/pulsar-functions/utils/target/pulsar-functions-utils.jar:/Users/tison/Brittani/pulsar/pulsar-config-validation/target/pulsar-config-validation.jar:/Users/tison/Brittani/pulsar/pulsar-io/core/target/pulsar-io-core.jar:/Users/tison/Brittani/pulsar/pulsar-functions/api-java/target/pulsar-functions-api.jar:/Users/tison/.m2/repository/net/jodah/typetools/0.5.0/typetools-0.5.0.jar:/Users/tison/.m2/repository/io/prometheus/simpleclient_httpserver/0.16.0/simpleclient_httpserver-0.16.0.jar:/Users/tison/.m2/repository/io/prometheus/jmx/collector/0.16.1/collector-0.16.1.jar:/Users/tison/Brittani/pulsar/pulsar-functions/secrets/target/pulsar-functions-secrets.jar:/Users/tison/Brittani/pulsar/pulsar-functions/proto/target/pulsar-functions-proto.jar:/Users/tison/.m2/repository/org/glassfish/jersey/media/jersey-media-multipart/2.34/jersey-media-multipart-2.34.jar:/Users/tison/.m2/repository/org/jvnet/mimepull/mimepull/1.9.13/mimepull-1.9.13.jar:/Users/tison/.m2/repository/org/apache/bookkeeper/bookkeeper-server/4.16.2/bookkeeper-server-4.16.2.jar:/Users/tison/Brittani/pulsar/pulsar-functions/localrun/target/pulsar-functions-local-runner-original.jar:/Users/tison/.m2/repository/io/grpc/grpc-all/1.45.1/grpc-all-1.45.1.jar:/Users/tison/.m2/repository/io/grpc/grpc-api/1.45.1/grpc-api-1.45.1.jar:/Users/tison/.m2/repository/io/grpc/grpc-auth/1.45.1/grpc-auth-1.45.1.jar:/Users/tison/.m2/repository/com/google/auth/google-auth-library-credentials/1.4.0/google-auth-library-credentials-1.4.0.jar:/Users/tison/.m2/repository/io/grpc/grpc-context/1.45.1/grpc-context-1.45.1.jar:/Users/tison/.m2/repository/io/grpc/grpc-core/1.45.1/grpc-core-1.45.1.jar:/Users/tison/.m2/repository/com/google/android/annotations/4.1.1.4/annotations-4.1.1.4.jar:/Users/tison/.m2/repository/org/codehaus/mojo/animal-sniffer-annotations/1.19/animal-sniffer-annotations-1.19.jar:/Users/tison/.m2/repository/io/grpc/grpc-grpclb/1.45.1/grpc-grpclb-1.45.1.jar:/Users/tison/.m2/repository/com/google/protobuf/protobuf-java-util/3.19.6/protobuf-java-util-3.19.6.jar:/Users/tison/.m2/repository/io/grpc/grpc-netty/1.45.1/grpc-netty-1.45.1.jar:/Users/tison/.m2/repository/io/grpc/grpc-protobuf/1.45.1/grpc-protobuf-1.45.1.jar:/Users/tison/.m2/repository/com/google/api/grpc/proto-google-common-protos/2.0.1/proto-google-common-protos-2.0.1.jar:/Users/tison/.m2/repository/io/grpc/grpc-protobuf-lite/1.45.1/grpc-protobuf-lite-1.45.1.jar:/Users/tison/.m2/repository/io/grpc/grpc-rls/1.45.1/grpc-rls-1.45.1.jar:/Users/tison/.m2/repository/com/google/auto/value/auto-value-annotations/1.9/auto-value-annotations-1.9.jar:/Users/tison/.m2/repository/io/grpc/grpc-services/1.45.1/grpc-services-1.45.1.jar:/Users/tison/.m2/repository/io/grpc/grpc-stub/1.45.1/grpc-stub-1.45.1.jar:/Users/tison/.m2/repository/io/grpc/grpc-xds/1.45.1/grpc-xds-1.45.1.jar:/Users/tison/.m2/repository/io/grpc/grpc-alts/1.45.1/grpc-alts-1.45.1.jar:/Users/tison/.m2/repository/com/google/re2j/re2j/1.5/re2j-1.5.jar:/Users/tison/.m2/repository/io/opencensus/opencensus-proto/0.2.0/opencensus-proto-0.2.0.jar:/Users/tison/.m2/repository/io/grpc/grpc-netty-shaded/1.45.1/grpc-netty-shaded-1.45.1.jar:/Users/tison/.m2/repository/org/bouncycastle/bcpkix-jdk18on/1.75/bcpkix-jdk18on-1.75.jar:/Users/tison/.m2/repository/org/bouncycastle/bcprov-jdk18on/1.75/bcprov-jdk18on-1.75.jar:/Users/tison/.m2/repository/org/bouncycastle/bcutil-jdk18on/1.75/bcutil-jdk18on-1.75.jar:/Users/tison/.m2/repository/io/perfmark/perfmark-api/0.19.0/perfmark-api-0.19.0.jar:/Users/tison/.m2/repository/com/google/code/findbugs/jsr305/3.0.2/jsr305-3.0.2.jar:/Users/tison/.m2/repository/org/apache/bookkeeper/http/vertx-http-server/4.16.2/vertx-http-server-4.16.2.jar:/Users/tison/.m2/repository/org/apache/bookkeeper/http/http-server/4.16.2/http-server-4.16.2.jar:/Users/tison/.m2/repository/io/vertx/vertx-core/4.3.8/vertx-core-4.3.8.jar:/Users/tison/.m2/repository/io/netty/netty-buffer/4.1.94.Final/netty-buffer-4.1.94.Final.jar:/Users/tison/.m2/repository/io/netty/netty-handler/4.1.94.Final/netty-handler-4.1.94.Final.jar:/Users/tison/.m2/repository/io/netty/netty-transport-native-unix-common/4.1.94.Final/netty-transport-native-unix-common-4.1.94.Final.jar:/Users/tison/.m2/repository/io/netty/netty-codec/4.1.94.Final/netty-codec-4.1.94.Final.jar:/Users/tison/.m2/repository/io/netty/netty-handler-proxy/4.1.94.Final/netty-handler-proxy-4.1.94.Final.jar:/Users/tison/.m2/repository/io/netty/netty-codec-http/4.1.94.Final/netty-codec-http-4.1.94.Final.jar:/Users/tison/.m2/repository/io/netty/netty-codec-http2/4.1.94.Final/netty-codec-http2-4.1.94.Final.jar:/Users/tison/.m2/repository/io/netty/netty-resolver/4.1.94.Final/netty-resolver-4.1.94.Final.jar:/Users/tison/.m2/repository/io/netty/netty-resolver-dns/4.1.94.Final/netty-resolver-dns-4.1.94.Final.jar:/Users/tison/.m2/repository/io/netty/netty-codec-dns/4.1.94.Final/netty-codec-dns-4.1.94.Final.jar:/Users/tison/.m2/repository/io/vertx/vertx-web/4.3.8/vertx-web-4.3.8.jar:/Users/tison/.m2/repository/io/vertx/vertx-web-common/4.3.8/vertx-web-common-4.3.8.jar:/Users/tison/.m2/repository/io/vertx/vertx-auth-common/4.3.8/vertx-auth-common-4.3.8.jar:/Users/tison/.m2/repository/io/vertx/vertx-bridge-common/4.3.8/vertx-bridge-common-4.3.8.jar:/Users/tison/.m2/repository/org/apache/bookkeeper/bookkeeper-common/4.16.2/bookkeeper-common-4.16.2.jar:/Users/tison/.m2/repository/org/jctools/jctools-core/2.1.2/jctools-core-2.1.2.jar:/Users/tison/.m2/repository/org/apache/bookkeeper/bookkeeper-common-allocator/4.16.2/bookkeeper-common-allocator-4.16.2.jar:/Users/tison/.m2/repository/org/apache/bookkeeper/bookkeeper-proto/4.16.2/bookkeeper-proto-4.16.2.jar:/Users/tison/.m2/repository/org/apache/bookkeeper/bookkeeper-slogger-slf4j/4.16.2/bookkeeper-slogger-slf4j-4.16.2.jar:/Users/tison/.m2/repository/org/apache/bookkeeper/bookkeeper-slogger-api/4.16.2/bookkeeper-slogger-api-4.16.2.jar:/Users/tison/.m2/repository/org/apache/bookkeeper/native-io/4.16.2/native-io-4.16.2.jar:/Users/tison/.m2/repository/org/rocksdb/rocksdbjni/7.9.2/rocksdbjni-7.9.2.jar:/Users/tison/.m2/repository/io/netty/netty-transport-native-epoll/4.1.94.Final/netty-transport-native-epoll-4.1.94.Final-linux-x86_64.jar:/Users/tison/.m2/repository/io/netty/netty-transport-classes-epoll/4.1.94.Final/netty-transport-classes-epoll-4.1.94.Final.jar:/Users/tison/.m2/repository/io/netty/netty-tcnative-boringssl-static/2.0.61.Final/netty-tcnative-boringssl-static-2.0.61.Final.jar:/Users/tison/.m2/repository/io/netty/netty-tcnative-classes/2.0.61.Final/netty-tcnative-classes-2.0.61.Final.jar:/Users/tison/.m2/repository/io/netty/netty-tcnative-boringssl-static/2.0.61.Final/netty-tcnative-boringssl-static-2.0.61.Final-linux-x86_64.jar:/Users/tison/.m2/repository/io/netty/netty-tcnative-boringssl-static/2.0.61.Final/netty-tcnative-boringssl-static-2.0.61.Final-linux-aarch_64.jar:/Users/tison/.m2/repository/io/netty/netty-tcnative-boringssl-static/2.0.61.Final/netty-tcnative-boringssl-static-2.0.61.Final-osx-x86_64.jar:/Users/tison/.m2/repository/io/netty/netty-tcnative-boringssl-static/2.0.61.Final/netty-tcnative-boringssl-static-2.0.61.Final-osx-aarch_64.jar:/Users/tison/.m2/repository/io/netty/netty-tcnative-boringssl-static/2.0.61.Final/netty-tcnative-boringssl-static-2.0.61.Final-windows-x86_64.jar:/Users/tison/.m2/repository/org/apache/bookkeeper/circe-checksum/4.16.2/circe-checksum-4.16.2.jar:/Users/tison/.m2/repository/commons-cli/commons-cli/1.5.0/commons-cli-1.5.0.jar:/Users/tison/.m2/repository/org/apache/httpcomponents/httpclient/4.5.13/httpclient-4.5.13.jar:/Users/tison/.m2/repository/org/apache/httpcomponents/httpcore/4.4.15/httpcore-4.4.15.jar:/Users/tison/.m2/repository/io/reactivex/rxjava3/rxjava/3.0.1/rxjava-3.0.1.jar:

root = /Users/tison/Brittani/pulsar/download/pulsar_functions/public/default/demo-sink/0/pulsar-sink-demo-1.0.0.nar

Ref - https://docs.oracle.com/javase/specs/jvms/se8/html/jvms-5.html#jvms-5.4.3.1

tisonkun commented 1 year ago

cc @jiangpengcheng @nlu90 @lhotari @nicoloboschi Although the root cause located, it's not quite easy to solve this issue. My workaround can work, but I'm not sure if extra affect coupled.

github-actions[bot] commented 1 year ago

The issue had no activity for 30 days, mark with Stale label.