Closed scrottty closed 1 year ago
This was solved by downgrading to Pulsar Version 2.10.4
and Java version 11
due to netty issues. See https://apache-pulsar.slack.com/archives/C5Z4T36F7/p1683674853984909 for slack discussion. Leaving open as it might be needed to be fixed in v3+
@scrottty thanks a lot, saved my time :)
faced the same issue pulsar 2.11.0
+ temurin-17-jdk
+ ubuntu20.04
, downgrading helped
The exact failing point is above Pulsar reflection usage, not about Netty. But other place may be.
@scrottty could you share the error log after "setting --add-opens=java.management/sun.management=ALL-UNNAMED
is as many places"?
temurin-17-jdk
@yurymuski what's your exactly error stack? The DnsResolverUtil
is added in 2.11.0 and the module visibility issue seems not exist in JDK 11 but JDK 17, so this combination may have no difference.
Well. I get it wrong. Then you mean you downgrade to JDK 11 + Pulsar 2.10.4 and it helps?
@lhotari It seems related to your previous change at https://github.com/apache/pulsar/pull/15219. I'm looking into this, but it would be great if you can take a look also.
No. It's about how we open module in localrun.
at org.apache.pulsar.functions.LocalRunner.startThreadedMode(LocalRunner.java:643) ~[org.apache.pulsar-pulsar-functions-local-runner-original-3.0.0.jar:3.0.0]
We are currently relying on script to put --add-opens java.base/sun.net=ALL-UNNAMED
. Some development is needed to convey the same manner to the local runner. Or perhaps adding those entries to the JAR file may be a better idea.
Ref -
Not exactly - we still use script function-localrunner
to start the process. Fulfill the OPTS
there should help. I'm working on a fix now.
You may test out https://github.com/apache/pulsar/pull/20417
Any rough idea when the fix will be released, tisonkun? If it takes long, is there any local patch I can apply. Much appreciated!
It can be cherry-picked to 3.0.1 and I saw an ongoing discussion for 3.0.1 on our dev@ mailing list.
cc @RobertIndie may you check this ticket also?
@rayintuit Now the fix is picked to branch-3.0 https://github.com/apache/pulsar/commit/d518b71db60125b410ee84acfa36cfabeba89024.
That means it will be released with 3.0.1. You can subscribe dev@pulsar.apache.org to catch up the related updates.
Running into a similar problem using Pulsar Client 3.0.0 and JDK 17 in one of our microservices:
2023-07-03 11:58:41.915 WARN 19312 --- [ main] o.a.p.c.u.n.DnsResolverUtil : : Cannot get DNS TTL settings from sun.net.InetAddressCachePolicy class
java.lang.IllegalAccessException: class org.apache.pulsar.common.util.netty.DnsResolverUtil cannot access class sun.net.InetAddressCachePolicy (in module java.base) because module java.base does not export sun.net to unnamed module @1a708820
This occurs when we build the PulsarClient
object from the client.api
package.
I have tried, as mentioned above, adding --add-opens java.base/sun.net=ALL-UNNAMED
to my VM args when running the service but I still get this exception.
On top of that I have also tried building branch-3.0
locally and I still get the exception with that build.
Perhaps this issue is more widespread than just the functions local runner?
It would be awesome if the fix can be added to v3.1.0, tisonkun. Much appreciated!
@rayintuit It should be included in v3.1.0 because we merge it into master before we cut branch-3.1.
If that is the case, it seems like the issue is still there as we are using v3.1.0 for Java 17. Any help looking into it would be greatly appreciated. Thanks again, tisonkun.
Java version
openjdk version "17.0.7" 2023-04-18 LTS
OpenJDK Runtime Environment Corretto-17.0.7.7.1 (build 17.0.7+7-LTS)
OpenJDK 64-Bit Server VM Corretto-17.0.7.7.1 (build 17.0.7+7-LTS, mixed mode, sharing)
pom.xml
<properties>
<pulsar-client-original.version>3.1.0</pulsar-client-original.version>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-client-original</artifactId>
<version>${pulsar-client-original.version}</version>
</dependency>
</dependencies>
Exception
java.lang.IllegalAccessException: class org.apache.pulsar.common.util.netty.DnsResolverUtil cannot access class sun.net.InetAddressCachePolicy (in module java.base) because module java.base does not export sun.net to unnamed module @fa4c865
at java.base/jdk.internal.reflect.Reflection.newIllegalAccessException(Reflection.java:392) ~[na:na]
at java.base/java.lang.reflect.AccessibleObject.checkAccess(AccessibleObject.java:674) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:560) ~[na:na]
at org.apache.pulsar.common.util.netty.DnsResolverUtil.<clinit>(DnsResolverUtil.java:46) ~[pulsar-client-all-2.11.0.jar:2.11.0]
at org.apache.pulsar.client.impl.ConnectionPool.createAddressResolver(ConnectionPool.java:159) ~[pulsar-client-all-2.11.0.jar:2.11.0]
at org.apache.pulsar.client.impl.ConnectionPool.lambda$new$1(ConnectionPool.java:126) ~[pulsar-client-all-2.11.0.jar:2.11.0]
at java.base/java.util.Optional.orElseGet(Optional.java:364) ~[na:na]
at org.apache.pulsar.client.impl.ConnectionPool.<init>(ConnectionPool.java:126) ~[pulsar-client-all-2.11.0.jar:2.11.0]
at org.apache.pulsar.client.impl.ConnectionPool.<init>(ConnectionPool.java:95) ~[pulsar-client-all-2.11.0.jar:2.11.0]
at org.apache.pulsar.client.impl.ConnectionPool.<init>(ConnectionPool.java:90) ~[pulsar-client-all-2.11.0.jar:2.11.0]
at org.apache.pulsar.client.impl.PulsarClientImpl.<init>(PulsarClientImpl.java:198) ~[pulsar-client-all-2.11.0.jar:2.11.0]
at org.apache.pulsar.client.impl.PulsarClientImpl.<init>(PulsarClientImpl.java:154) ~[pulsar-client-all-2.11.0.jar:2.11.0]
at org.apache.pulsar.client.impl.ClientBuilderImpl.build(ClientBuilderImpl.java:63) ~[pulsar-client-all-2.11.0.jar:2.11.0]
at org.springframework.pulsar.config.PulsarClientFactoryBean.createInstance(PulsarClientFactoryBean.java:54) ~[spring-pulsar-0.2.0.jar:0.2.0]
at org.springframework.pulsar.config.PulsarClientFactoryBean.createInstance(PulsarClientFactoryBean.java:36) ~[spring-pulsar-0.2.0.jar:0.2.0]
at org.springframework.beans.factory.config.AbstractFactoryBean.afterPropertiesSet(AbstractFactoryBean.java:142) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1816) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1766) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:598) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:520) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:326) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:324) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:254) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1417) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1337) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:888) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:791) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:550) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1332) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1162) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:560) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:520) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:326) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:324) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:254) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1417) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1337) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:888) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:791) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:550) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1332) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1162) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:560) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:520) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:326) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:324) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:254) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1417) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1337) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:888) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:791) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:245) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1352) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1189) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:560) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:520) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:326) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:324) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:254) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1417) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1337) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:888) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:791) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:245) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1352) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1189) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:560) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:520) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:326) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:324) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:973) ~[spring-beans-6.0.9.jar:6.0.9]
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:941) ~[spring-context-6.0.9.jar:6.0.9]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:608) ~[spring-context-6.0.9.jar:6.0.9]
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:146) ~[spring-boot-3.1.0.jar:3.1.0]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:733) ~[spring-boot-3.1.0.jar:3.1.0]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:435) ~[spring-boot-3.1.0.jar:3.1.0]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:311) ~[spring-boot-3.1.0.jar:3.1.0]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1305) ~[spring-boot-3.1.0.jar:3.1.0]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1294) ~[spring-boot-3.1.0.jar:3.1.0]
at com.intuit.identity.core.verifier.VatIdVerifierApplication.main(VatIdVerifierApplication.java:10) ~[classes/:na]
@rayintuit the patch only modifies scripts to add-opens
for related modules.
If you're doing packaging by yourself, you may add-opens by yourself. I'm not sure if we can add-opens in the JARs delivered and convert it out-of-the-box to downsteams.
Search before asking
Version
Pulsar Version: v3.0.0 OS Version: macOS Ventura 13.3.1 (M1 Mac) Java Version: openjdk 17.0.7 2023-04-18
Minimal reproduce step
Running through the Debezium example here: https://pulsar.apache.org/docs/3.0.x/io-cdc-debezium/
What did you expect to see?
The
debezium-postgres-source
connector successfully connect to the DBWhat did you see instead?
Fails when running
bin/pulsar-admin source localrun --source-config-file debezium-postgres-source-config.yaml
The error is:Same as https://github.com/apache/pulsar/issues/15349
Anything else?
On a M1 Mac. Have tried older versions and running through Docker to no avail. Even tried editing
pulsar-admin-common.sh
to try different options on Line 98 as per the suggestions in https://github.com/apache/pulsar/issues/15349 but nothing workedAre you willing to submit a PR?