search

apache / pulsar

Apache Pulsar - distributed pub-sub messaging system
https://pulsar.apache.org/
Apache License 2.0
14.23k stars 3.58k forks source link

Pulsar Function random IO Exception error #7650

Closed opera33 closed 1 year ago

opera33 commented 4 years ago

Describe the bug Randomly getting IO exception with Pulsar Function, We have about 500 of the functions running on 1 cluster with 3 nodes, using same jar (fat jar) file with 2 brokers with function workers, and 1 broker without function worker. (all running 2.5.2)

Some of the function may randomly hit IO Exception and upon investigating, this is the error message we got in the log.

Expected behavior Function run as usual

Desktop (please complete the following information):

Additional context

01:33:40.240 [public/default/logofb20337-0] INFO  org.apache.pulsar.functions.instance.JavaInstanceRunnable - Load JAR: /home/ubuntu/apache-pulsar-2.5.2/functions/pulsar-1.0-jar-with-dependencies.jar
01:33:42.922 [public/default/logofb20337-0] ERROR org.apache.pulsar.functions.instance.JavaInstanceRunnable - [public/default/logofb20337:0] Uncaught exception in Java Instance
java.io.IOException: Unable to delete /tmp/pulsar-nar/pulsar-1.0-jar-with-dependencies.jar-unpacked/org/apache/pulsar/shade/org/asynchttpclient/netty/channel/DefaultChannelPool$PoolLeaseStrategy$2.class
        at org.apache.pulsar.common.nar.FileUtils.deleteFile(FileUtils.java:213) ~[org.apache.pulsar-pulsar-common-2.5.2.jar:2.5.2]
        at org.apache.pulsar.common.nar.FileUtils.deleteFiles(FileUtils.java:202) ~[org.apache.pulsar-pulsar-common-2.5.2.jar:2.5.2]
        at org.apache.pulsar.common.nar.FileUtils.deleteFile(FileUtils.java:209) ~[org.apache.pulsar-pulsar-common-2.5.2.jar:2.5.2]
        at org.apache.pulsar.common.nar.FileUtils.deleteFiles(FileUtils.java:202) ~[org.apache.pulsar-pulsar-common-2.5.2.jar:2.5.2]
        at org.apache.pulsar.common.nar.FileUtils.deleteFile(FileUtils.java:209) ~[org.apache.pulsar-pulsar-common-2.5.2.jar:2.5.2]
        at org.apache.pulsar.common.nar.FileUtils.deleteFiles(FileUtils.java:202) ~[org.apache.pulsar-pulsar-common-2.5.2.jar:2.5.2]
        at org.apache.pulsar.common.nar.FileUtils.deleteFile(FileUtils.java:209) ~[org.apache.pulsar-pulsar-common-2.5.2.jar:2.5.2]
        at org.apache.pulsar.common.nar.FileUtils.deleteFiles(FileUtils.java:202) ~[org.apache.pulsar-pulsar-common-2.5.2.jar:2.5.2]
        at org.apache.pulsar.common.nar.FileUtils.deleteFile(FileUtils.java:209) ~[org.apache.pulsar-pulsar-common-2.5.2.jar:2.5.2]
        at org.apache.pulsar.common.nar.FileUtils.deleteFiles(FileUtils.java:202) ~[org.apache.pulsar-pulsar-common-2.5.2.jar:2.5.2]
        at org.apache.pulsar.common.nar.FileUtils.deleteFile(FileUtils.java:209) ~[org.apache.pulsar-pulsar-common-2.5.2.jar:2.5.2]
        at org.apache.pulsar.common.nar.FileUtils.deleteFiles(FileUtils.java:202) ~[org.apache.pulsar-pulsar-common-2.5.2.jar:2.5.2]
        at org.apache.pulsar.common.nar.FileUtils.deleteFile(FileUtils.java:209) ~[org.apache.pulsar-pulsar-common-2.5.2.jar:2.5.2]
        at org.apache.pulsar.common.nar.FileUtils.deleteFiles(FileUtils.java:202) ~[org.apache.pulsar-pulsar-common-2.5.2.jar:2.5.2]
        at org.apache.pulsar.common.nar.FileUtils.deleteFile(FileUtils.java:209) ~[org.apache.pulsar-pulsar-common-2.5.2.jar:2.5.2]
        at org.apache.pulsar.common.nar.FileUtils.deleteFiles(FileUtils.java:202) ~[org.apache.pulsar-pulsar-common-2.5.2.jar:2.5.2]
        at org.apache.pulsar.common.nar.FileUtils.deleteFile(FileUtils.java:209) ~[org.apache.pulsar-pulsar-common-2.5.2.jar:2.5.2]
        at org.apache.pulsar.common.nar.FileUtils.deleteFiles(FileUtils.java:202) ~[org.apache.pulsar-pulsar-common-2.5.2.jar:2.5.2]
        at org.apache.pulsar.common.nar.FileUtils.deleteFile(FileUtils.java:209) ~[org.apache.pulsar-pulsar-common-2.5.2.jar:2.5.2]
        at org.apache.pulsar.common.nar.NarUnpacker.unpackNar(NarUnpacker.java:73) ~[org.apache.pulsar-pulsar-common-2.5.2.jar:2.5.2]
        at org.apache.pulsar.common.nar.NarClassLoader.getFromArchive(NarClassLoader.java:151) ~[org.apache.pulsar-pulsar-common-2.5.2.jar:2.5.2]
        at org.apache.pulsar.functions.utils.functioncache.FunctionCacheEntry.<init>(FunctionCacheEntry.java:75) ~[org.apache.pulsar-pulsar-functions-utils-2.5.2.jar:2.5.2]
        at org.apache.pulsar.functions.utils.functioncache.FunctionCacheManagerImpl.registerFunctionInstanceWithArchive(FunctionCacheManagerImpl.java:128) ~[org.apache.pulsar-pulsar-functions-utils-2.5.2.jar:2.5.2]
        at org.apache.pulsar.functions.instance.JavaInstanceRunnable.loadJars(JavaInstanceRunnable.java:306) ~[org.apache.pulsar-pulsar-functions-instance-2.5.2.jar:?]
        at org.apache.pulsar.functions.instance.JavaInstanceRunnable.setupJavaInstance(JavaInstanceRunnable.java:190) ~[org.apache.pulsar-pulsar-functions-instance-2.5.2.jar:?]
        at org.apache.pulsar.functions.instance.JavaInstanceRunnable.run(JavaInstanceRunnable.java:245) [org.apache.pulsar-pulsar-functions-instance-2.5.2.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
opera33 commented 4 years ago

In pulsar-nar/*.jar-unpacked/ directory, the files in it doesn't seem to be complete, i think prompted the error, do you have a quick fix for this? Tried restarting broker, but not helpful.

tisonkun commented 1 year ago

Closed as stale. Please create a new issue if it's still relevant to the maintained versions.