search

apache / incubator-streampark

Make stream processing easier! Easy-to-use streaming application development framework and operation platform.
https://streampark.apache.org/
Apache License 2.0
3.91k stars 1.01k forks source link

[Bug] Flink on Yarn per-job mode submission task failed #3487

Open Tandoy opened 10 months ago

Tandoy commented 10 months ago

Search before asking

Java Version

1.8.0_331

Scala Version

2.12.x

StreamPark Version

2.1.1

Flink Version

1.14.1

deploy mode

yarn-per-job

What happened

Mode: Custom Code Error when submitting task: It shows that there is no relevant jar package. And the jar package displayed in the log error is inconsistent with the uploaded jar.

Error Exception

java.util.concurrent.CompletionException: java.lang.reflect.InvocationTargetException
    at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
    at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
    at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1606)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.streampark.flink.client.FlinkClient$.$anonfun$proxy$1(FlinkClient.scala:80)
    at org.apache.streampark.flink.proxy.FlinkShimsProxy$.$anonfun$proxy$1(FlinkShimsProxy.scala:60)
    at org.apache.streampark.common.util.ClassLoaderUtils$.runAsClassLoader(ClassLoaderUtils.scala:38)
    at org.apache.streampark.flink.proxy.FlinkShimsProxy$.proxy(FlinkShimsProxy.scala:60)
    at org.apache.streampark.flink.client.FlinkClient$.proxy(FlinkClient.scala:75)
    at org.apache.streampark.flink.client.FlinkClient$.submit(FlinkClient.scala:49)
    at org.apache.streampark.flink.client.FlinkClient.submit(FlinkClient.scala)
    at org.apache.streampark.console.core.service.impl.ApplicationServiceImpl.lambda$start$10(ApplicationServiceImpl.java:1544)
    at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
    ... 3 more
Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: /data/modules/flink-1.14.1/lib/flinkcdc-1.0-SNAPSHOT.jar (没有那个文件或目录)
    at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3089)
    at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:3036)
    at org.apache.hadoop.conf.Configuration.loadProps(Configuration.java:2914)
    at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2896)
    at org.apache.hadoop.conf.Configuration.addResource(Configuration.java:1013)
    at org.apache.flink.yarn.Utils.getYarnAndHadoopConfiguration(Utils.java:681)
    at org.apache.flink.yarn.YarnClusterClientFactory.getClusterDescriptor(YarnClusterClientFactory.java:80)
    at org.apache.flink.yarn.YarnClusterClientFactory.createClusterDescriptor(YarnClusterClientFactory.java:61)
    at org.apache.flink.yarn.YarnClusterClientFactory.createClusterDescriptor(YarnClusterClientFactory.java:43)
    at org.apache.streampark.flink.client.impl.YarnPerJobClient$.doSubmit(YarnPerJobClient.scala:70)
    at org.apache.streampark.flink.client.trait.FlinkClientTrait.submit(FlinkClientTrait.scala:125)
    at org.apache.streampark.flink.client.trait.FlinkClientTrait.submit$(FlinkClientTrait.scala:62)
    at org.apache.streampark.flink.client.impl.YarnPerJobClient$.submit(YarnPerJobClient.scala:40)
    at org.apache.streampark.flink.client.FlinkClientHandler$.submit(FlinkClientHandler.scala:40)
    at org.apache.streampark.flink.client.FlinkClientHandler.submit(FlinkClientHandler.scala)
    ... 16 more
Caused by: java.io.FileNotFoundException: /data/modules/flink-1.14.1/lib/flinkcdc-1.0-SNAPSHOT.jar (没有那个文件或目录)
    at java.util.zip.ZipFile.open(Native Method)
    at java.util.zip.ZipFile.<init>(ZipFile.java:226)
    at java.util.zip.ZipFile.<init>(ZipFile.java:155)
    at java.util.jar.JarFile.<init>(JarFile.java:170)
    at java.util.jar.JarFile.<init>(JarFile.java:107)
    at sun.net.www.protocol.jar.URLJarFile.<init>(URLJarFile.java:93)
    at sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:69)
    at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:168)
    at sun.net.www.protocol.jar.JarFileFactory.getOrCreate(JarFileFactory.java:91)
    at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:129)
    at sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:172)
    at org.apache.hadoop.conf.Configuration.parse(Configuration.java:3009)
    at org.apache.hadoop.conf.Configuration.getStreamReader(Configuration.java:3105)
    at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3063)
    ... 30 more

Screenshots

微信截图_20240112132304

微信截图_20240112132321

Are you willing to submit PR?

Code of Conduct

Tandoy commented 10 months ago

When I put the corresponding jar package into the specified directory, the following error occurred: 微信截图_20240112140953