streampark commons-cli NoSuchMethodError

pj001 commented 8 months ago

Search before asking

[X] I had searched in the issues and found no similar issues.

Java Version

1.8

Scala Version

2.11.x

StreamPark Version

2.1.2

Flink Version

1.13.6

deploy mode

yarn-application

What happened

streampark 使用 flink1.13.6 （commons-cli-1.3.1）提交任务，出现 commons-cli 冲突，我这边是首次提交无问题，某次重启任务时此问题出现： 1.检查项目是否打包依赖，排除commons-cli打包后提交任务。问题未解决 2.尝试将streampark lib 目录包更换与flink对应版本一致，重启streampark。问题未解决。 3.尝试替换streampark hdfs flink 与本地备份的commons-cli jar,重启streampark。问题未解决。 4.删除本地streampark workspace，重启集群，启动任务，问题未解决。 5.删除hdfs streampark workspace,重启集群，启动任务，问题未解决。 6.删除本地streampark workspace，hdfs streampark workspace,替换 flink1.13 streampark2.1.2 commons-cli 为 commons-cli-1.3.1 问题解决。 7.回滚streampark2.1.1,问题解决。猜测可能是streampark重启任务任务时从本地或者hdfs的备份加载flink依赖导致失败，或者是加载了整个flink 任务（flink项目代码）的 jar ，导致冲突。

Error Exception

java.util.concurrent.CompletionException: java.lang.reflect.InvocationTargetException
    at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
    at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
    at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1592)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.streampark.flink.client.FlinkClient$.$anonfun$proxy$1(FlinkClient.scala:80)
    at org.apache.streampark.flink.proxy.FlinkShimsProxy$.$anonfun$proxy$1(FlinkShimsProxy.scala:60)
    at org.apache.streampark.common.util.ClassLoaderUtils$.runAsClassLoader(ClassLoaderUtils.scala:38)
    at org.apache.streampark.flink.proxy.FlinkShimsProxy$.proxy(FlinkShimsProxy.scala:60)
    at org.apache.streampark.flink.client.FlinkClient$.proxy(FlinkClient.scala:75)
    at org.apache.streampark.flink.client.FlinkClient$.submit(FlinkClient.scala:49)
    at org.apache.streampark.flink.client.FlinkClient.submit(FlinkClient.scala)
    at org.apache.streampark.console.core.service.impl.ApplicationServiceImpl.lambda$start$10(ApplicationServiceImpl.java:1566)
    at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
    ... 3 more
Caused by: java.lang.NoSuchMethodError: org.apache.commons.cli.Option.builder(Ljava/lang/String;)Lorg/apache/commons/cli/Option$Builder;
    at org.apache.flink.client.cli.DynamicPropertiesUtil.<clinit>(DynamicPropertiesUtil.java:39)
    at org.apache.flink.client.cli.GenericCLI.addGeneralOptions(GenericCLI.java:108)
    at org.apache.flink.client.cli.CliFrontend.<init>(CliFrontend.java:136)
    at org.apache.flink.client.cli.CliFrontend.<init>(CliFrontend.java:119)
    at org.apache.streampark.flink.client.trait.FlinkClientTrait.getCustomCommandLines(FlinkClientTrait.scala:293)
    at org.apache.streampark.flink.client.trait.FlinkClientTrait.getCommandLineOptions(FlinkClientTrait.scala:379)
    at org.apache.streampark.flink.client.trait.FlinkClientTrait.getCommandLineOptions$(FlinkClientTrait.scala:378)
    at org.apache.streampark.flink.client.impl.YarnApplicationClient$.getCommandLineOptions(YarnApplicationClient.scala:44)
    at org.apache.streampark.flink.client.trait.FlinkClientTrait.getCommandLineAndFlinkConfig(FlinkClientTrait.scala:309)
    at org.apache.streampark.flink.client.trait.FlinkClientTrait.submit(FlinkClientTrait.scala:81)
    at org.apache.streampark.flink.client.trait.FlinkClientTrait.submit$(FlinkClientTrait.scala:60)
    at org.apache.streampark.flink.client.impl.YarnApplicationClient$.submit(YarnApplicationClient.scala:44)
    at org.apache.streampark.flink.client.FlinkClientHandler$.submit(FlinkClientHandler.scala:40)
    at org.apache.streampark.flink.client.FlinkClientHandler.submit(FlinkClientHandler.scala)
    ... 16 more

Screenshots

Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!(您是否要贡献这个PR?)

Code of Conduct

[X] I agree to follow this project's Code of Conduct

pj001 commented 8 months ago

根据实测，streampark2.1.2 与此包存在冲突。

wolfboys commented 8 months ago

You can check if there are common-cli conflicts in hadoop-related packages. e.g flink-shaded-hadoop-x-uber.jar

sunderdont commented 2 months ago

commons-cli-1.2 in flink-shaded-hadoop-3-uber-xxx.jar is lower than streampark-2.1.2's commons-cli-1.5.0，who is short of org.apache.commons.cli.Option$Builder .

consider flink-shaded-hadoop-2-uber-2.8.3-10.0.jar，who's commons-cli is 1.3，it's usable.

apache / incubator-streampark