search

DTStack / chunjun

A data integration framework
https://dtstack.github.io/chunjun/
Apache License 2.0
4.01k stars 1.69k forks source link

[Bug] [chunjun-dist] 导入chunjun-dist依赖 flink 启动失败 #1656

Open liuzx8888 opened 1 year ago

liuzx8888 commented 1 year ago

Search before asking

What happened

版本： FLINK ：1.16.1 hadoop ：3.3.0 chunjun ：master

把依赖添加到flink lib，启动flink失败，去掉 chunjun-dist 就能正常的启动成功 cp -r chunjun-dist $FLINK_HOME/lib

What you expected to happen

2023-05-07 12:15:54,289 WARN  org.apache.flink.runtime.util.EnvironmentInformation         [] - Error while accessing user/group information via Hadoop utils.
java.lang.reflect.InvocationTargetException: null
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_291]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_291]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_291]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_291]
    at org.apache.flink.runtime.util.EnvironmentInformation.getHadoopUser(EnvironmentInformation.java:215) [chunjun-restore-mysql.jar:?]
    at org.apache.flink.runtime.util.EnvironmentInformation.logEnvironmentInfo(EnvironmentInformation.java:434) [chunjun-restore-mysql.jar:?]
    at org.apache.flink.runtime.entrypoint.StandaloneSessionClusterEntrypoint.main(StandaloneSessionClusterEntrypoint.java:44) [chunjun-restore-mysql.jar:?]
Caused by: java.lang.NoSuchMethodError: com.google.common.cache.CacheBuilder.ticker(Lcom/google/common/base/Ticker;)Lcom/google/common/cache/CacheBuilder;
    at org.apache.hadoop.security.Groups.<init>(Groups.java:138) ~[hadoop-common-3.3.0.jar:?]
    at org.apache.hadoop.security.Groups.<init>(Groups.java:102) ~[hadoop-common-3.3.0.jar:?]
    at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:451) ~[hadoop-common-3.3.0.jar:?]
    at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:338) ~[hadoop-common-3.3.0.jar:?]
    at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:300) ~[hadoop-common-3.3.0.jar:?]
    at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:567) ~[hadoop-common-3.3.0.jar:?]
    ... 7 more

2023-05-07 11:08:24,150 INFO  org.apache.flink.configuration.GlobalConfiguration           [] - Loading dynamic configuration property: jobmanager.memory.off-heap.size, 134217728b
2023-05-07 11:08:24,150 INFO  org.apache.flink.configuration.GlobalConfiguration           [] - Loading dynamic configuration property: jobmanager.memory.jvm-overhead.min, 201326592b
2023-05-07 11:08:24,151 INFO  org.apache.flink.configuration.GlobalConfiguration           [] - Loading dynamic configuration property: jobmanager.memory.jvm-metaspace.size, 268435456b
2023-05-07 11:08:24,151 INFO  org.apache.flink.configuration.GlobalConfiguration           [] - Loading dynamic configuration property: jobmanager.memory.heap.size, 469762048b
2023-05-07 11:08:24,151 INFO  org.apache.flink.configuration.GlobalConfiguration           [] - Loading dynamic configuration property: jobmanager.memory.jvm-overhead.max, 201326592b
2023-05-07 11:08:24,224 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint        [] - Starting StandaloneSessionClusterEntrypoint.
2023-05-07 11:08:24,303 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint        [] - Install default filesystem.
2023-05-07 11:08:24,422 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint        [] - Install security context.
2023-05-07 11:08:24,546 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint        [] - Shutting StandaloneSessionClusterEntrypoint down with application status FAILED. Diagnostics java.lang.NoSuchMethodError: com.google.common.cache.CacheBuilder.ticker(Lcom/google/common/base/Ticker;)Lcom/google/common/cache/CacheBuilder;
    at org.apache.hadoop.security.Groups.<init>(Groups.java:138)
    at org.apache.hadoop.security.Groups.<init>(Groups.java:102)
    at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:451)
    at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:338)
    at org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:366)
    at org.apache.flink.runtime.security.modules.HadoopModule.install(HadoopModule.java:67)
    at org.apache.flink.runtime.security.SecurityUtils.installModules(SecurityUtils.java:76)
    at org.apache.flink.runtime.security.SecurityUtils.install(SecurityUtils.java:57)
    at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.installSecurityContext(ClusterEntrypoint.java:274)
    at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:226)
    at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:729)
    at org.apache.flink.runtime.entrypoint.StandaloneSessionClusterEntrypoint.main(StandaloneSessionClusterEntrypoint.java:59)
.

How to reproduce

cp -r chunjun-dist $FLINK_HOME/lib

Anything else

No response

Version

master

Are you willing to submit PR?

Code of Conduct

ll076110 commented 1 year ago

将flink-shaded-hadoop-uber包放入lib目录里试试

getwtf commented 1 year ago

将flink-shaded-hadoop-uber包放入lib目录里试试

`flink lib 添加 flink-shaded-hadoop-2-2.7.5-10.0，出现新的问题：

Log Type: jobmanager.err

Log Upload Time: 星期一 五月 08 10:06:49 +0800 2023

Log Length: 1548

SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/home/hadoop-3.1.3/data/nm-local-dir/usercache/root/appcache/application_1682257958270_0172/filecache/26/chunjun-connector-binlog.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/hadoop-3.1.3/data/nm-local-dir/usercache/root/appcache/application_1682257958270_0172/filecache/25/chunjun-connector-hive3.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/hadoop-3.1.3/data/nm-local-dir/usercache/root/appcache/application_1682257958270_0172/filecache/16/chunjun-metrics-prometheus.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/hadoop-3.1.3/data/nm-local-dir/usercache/root/appcache/application_1682257958270_0172/filecache/21/log4j-slf4j-impl-2.12.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/hadoop-3.1.3/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] Exception in thread "main" java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.security.UserGroupInformation at org.apache.flink.yarn.entrypoint.YarnEntrypointUtils.logYarnEnvironmentInformation(YarnEntrypointUtils.java:116) at org.apache.flink.yarn.entrypoint.YarnJobClusterEntrypoint.main(YarnJobClusterEntrypoint.java:83)`

ll076110 commented 1 year ago

@getwtf 以什么模式启动的

laixueyong commented 1 month ago

@ll076110 per-job @getwtf 我也遇到相同的问题 您解决了嘛