rollerQ
commented
8 months ago add log：
dlink] 2024-02-02 14:10:46 CST INFO org.apache.flink.yarn.YarnClusterDescriptor 206 getLocalFlinkDistPath - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
[dlink] 2024-02-02 14:10:46 CST WARN org.apache.flink.yarn.YarnClusterDescriptor 333 isReadyForDeployment - Neither the HADOOP_CONF_DIR nor the YARN_CONF_DIR environment variable is set. The Flink YARN Client needs one of these to be set to properly load the Hadoop configuration for accessing YARN.
[dlink] 2024-02-02 14:10:46 CST INFO org.apache.flink.yarn.YarnClusterDescriptor 643 logIfComponentMemNotIntegerMultipleOfYarnMinAllocation - The configured JobManager memory is 768 MB. YARN will allocate 1024 MB to make up an integer multiple of its minimum allocation memory (1024 MB, configured via 'yarn.scheduler.minimum-allocation-mb'). The extra 256 MB may not be used by Flink.
[dlink] 2024-02-02 14:10:46 CST INFO org.apache.flink.yarn.YarnClusterDescriptor 550 deployInternal - Cluster specification: ClusterSpecification{masterMemoryMB=1024, taskManagerMemoryMB=1024, slotsPerTaskManager=1}
[dlink] 2024-02-02 14:10:46 CST INFO org.apache.flink.runtime.util.config.memory.MemoryBackwardsCompatibilityUtils 56 lambda$getConfWithLegacyHeapSizeMappedToNewConfigOption$0 - 'jobmanager.memory.process.size' is not specified, use the configured deprecated task manager heap value (1024.000mb (1073741824 bytes)) for it.
[dlink] 2024-02-02 14:10:46 CST INFO org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils 282 capToMinMax - The derived from fraction jvm overhead memory (102.400mb (107374184 bytes)) is less than its min value 192.000mb (201326592 bytes), min value will be used instead
[dlink] 2024-02-02 14:10:46 CST INFO org.apache.flink.yarn.YarnClusterDescriptor 1058 startAppMaster - Submitting application master application_1670294478436_4074955
[dlink] 2024-02-02 14:10:46 CST INFO org.apache.hadoop.io.retry.RetryInvocationHandler 400 log - org.apache.flink.hadoop.shaded.com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag., while invoking ApplicationClientProtocolPBClientImpl.getApplicationReport over 17. Trying to failover immediately.
[dlink] 2024-02-02 14:10:46 CST INFO org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider 100 performFailover - Failing over to 18
[dlink] 2024-02-02 14:10:46 CST INFO org.apache.hadoop.io.retry.RetryInvocationHandler 400 log - java.net.ConnectException: Call From hadoop80/xxxx to xxxx:8032 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused, while invoking ApplicationClientProtocolPBClientImpl.getApplicationReport over 18 after 1 failover attempts. Trying to failover after sleeping for 25596ms.
[dl
.....
[dlink] 2024-02-02 11:47:16 CST INFO org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider 100 performFailover - Failing over to 17
[dlink] 2024-02-02 11:47:16 CST ERROR com.dlink.utils.LogUtil 46 getError - org.apache.flink.client.deployment.ClusterDeploymentException: Could not deploy Yarn job cluster.
at org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:460)
at com.dlink.gateway.yarn.YarnPerJobGateway.submitJobGraph(YarnPerJobGateway.java:100)
at com.dlink.job.JobManager.submitByGateway(JobManager.java:629)
at com.dlink.job.JobManager.executeSql(JobManager.java:486)
at com.dlink.service.impl.TaskServiceImpl.submitTask(TaskServiceImpl.java:251)
at com.dlink.service.impl.TaskServiceImpl$$FastClassBySpringCGLIB$$168bea78.invoke(
Zzm0809
github-actions[bot]
aiwenmo
zackyoungh
Search before asking
What happened
1.Environment： dinky 0.7.2 flink1.12(hw-mrs)：Because Huawei Cloud flink cannot be upgraded, only a lower version of dinky can be used
2.Flink yarn per job task: use example sql 【hello word】：https://cloud.tencent.com/developer/article/1925451：五、Yarn-Per-Job 实践
3.Configuration：
4.issue: The task can run normally on yarn, but dinky continues to circle on the submit page and stops after a time out. The following error is displayed in dinky's log
I've tried all kinds of packages： flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar、 flink-shaded-hadoop-2-uber-2.8.3-10.0.jar flink-shaded-hadoop-2-uber-3.0.0-cdh6.2.0-7.0.jar.tar.gz flink-shaded-hadoop2-uber-1.7.0.0101-hw-ei-12.jar
What you expected to happen
dinky can feedback the submitted task status normally. The operation and maintenance center displays the task that has just been submitted, but it does not
How to reproduce
flink test connected to Huawei cloud may be required
Anything else
no
Version
0.7.2
Are you willing to submit PR?
Code of Conduct