search

DataLinkDC / dinky

Dinky is a real-time data development platform based on Apache Flink, enabling agile data development, deployment and operation.
http://www.dinky.org.cn
Apache License 2.0
3.11k stars 1.14k forks source link

[Bug] [Devops] When using Dinky dev version ，data sources that support HDFS to manage paimon #3872

Open dsanww opened 6 days ago

dsanww commented 6 days ago

Search before asking

What happened

When using Dinky dev version(commit:114cc18179bb857c93c34324b32438f97ef8af1c) ，data sources that support HDFS to manage paimon An error was reported while submitting the paimon task： Caused by: java.lang.RuntimeException: java.lang.NullPointerException at org.apache.paimon.utils.HadoopUtils.readHadoopXml(HadoopUtils.java:220) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT] at org.apache.paimon.utils.HadoopUtils.addHadoopConfIfFound(HadoopUtils.java:175) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT] at org.apache.paimon.utils.HadoopUtils.getHadoopConfiguration(HadoopUtils.java:113) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT] at org.apache.paimon.catalog.CatalogContext.(CatalogContext.java:53) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT] at org.apache.paimon.catalog.CatalogContext.create(CatalogContext.java:73) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT] at org.apache.paimon.flink.FlinkCatalogFactory.createPaimonCatalog(FlinkCatalogFactory.java:81) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT] at org.apache.paimon.flink.action.ActionBase.initPaimonCatalog(ActionBase.java:78) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT] at org.apache.paimon.flink.action.ActionBase.(ActionBase.java:65) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT] at org.apache.paimon.flink.action.cdc.SynchronizationActionBase.(SynchronizationActionBase.java:77) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT] at org.apache.paimon.flink.action.cdc.SyncDatabaseActionBase.(SyncDatabaseActionBase.java:68) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT] at org.apache.paimon.flink.action.cdc.mysql.MySqlSyncDatabaseAction.(MySqlSyncDatabaseAction.java:109) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT] at org.apache.paimon.flink.action.cdc.mysql.MySqlSyncDatabaseActionFactory.createAction(MySqlSyncDatabaseActionFactory.java:52) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT] at org.apache.paimon.flink.action.cdc.mysql.MySqlSyncDatabaseActionFactory.createAction(MySqlSyncDatabaseActionFactory.java:31) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT] at org.apache.paimon.flink.action.cdc.SynchronizationActionFactoryBase.create(SynchronizationActionFactoryBase.java:45) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT] at org.apache.paimon.flink.action.cdc.SyncDatabaseActionFactoryBase.create(SyncDatabaseActionFactoryBase.java:47) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT] at org.apache.paimon.flink.action.ActionFactory.createAction(ActionFactory.java:87) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT] at org.apache.paimon.flink.action.FlinkActions.main(FlinkActions.java:43) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_252] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_252] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_252] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_252] at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) ~[flink-dist-1.18.0.jar:1.18.0] at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222) ~[flink-dist-1.18.0.jar:1.18.0] at org.apache.flink.client.program.PackagedProgramUtils.getPipelineFromProgram(PackagedProgramUtils.java:145) ~[dinky-app-1.18-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.18.0] at org.dinky.trans.dml.ExecuteJarOperation.getStreamGraph(ExecuteJarOperation.java:118) ~[dinky-app-1.18-1.2.0-SNAPSHOT-jar-with-dependencies.jar:?] at org.dinky.trans.dml.ExecuteJarOperation.getStreamGraph(ExecuteJarOperation.java:84) ~[dinky-app-1.18-1.2.0-SNAPSHOT-jar-with-dependencies.jar:?] at org.dinky.app.flinksql.Submitter.executeJarJob(Submitter.java:263) ~[dinky-app-1.18-1.2.0-SNAPSHOT-jar-with-dependencies.jar:?] at org.dinky.app.flinksql.Submitter.submit(Submitter.java:142) ~[dinky-app-1.18-1.2.0-SNAPSHOT-jar-with-dependencies.jar:?] at org.dinky.app.MainApp.main(MainApp.java:53) ~[dinky-app-1.18-1.2.0-SNAPSHOT-jar-with-dependencies.jar:?] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_252] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_252] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_252] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_252] at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) ~[flink-dist-1.18.0.jar:1.18.0]

What you expected to happen

no error

How to reproduce

1、Set dinky data source paimon to HDFS and specify the hdfs configuration path 2、run paimon with flink application job on dinky

Anything else

No response

Version

dev

Are you willing to submit PR?

Code of Conduct

github-actions[bot] commented 6 days ago

Hello @dsanww, this issue is about CDC/CDCSOURCE, so I assign it to @aiwenmo. If you have any questions, you can comment and reply.

你好 @dsanww, 这个 issue 是关于 CDC/CDCSOURCE 的，所以我把它分配给了 @aiwenmo。如有任何问题，可以评论回复。