search

apache / hudi

Upserts, Deletes And Incremental Processing on Big Data.
https://hudi.apache.org/
Apache License 2.0
5.24k stars 2.39k forks source link

Hudi timelineserver doesn't support kerberos authenticated file system #10762

Closed ligou525 closed 4 months ago

ligou525 commented 4 months ago

For hudi table using file system with kerberos authentication, the timeline server handler doesn't work. Handlers of timeline server doesn't consider the authenticated filesystem, authentication exception is thrown when creating HoodieTableMetaClient as creating the filesystem: public SyncableFileSystemView getFileSystemView(String basePath) { return globalViewMap.computeIfAbsent(basePath, (path) -> { HoodieTableMetaClient metaClient = HoodieTableMetaClient.builder().setConf(conf.newCopy()).setBasePath(path).build(); return viewCreator.apply(metaClient, viewStorageConfig); }); }

Expected behavior

timeline server works well for kerberos-authenticated filesystem

Environment Description

Additional context

Stacktrace

24/02/19 21:57:12 INFO view.RemoteHoodieTableFileSystemView: Sending request : (http://172.16.48.4:27329/v1/hoodie/view/compactions/pending/?basepath=hdfs%3A%2F%2Fhdfs_host%3A8020%2Fuser%2Fhdfs%2Ftest_location%2Ftest_decimal_hdfs&lastinstantts=20240219215711252&timeline hash=4a679c704ec969a1ad4b2f434ac012ff10005ba03948c375356913f52e6d8ed4) 24/02/19 21:57:12 INFO table.HoodieTableMetaClient: Loading HoodieTableMetaClient from hdfs://172.16.48.6:8020/user/hdfs/test_location/test_decimal_hdfs 24/02/19 21:57:12 ERROR service.RequestHandler: Got runtime exception servicing request basepath=hdfs%3A%2F%2F2Fhdfs_host%3A8020%2Fuser%2Fhdfs%2Ftest_location%2Ftest_decimal_hdfs&lastinstantts=20240219215711252&timelinehash=4a679c704ec969a1ad4b2f434ac012ff10005ba03948c375 356913f52e6d8ed4 org.apache.hudi.exception.HoodieIOException: Could not check if hdfs://2Fhdfs_host:8020/user/hdfs/test_location/test_decimal_hdfs is a valid table at org.apache.hudi.exception.TableNotFoundException.checkTableValidity(TableNotFoundException.java:59) at org.apache.hudi.common.table.HoodieTableMetaClient.(HoodieTableMetaClient.java:124) at org.apache.hudi.common.table.HoodieTableMetaClient.(HoodieTableMetaClient.java:78) at org.apache.hudi.common.table.HoodieTableMetaClient$Builder.build(HoodieTableMetaClient.java:670) at org.apache.hudi.common.table.view.FileSystemViewManager.lambda$getFileSystemView$0(FileSystemViewManager.java:98) at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660) at org.apache.hudi.common.table.view.FileSystemViewManager.getFileSystemView(FileSystemViewManager.java:97) at org.apache.hudi.timeline.service.RequestHandler.isLocalViewBehind(RequestHandler.java:125) at org.apache.hudi.timeline.service.RequestHandler.syncIfLocalViewBehind(RequestHandler.java:153) at org.apache.hudi.timeline.service.RequestHandler.access$100(RequestHandler.java:66) at org.apache.hudi.timeline.service.RequestHandler$ViewHandler.handle(RequestHandler.java:495) at io.javalin.security.SecurityUtil.noopAccessManager(SecurityUtil.kt:22) at io.javalin.Javalin.lambda$addHandler$0(Javalin.java:606) at io.javalin.core.JavalinServlet$service$2$1.invoke(JavalinServlet.kt:46) at io.javalin.core.JavalinServlet$service$2$1.invoke(JavalinServlet.kt:17) at io.javalin.core.JavalinServlet$service$1.invoke(JavalinServlet.kt:143) at io.javalin.core.JavalinServlet$service$2.invoke(JavalinServlet.kt:41) at io.javalin.core.JavalinServlet.service(JavalinServlet.kt:107) at io.javalin.core.util.JettyServerUtil$initialize$httpHandler$1.doHandle(JettyServerUtil.kt:72) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1668) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:61) at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:174) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:502) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: DestHost:destPort 2Fhdfs_host:8020 , LocalHost:localPort core-25c178a-2/2Fclient_host:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:842) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:817) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1583) at org.apache.hadoop.ipc.Client.call(Client.java:1525) at org.apache.hadoop.ipc.Client.call(Client.java:1422) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:231) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)

danny0405 commented 4 months ago

Hi, is the issue already resoved? Can you share your experience here?