search

apache / doris

Apache Doris is an easy-to-use, high performance and unified analytics database.
https://doris.apache.org
Apache License 2.0
12.79k stars 3.29k forks source link

[Bug] doris query hudi Could not initialize class org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex #34572

Open xiaofan2022 opened 6 months ago

xiaofan2022 commented 6 months ago

Search before asking

Version

hudi 0.14.1 doris 2.0.9 hbase 2.5.6

What's Wrong?

query hudi error，fe.out Exception in thread "ExternalMetaCacheMgr-0" java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/unsafe/HBasePlatformDependent at org.apache.hadoop.hbase.util.Bytes.(Bytes.java:130) at org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex.(HFileBootstrapIndex.java:95) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hudi.common.util.ReflectionUtils.getClass(ReflectionUtils.java:55) at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:79) at org.apache.hudi.common.bootstrap.index.BootstrapIndex.getBootstrapIndex(BootstrapIndex.java:163) at org.apache.hudi.common.table.view.AbstractTableFileSystemView.init(AbstractTableFileSystemView.java:118) at org.apache.hudi.common.table.view.HoodieTableFileSystemView.init(HoodieTableFileSystemView.java:113) at org.apache.hudi.common.table.view.HoodieTableFileSystemView.(HoodieTableFileSystemView.java:107) at org.apache.hudi.common.table.view.HoodieTableFileSystemView.(HoodieTableFileSystemView.java:98) at org.apache.hudi.common.table.view.HoodieTableFileSystemView.(HoodieTableFileSystemView.java:176) at org.apache.doris.planner.external.hudi.HudiScanNode.lambda$null$5(HudiScanNode.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.unsafe.HBasePlatformDependent at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 16 more Exception in thread "ExternalMetaCacheMgr-2" java.lang.NoClassDefFoundError: Could not initialize class org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hudi.common.util.ReflectionUtils.getClass(ReflectionUtils.java:55) at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:79) at org.apache.hudi.common.bootstrap.index.BootstrapIndex.getBootstrapIndex(BootstrapIndex.java:163) at org.apache.hudi.common.table.view.AbstractTableFileSystemView.init(AbstractTableFileSystemView.java:118) at org.apache.hudi.common.table.view.HoodieTableFileSystemView.init(HoodieTableFileSystemView.java:113) at org.apache.hudi.common.table.view.HoodieTableFileSystemView.(HoodieTableFileSystemView.java:107) at org.apache.hudi.common.table.view.HoodieTableFileSystemView.(HoodieTableFileSystemView.java:98) at org.apache.hudi.common.table.view.HoodieTableFileSystemView.(HoodieTableFileSystemView.java:176) at org.apache.doris.planner.external.hudi.HudiScanNode.lambda$null$5(HudiScanNode.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Exception in thread "ExternalMetaCacheMgr-4" java.lang.NoClassDefFoundError: Could not initialize class org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hudi.common.util.ReflectionUtils.getClass(ReflectionUtils.java:55) at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:79) at org.apache.hudi.common.bootstrap.index.BootstrapIndex.getBootstrapIndex(BootstrapIndex.java:163) at org.apache.hudi.common.table.view.AbstractTableFileSystemView.init(AbstractTableFileSystemView.java:118) at org.apache.hudi.common.table.view.HoodieTableFileSystemView.init(HoodieTableFileSystemView.java:113) at org.apache.hudi.common.table.view.HoodieTableFileSystemView.(HoodieTableFileSystemView.java:107) at org.apache.hudi.common.table.view.HoodieTableFileSystemView.(HoodieTableFileSystemView.java:98) at org.apache.hudi.common.table.view.HoodieTableFileSystemView.(HoodieTableFileSystemView.java:176) at org.apache.doris.planner.external.hudi.HudiScanNode.lambda$null$5(HudiScanNode.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Exception in thread "ExternalMetaCacheMgr-6" java.lang.NoClassDefFoundError: Could not initialize class org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hudi.common.util.ReflectionUtils.getClass(ReflectionUtils.java:55) at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:79) at org.apache.hudi.common.bootstrap.index.BootstrapIndex.getBootstrapIndex(BootstrapIndex.java:163) at org.apache.hudi.common.table.view.AbstractTableFileSystemView.init(AbstractTableFileSystemView.java:118) at org.apache.hudi.common.table.view.HoodieTableFileSystemView.init(HoodieTableFileSystemView.java:113) at org.apache.hudi.common.table.view.HoodieTableFileSystemView.(HoodieTableFileSystemView.java:107) at org.apache.hudi.common.table.view.HoodieTableFileSystemView.(HoodieTableFileSystemView.java:98) at org.apache.hudi.common.table.view.HoodieTableFileSystemView.(HoodieTableFileSystemView.java:176) at org.apache.doris.planner.external.hudi.HudiScanNode.lambda$null$5(HudiScanNode.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Exception in thread "ExternalMetaCacheMgr-8" java.lang.NoClassDefFoundError: Could not initialize class org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hudi.common.util.ReflectionUtils.getClass(ReflectionUtils.java:55) at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:79) at org.apache.hudi.common.bootstrap.index.BootstrapIndex.getBootstrapIndex(BootstrapIndex.java:163) at org.apache.hudi.common.table.view.AbstractTableFileSystemView.init(AbstractTableFileSystemView.java:118) at org.apache.hudi.common.table.view.HoodieTableFileSystemView.init(HoodieTableFileSystemView.java:113) at org.apache.hudi.common.table.view.HoodieTableFileSystemView.(HoodieTableFileSystemView.java:107) at org.apache.hudi.common.table.view.HoodieTableFileSystemView.(HoodieTableFileSystemView.java:98) at org.apache.hudi.common.table.view.HoodieTableFileSystemView.(HoodieTableFileSystemView.java:176) at org.apache.doris.planner.external.hudi.HudiScanNode.lambda$null$5(HudiScanNode.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Exception in thread "ExternalMetaCacheMgr-10" java.lang.NoClassDefFoundError: Could not initialize class org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hudi.common.util.ReflectionUtils.getClass(ReflectionUtils.java:55) at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:79) at org.apache.hudi.common.bootstrap.index.BootstrapIndex.getBootstrapIndex(BootstrapIndex.java:163) at org.apache.hudi.common.table.view.AbstractTableFileSystemView.init(AbstractTableFileSystemView.java:118) at org.apache.hudi.common.table.view.HoodieTableFileSystemView.init(HoodieTableFileSystemView.java:113) at org.apache.hudi.common.table.view.HoodieTableFileSystemView.(HoodieTableFileSystemView.java:107) at org.apache.hudi.common.table.view.HoodieTableFileSystemView.(HoodieTableFileSystemView.java:98) at org.apache.hudi.common.table.view.HoodieTableFileSystemView.(HoodieTableFileSystemView.java:176) at org.apache.doris.planner.external.hudi.HudiScanNode.lambda$null$5(HudiScanNode.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Exception in thread "ExternalMetaCacheMgr-12" java.lang.NoClassDefFoundError: Could not initialize class org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hudi.common.util.ReflectionUtils.getClass(ReflectionUtils.java:55) at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:79) at org.apache.hudi.common.bootstrap.index.BootstrapIndex.getBootstrapIndex(BootstrapIndex.java:163) at org.apache.hudi.common.table.view.AbstractTableFileSystemView.init(AbstractTableFileSystemView.java:118) at org.apache.hudi.common.table.view.HoodieTableFileSystemView.init(HoodieTableFileSystemView.java:113) at org.apache.hudi.common.table.view.HoodieTableFileSystemView.(HoodieTableFileSystemView.java:107) at org.apache.hudi.common.table.view.HoodieTableFileSystemView.(HoodieTableFileSystemView.java:98) at org.apache.hudi.common.table.view.HoodieTableFileSystemView.(HoodieTableFileSystemView.java:176) at org.apache.doris.planner.external.hudi.HudiScanNode.lambda$null$5(HudiScanNode.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Exception in thread "ExternalMetaCacheMgr-14" java.lang.NoClassDefFoundError: Could not initialize class org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hudi.common.util.ReflectionUtils.getClass(ReflectionUtils.java:55) at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:79) at org.apache.hudi.common.bootstrap.index.BootstrapIndex.getBootstrapIndex(BootstrapIndex.java:163) at org.apache.hudi.common.table.view.AbstractTableFileSystemView.init(AbstractTableFileSystemView.java:118) at org.apache.hudi.common.table.view.HoodieTableFileSystemView.init(HoodieTableFileSystemView.java:113) at org.apache.hudi.common.table.view.HoodieTableFileSystemView.(HoodieTableFileSystemView.java:107) at org.apache.hudi.common.table.view.HoodieTableFileSystemView.(HoodieTableFileSystemView.java:98) at org.apache.hudi.common.table.view.HoodieTableFileSystemView.(HoodieTableFileSystemView.java:176) at org.apache.doris.planner.external.hudi.HudiScanNode.lambda$null$5(HudiScanNode.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

What You Expected?

Use the catalog to check hudi usage

How to Reproduce?

use catalog query hudi data

Anything Else?

No response

Are you willing to submit PR?

Code of Conduct

xiaofan2022 commented 6 months ago

I found that doris version 2.0.9 hbase was upgraded from 2.4.9 to 2.5.8. 2.5.8 hbase-common relies on hbase-unsafe.jar, but the package is not available in doris_fe lib

yxfff commented 5 months ago

I found that doris version 2.0.9 hbase was upgraded from 2.4.9 to 2.5.8. 2.5.8 hbase-common relies on hbase-unsafe.jar, but the package is not available in doris_fe lib

I add habse-unsafe-4.1.7.jar in fe/lib but it don't work