[SUPPORT] Compatible with multiple HBASE version or hbase: 2.1.0-cdh6.3.2

[meitianjinbu]

Describe the problem you faced

version: 0.11.0-rc2 hbase: 2.1.0-cdh6.3.2 only support HBASE 2.4.9+ version

Expected behavior

Compatible with hbase: 2.1.0-cdh6.3.2

[codope]

@meitianjinbu hbase 2.4.x should generally be compatible with 2.1.x ? Is the Cloudera version of Hbase different from the open source hbase? Can you share the error stacktrace if any?

[yihua]

@meitianjinbu Following up on this, the Hudi bundles have shaded HBase-related classes so the bundle should work. In what flow do you see compatibility issues, e.g., writing, querying, SparkHoodieHBaseIndex, etc.? Could you also provide reproducible steps and error stacktrace if there is any?

[CodeCooker17]

@meitianjinbu Following up on this, the Hudi bundles have shaded HBase-related classes so the bundle should work. In what flow do you see compatibility issues, e.g., writing, querying, SparkHoodieHBaseIndex, etc.? Could you also provide reproducible steps and error stacktrace if there is any?

The default value of hbase.defaults.for.version.skip in hbase-site.xml is false. This will cause the following exception to occur once the local hbase version is lower than 2.4.9. So it would be better if it is set to true by default. Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238) ... 21 more Caused by: java.lang.ExceptionInInitializerError at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:658) ... 26 more Caused by: java.lang.RuntimeException: hbase-default.xml file seems to be for an older version of HBase (2.4.9), this version is 1.3.0 at org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:71) at org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:81) at org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:96) at org.apache.hadoop.hbase.client.ConnectionManager.<clinit>(ConnectionManager.java:235)

[yihua]

@CodeCooker17 The error stacktrace shows that this version is 1.3.0, which is different from 2.1.0-cdh6.3.2. Could you check if this is due to other version of HBase library introduced? You may also try to set hbase.defaults.for.version.skip to true and see if that helps you to resolve the compatibility issue. Generally, you need to make sure there is no HBase config conflicts between HBase 1.x and 2.x that can cause other issues after skipping the version check. Let me know what you find.

[blackburt342865189]

I also encountered this problem：

2022-05-08 09:04:30,885 WARN org.apache.hive.service.server.HiveServer2: [main]: Error starting HiveServer2 on attempt 1, will retry in 60000ms java.lang.RuntimeException: hbase-default.xml file seems to be for an older version of HBase (2.4.9), this version is 2.1.0-cdh6.3.4 at org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:70) ~[hbase-common-2.1.0-cdh6.3.4.jar:?] at org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:80) ~[hbase-common-2.1.0-cdh6.3.4.jar:?] at org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.initiateSparkConf(HiveSparkClientFactory.java:83) ~[hive-exec-2.1.1-cdh6.3.4.jar:2.1.1-cdh6.3.4] at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionManagerImpl.setup(SparkSessionManagerImpl.java:98) ~[hive-exec-2.1.1-cdh6.3.4.jar:2.1.1-cdh6.3.4] at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:595) [hive-service-2.1.1-cdh6.3.4.jar:2.1.1-cdh6.3.4] at org.apache.hive.service.server.HiveServer2.access$700(HiveServer2.java:94) [hive-service-2.1.1-cdh6.3.4.jar:2.1.1-cdh6.3.4] at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:827) [hive-service-2.1.1-cdh6.3.4.jar:2.1.1-cdh6.3.4] at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:696) [hive-service-2.1.1-cdh6.3.4.jar:2.1.1-cdh6.3.4] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_281] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_281] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_281] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_281] at org.apache.hadoop.util.RunJar.run(RunJar.java:313) [hadoop-common-3.0.0-cdh6.3.4.jar:?] at org.apache.hadoop.util.RunJar.main(RunJar.java:227) [hadoop-common-3.0.0-cdh6.3.4.jar:?]

[danny0405]

I also encountered this problem：

2022-05-08 09:04:30,885 WARN org.apache.hive.service.server.HiveServer2: [main]: Error starting HiveServer2 on attempt 1, will retry in 60000ms java.lang.RuntimeException: hbase-default.xml file seems to be for an older version of HBase (2.4.9), this version is 2.1.0-cdh6.3.4 at org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:70) ~[hbase-common-2.1.0-cdh6.3.4.jar:?] at org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:80) ~[hbase-common-2.1.0-cdh6.3.4.jar:?] at org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.initiateSparkConf(HiveSparkClientFactory.java:83) ~[hive-exec-2.1.1-cdh6.3.4.jar:2.1.1-cdh6.3.4] at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionManagerImpl.setup(SparkSessionManagerImpl.java:98) ~[hive-exec-2.1.1-cdh6.3.4.jar:2.1.1-cdh6.3.4] at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:595) [hive-service-2.1.1-cdh6.3.4.jar:2.1.1-cdh6.3.4] at org.apache.hive.service.server.HiveServer2.access$700(HiveServer2.java:94) [hive-service-2.1.1-cdh6.3.4.jar:2.1.1-cdh6.3.4] at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:827) [hive-service-2.1.1-cdh6.3.4.jar:2.1.1-cdh6.3.4] at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:696) [hive-service-2.1.1-cdh6.3.4.jar:2.1.1-cdh6.3.4] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_281] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_281] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_281] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_281] at org.apache.hadoop.util.RunJar.run(RunJar.java:313) [hadoop-common-3.0.0-cdh6.3.4.jar:?] at org.apache.hadoop.util.RunJar.main(RunJar.java:227) [hadoop-common-3.0.0-cdh6.3.4.jar:?]

The stacktrace throws from Hive on Spark ?

[blackburt342865189]

yes,hive on spark, put hudi-hadoop-mr-bundle-0.12.0-SNAPSHOT into auxlib/, after restart hiveserver2,encountered ,hiveserver2 donot Normal woking.

[nsivabalan]

hey @meitianjinbu : are we still looking for any assistance on this regard. btw, we added an FAQ on hbase conflicting w/ metadata table https://hudi.apache.org/docs/faq#how-can-i-resolve-the-runtimeexception-saying-hbase-defaultxml-file-seems-to-be-for-an-older-version-of-hbase let us know if the FAQ helps you.

feel free to close the issue if you get it resolved.

[cnfree0355]

Build hudi 0.13 with cdh 2.1.0-cdh6.3.2 error

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile (default-compile) on project hudi-common: Compilation failure: Compilation failure: [ERROR] /Users/apple/Documents/Java_Projects/hudi/hudi-common/src/main/java/org/apache/hudi/io/storage/HoodieAvroHFileReader.java:[48,40] cannot find symbol [ERROR] symbol: class HFileInfo [ERROR] location: package org.apache.hadoop.hbase.io.hfile [ERROR] /Users/apple/Documents/Java_Projects/hudi/hudi-common/src/main/java/org/apache/hudi/io/storage/HoodieHFileUtils.java:[29,40] cannot find symbol [ERROR] symbol: class HFileInfo [ERROR] location: package org.apache.hadoop.hbase.io.hfile [ERROR] /Users/apple/Documents/Java_Projects/hudi/hudi-common/src/main/java/org/apache/hudi/io/storage/HoodieHFileUtils.java:[30,40] cannot find symbol [ERROR] symbol: class ReaderContext [ERROR] location: package org.apache.hadoop.hbase.io.hfile [ERROR] /Users/apple/Documents/Java_Projects/hudi/hudi-common/src/main/java/org/apache/hudi/io/storage/HoodieHFileUtils.java:[31,40] cannot find symbol [ERROR] symbol: class ReaderContextBuilder [ERROR] location: package org.apache.hadoop.hbase.io.hfile [ERROR] -> [Help 1]

Beacuse in hbase version 2.1.0-cdh6.3.2 with hbase-common-2.1.0-cdh6.3.2.jar which dose not include the org.apache.hadoop.hbase.io.hfile.{HFileInfo,ReaderContext,ReaderContextBuilder} class ,So it build failed