Found during the demo bumps in preparation of 24.11.
When trying to bump HMS from 3.1.3 to 4.0.0 the schematool fails. This prevents probably all our customers to upgrade to HMS 4.0.0
+ echo copying /stackable/mount/hdfs-config to /stackable/config
copying /stackable/mount/hdfs-config to /stackable/config
+ cp -RL /stackable/mount/hdfs-config/core-site.xml /stackable/mount/hdfs-config/hdfs-site.xml /stackable/config
replacing xxx_db_username_xxx and xxx_db_password_xxx with secret values.
+ echo replacing xxx_db_username_xxx and xxx_db_password_xxx with secret values.
+ sed -i 's|xxx_db_username_xxx|hive|g' /stackable/config/hive-site.xml
+ sed -i 's|xxx_db_password_xxx|hive|g' /stackable/config/hive-site.xml
++ grep -oP 'default_realm = \K.*' /stackable/kerberos/krb5.conf
+ export KERBEROS_REALM=KNAB.COM
+ KERBEROS_REALM=KNAB.COM
+ sed -i -e 's/${env.KERBEROS_REALM}/KNAB.COM/g' /stackable/config/hive-site.xml
+ sed -i -e 's/${env.KERBEROS_REALM}/KNAB.COM/g' /stackable/config/core-site.xml
+ sed -i -e 's/${env.KERBEROS_REALM}/KNAB.COM/g' /stackable/config/hdfs-site.xml
+ rm -f /stackable/log/_vector/shutdown
+ prepare_signal_handlers
+ unset term_child_pid
+ unset term_kill_needed
+ trap handle_term_signal TERM
+ wait_for_termination 31
+ set +e
+ term_child_pid=31
+ [[ -v term_kill_needed ]]
+ wait 31
+ bin/start-metastore --config /stackable/config --db-type postgres --hive-bin-dir bin
processing arg --config
processing arg --db-type
processing arg --hive-bin-dir
processing arg
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/stackable/apache-hive-metastore-4.0.0-bin/lib/log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/stackable/hadoop-3.3.6/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
2024-11-15T09:07:07,132 INFO [main] conf.MetastoreConf: Found configuration file: file:/stackable/config/hive-site.xml
2024-11-15T09:07:07,417 INFO [main] conf.MetastoreConf: Unable to find config file: hivemetastore-site.xml
2024-11-15T09:07:07,419 INFO [main] conf.MetastoreConf: Found configuration file: file:/stackable/apache-hive-metastore-4.0.0-bin/conf/metastore-site.xml
Starting metastore validation
Validating schema version
[FAIL]
Metastore schema version is not compatible. Hive Version: 4.0.0, Database Schema Version: 3.1.0
Validating sequence number for SEQUENCE_TABLE
[SUCCESS]
Validating metastore schema tables
[SUCCESS]
Validating DFS locations
[SUCCESS]
Validating columns for incorrect NULL values.
[SUCCESS]
Done with metastore validation: [FAIL]
2024-11-15T09:07:08,192 ERROR [main] schematool.MetastoreSchemaTool: Validation failed
Validation failed
2024-11-15T09:07:08,193 ERROR [main] schematool.MetastoreSchemaTool: Use --verbose for detailed stacktrace.
Use --verbose for detailed stacktrace.
2024-11-15T09:07:08,193 ERROR [main] schematool.MetastoreSchemaTool: *** schemaTool failed ***
*** schemaTool failed ***
No valid schema found, initializing schema ...
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/stackable/apache-hive-metastore-4.0.0-bin/lib/log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/stackable/hadoop-3.3.6/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
2024-11-15T09:07:11,626 INFO [main] conf.MetastoreConf: Found configuration file: file:/stackable/config/hive-site.xml
2024-11-15T09:07:11,888 INFO [main] conf.MetastoreConf: Unable to find config file: hivemetastore-site.xml
2024-11-15T09:07:11,890 INFO [main] conf.MetastoreConf: Found configuration file: file:/stackable/apache-hive-metastore-4.0.0-bin/conf/metastore-site.xml
Initializing the schema to: 4.0.0
2024-11-15T09:07:11,940 INFO [main] schematool.HiveSchemaHelper: Metastore connection URL: jdbc:postgresql://postgresql-hive-iceberg:5432/hive
Metastore connection URL: jdbc:postgresql://postgresql-hive-iceberg:5432/hive
2024-11-15T09:07:11,940 INFO [main] schematool.HiveSchemaHelper: Metastore connection Driver : org.postgresql.Driver
Metastore connection Driver : org.postgresql.Driver
2024-11-15T09:07:11,940 INFO [main] schematool.HiveSchemaHelper: Metastore connection User: hive
Metastore connection User: hive
Starting metastore schema initialization to 4.0.0
Initialization script hive-schema-4.0.0.postgres.sql
2024-11-15T09:07:12,417 INFO [main] schematool.MetastoreSchemaTool: Going to run command <-u jdbc:postgresql://postgresql-hive-iceberg:5432/hive -d org.postgresql.Driver -n hive -p [passwd stripped] --isolation=TRANSACTION_READ_COMMITTED -f /stackable/apache-hive-metastore-4.0.0-bin/scripts/metastore/upgrade/postgres/hive-schema-4.0.0.postgres.sql>
Nov 15, 2024 9:07:12 AM org.jline.utils.Log logr
WARNING: Unable to create a system terminal, creating a dumb terminal (enable debug logging for more information)
1/706 --
2/706 -- PostgreSQL database dump
3/706 --
4/706 SET statement_timeout = 0;
No rows affected (0.001 seconds)
5/706 SET client_encoding = 'UTF8';
No rows affected (0.001 seconds)
6/706 SET standard_conforming_strings = off;
No rows affected (0.001 seconds)
7/706 SET check_function_bodies = false;
No rows affected (0.001 seconds)
8/706 SET client_min_messages = warning;
No rows affected (0.035 seconds)
9/706 SET escape_string_warning = off;
No rows affected (0.001 seconds)
10/706 SET search_path = public, pg_catalog;
No rows affected (0.001 seconds)
11/706 SET default_tablespace = '';
No rows affected (0.001 seconds)
12/706 SET default_with_oids = false;
No rows affected (0 seconds)
13/706 --
14/706 -- Name: BUCKETING_COLS; Type: TABLE; Schema: public; Owner: hiveuser; Tablespace:
15/706 --
16/706 CREATE TABLE "BUCKETING_COLS" (
"SD_ID" bigint NOT NULL,
"BUCKET_COL_NAME" character varying(256) DEFAULT NULL::character varying,
"INTEGER_IDX" bigint NOT NULL
);
Error: ERROR: relation "BUCKETING_COLS" already exists (state=42P07,code=0)
org.postgresql.util.PSQLException: ERROR: relation "BUCKETING_COLS" already exists
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2676)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2366)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:356)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:496)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:413)
at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:333)
at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:319)
at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:295)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:290)
at sqlline.Commands.executeSingleQuery(Commands.java:1054)
at sqlline.Commands.execute(Commands.java:1003)
at sqlline.Commands.sql(Commands.java:967)
at sqlline.SqlLine.dispatch(SqlLine.java:734)
at sqlline.SqlLine.runCommands(SqlLine.java:1736)
at sqlline.Commands.run(Commands.java:1593)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38)
at sqlline.SqlLine.dispatch(SqlLine.java:730)
at sqlline.SqlLine.initArgs(SqlLine.java:459)
at sqlline.SqlLine.begin(SqlLine.java:515)
at org.apache.hadoop.hive.metastore.tools.schematool.MetastoreSchemaTool.execSql(MetastoreSchemaTool.java:326)
at org.apache.hadoop.hive.metastore.tools.schematool.MetastoreSchemaTool.execSql(MetastoreSchemaTool.java:303)
at org.apache.hadoop.hive.metastore.tools.schematool.SchemaToolTaskInit.execute(SchemaToolTaskInit.java:62)
at org.apache.hadoop.hive.metastore.tools.schematool.MetastoreSchemaTool.run(MetastoreSchemaTool.java:484)
at org.apache.hadoop.hive.metastore.tools.schematool.MetastoreSchemaTool.run(MetastoreSchemaTool.java:429)
at org.apache.hadoop.hive.metastore.tools.schematool.MetastoreSchemaTool.main(MetastoreSchemaTool.java:425)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.hadoop.util.RunJar.run(RunJar.java:328)
at org.apache.hadoop.util.RunJar.main(RunJar.java:241)
Aborting command set because "force" is false and command failed: "CREATE TABLE "BUCKETING_COLS" (
"SD_ID" bigint NOT NULL,
"BUCKET_COL_NAME" character varying(256) DEFAULT NULL::character varying,
"INTEGER_IDX" bigint NOT NULL
);"
2024-11-15T09:07:13,094 ERROR [main] schematool.MetastoreSchemaTool: Schema initialization FAILED! Metastore state would be inconsistent!
Schema initialization FAILED! Metastore state would be inconsistent!
2024-11-15T09:07:13,095 ERROR [main] schematool.MetastoreSchemaTool: Underlying cause: java.io.IOException : Schema script failed, errorcode OTHER
Underlying cause: java.io.IOException : Schema script failed, errorcode OTHER
2024-11-15T09:07:13,095 ERROR [main] schematool.MetastoreSchemaTool: Use --verbose for detailed stacktrace.
Use --verbose for detailed stacktrace.
2024-11-15T09:07:13,096 ERROR [main] schematool.MetastoreSchemaTool: *** schemaTool failed ***
*** schemaTool failed ***
+ trap - TERM
+ wait 31
+ set -e
+ mkdir -p /stackable/log/_vector
+ touch /stackable/log/_vector/shutdown
Affected Stackable version
0.0.0-dev
Affected Apache Hive version
3.1.3 -> 4.0.0
Current and expected behavior
Found during the demo bumps in preparation of 24.11. When trying to bump HMS from 3.1.3 to 4.0.0 the schematool fails. This prevents probably all our customers to upgrade to HMS 4.0.0
Possible solution
No response
Additional context
No response
Environment
No response
Would you like to work on fixing this bug?
None