search

liquibase / liquibase-databricks

Apache License 2.0
26 stars 7 forks source link

Liquibase fails to initialize databasechangeloglock table in Azure Databricks #98

Open BillCM opened 11 months ago

BillCM commented 11 months ago
  1. run liquibase --changeLogFile=changelog.databricks.sql generateChangeLog: success
  2. add create view to the changelog
  3. run liquibase --changeLogFile=changelog.databricks.sql update
    • liquibase creates databasechangelog
    • liquibase creates databasechangeloglock
    • but it fails to initialize the table, leading to 10 retries and eventual failure
[2024-01-05 16:36:32] FINE [liquibase.executor] Create Database Lock Table
[2024-01-05 16:36:32] FINE [liquibase.executor] CREATE TABLE development_sandbox.test.DATABASECHANGELOGLOCK (ID INT NOT NULL, LOCKED BOOLEAN NOT NULL, LOCKGRANTED TIMESTAMP, LOCKEDBY VARCHAR(255), CONSTRAINT PK_DATABASECHANGELOGLOCK PRIMARY KEY (ID))
[2024-01-05 16:36:35] FINE [liquibase.executor] -1 row(s) affected
[2024-01-05 16:36:35] FINE [liquibase.executor] Changelog query completed.
[2024-01-05 16:36:35] FINE [liquibase.lockservice] Created database lock table with name: development_sandbox.test.DATABASECHANGELOGLOCK
[2024-01-05 16:36:36] FINE [liquibase.lockservice] Failed to create or initialize the lock table, trying again, iteration 1 of 10
liquibase.exception.UnexpectedLiquibaseException: liquibase.exception.DatabaseException: liquibase.exception.DatabaseException: Error executing SQL SELECT COUNT(*) FROM development_sandbox.test.DATABASECHANGELOGLOCK: [Databricks][DatabricksJDBCDriver](500618) Error occured while deserializing arrow data: sun.misc.Unsafe or java.nio.DirectByteBuffer.<init>(long, int) not available
    at liquibase.lockservice.StandardLockService.isDatabaseChangeLogLockTableInitialized(StandardLockService.java:219)
    at liquibase.lockservice.StandardLockService.init(StandardLockService.java:135)
    at liquibase.command.core.helpers.DatabaseChangelogCommandStep.checkLiquibaseTables(DatabaseChangelogCommandStep.java:145)
    at liquibase.command.core.helpers.DatabaseChangelogCommandStep.run(DatabaseChangelogCommandStep.java:91)
    at liquibase.command.CommandScope.execute(CommandScope.java:217)
    at liquibase.integration.commandline.CommandRunner.call(CommandRunner.java:55)
    at liquibase.integration.commandline.CommandRunner.call(CommandRunner.java:24)
    at picocli.CommandLine.executeUserObject(CommandLine.java:2041)
    at picocli.CommandLine.access$1500(CommandLine.java:148)
    at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461)
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2453)
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2415)
    at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2273)
    at picocli.CommandLine$RunLast.execute(CommandLine.java:2417)
    at picocli.CommandLine.execute(CommandLine.java:2170)
    at liquibase.integration.commandline.LiquibaseCommandLine.lambda$execute$2(LiquibaseCommandLine.java:391)
    at liquibase.Scope.child(Scope.java:195)
    at liquibase.Scope.child(Scope.java:171)
    at liquibase.integration.commandline.LiquibaseCommandLine.lambda$execute$3(LiquibaseCommandLine.java:366)
    at liquibase.Scope.child(Scope.java:195)
    at liquibase.Scope.child(Scope.java:171)
    at liquibase.integration.commandline.LiquibaseCommandLine.execute(LiquibaseCommandLine.java:363)
    at liquibase.integration.commandline.LiquibaseCommandLine.main(LiquibaseCommandLine.java:103)
    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
    at java.base/java.lang.reflect.Method.invoke(Method.java:580)
    at liquibase.integration.commandline.LiquibaseLauncher.main(LiquibaseLauncher.java:116)
Caused by: liquibase.exception.DatabaseException: liquibase.exception.DatabaseException: Error executing SQL SELECT COUNT(*) FROM development_sandbox.test.DATABASECHANGELOGLOCK: [Databricks][DatabricksJDBCDriver](500618) Error occured while deserializing arrow data: sun.misc.Unsafe or java.nio.DirectByteBuffer.<init>(long, int) not available
    at liquibase.executor.jvm.ChangelogJdbcMdcListener.query(ChangelogJdbcMdcListener.java:60)
    at liquibase.lockservice.StandardLockService.isDatabaseChangeLogLockTableInitialized(StandardLockService.java:216)
    ... 25 more
Caused by: liquibase.exception.DatabaseException: Error executing SQL SELECT COUNT(*) FROM development_sandbox.test.DATABASECHANGELOGLOCK: [Databricks][DatabricksJDBCDriver](500618) Error occured while deserializing arrow data: sun.misc.Unsafe or java.nio.DirectByteBuffer.<init>(long, int) not available
    at liquibase.executor.jvm.JdbcExecutor.execute(JdbcExecutor.java:98)
    at liquibase.executor.jvm.JdbcExecutor.query(JdbcExecutor.java:224)
    at liquibase.executor.jvm.JdbcExecutor.query(JdbcExecutor.java:232)
    at liquibase.executor.jvm.JdbcExecutor.queryForObject(JdbcExecutor.java:240)
    at liquibase.executor.jvm.JdbcExecutor.queryForObject(JdbcExecutor.java:255)
    at liquibase.executor.jvm.JdbcExecutor.queryForInt(JdbcExecutor.java:276)
    at liquibase.executor.jvm.JdbcExecutor.queryForInt(JdbcExecutor.java:271)
    at liquibase.lockservice.StandardLockService.lambda$isDatabaseChangeLogLockTableInitialized$2(StandardLockService.java:216)
    at liquibase.executor.jvm.ChangelogJdbcMdcListener.lambda$query$1(ChangelogJdbcMdcListener.java:54)
    at liquibase.Scope.child(Scope.java:195)
    at liquibase.Scope.child(Scope.java:171)
    at liquibase.executor.jvm.ChangelogJdbcMdcListener.query(ChangelogJdbcMdcListener.java:54)
    ... 26 more
Caused by: java.sql.SQLException: [Databricks][DatabricksJDBCDriver](500618) Error occured while deserializing arrow data: sun.misc.Unsafe or java.nio.DirectByteBuffer.<init>(long, int) not available
    at com.databricks.client.spark.arrow.ArrowBuffer.deserializeBatch(Unknown Source)
    at com.databricks.client.spark.arrow.ArrowBuffer.handleInitializeBuffer(Unknown Source)
    at com.databricks.client.hivecommon.api.HiveServer2BaseBuffer.initializeBuffer(Unknown Source)
    at com.databricks.client.hivecommon.api.RowsetBuffer.initializeBuffer(Unknown Source)
    at com.databricks.client.hivecommon.api.HS2Client.getRowSetInformation(Unknown Source)
    at com.databricks.client.hivecommon.api.HS2Client.fetchFromServer(Unknown Source)
    at com.databricks.client.spark.jdbc.DowloadableFetchClient.fetchNRows(Unknown Source)
    at com.databricks.client.hivecommon.api.HS2Client.fetchRows(Unknown Source)
    at com.databricks.client.hivecommon.dataengine.BackgroundFetcher.run(Unknown Source)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
Caused by: com.databricks.client.support.exceptions.GeneralException: [Databricks][DatabricksJDBCDriver](500618) Error occured while deserializing arrow data: sun.misc.Unsafe or java.nio.DirectByteBuffer.<init>(long, int) not available
    ... 13 more
Caused by: java.lang.UnsupportedOperationException: sun.misc.Unsafe or java.nio.DirectByteBuffer.<init>(long, int) not available
    at com.databricks.client.jdbc42.internal.apache.arrow.memory.util.MemoryUtil.directBuffer(MemoryUtil.java:167)
    at com.databricks.client.jdbc42.internal.apache.arrow.memory.ArrowBuf.getDirectBuffer(ArrowBuf.java:228)
    at com.databricks.client.jdbc42.internal.apache.arrow.memory.ArrowBuf.nioBuffer(ArrowBuf.java:223)
    at com.databricks.client.jdbc42.internal.apache.arrow.vector.ipc.ReadChannel.readFully(ReadChannel.java:87)
    at com.databricks.client.jdbc42.internal.apache.arrow.vector.ipc.message.MessageSerializer.readMessageBody(MessageSerializer.java:727)
    at com.databricks.client.jdbc42.internal.apache.arrow.vector.ipc.message.MessageSerializer.deserializeRecordBatch(MessageSerializer.java:363)
    at com.databricks.client.spark.arrow.ArrowBuffer.deserializeBatch(Unknown Source)
    at com.databricks.client.spark.arrow.ArrowBuffer.handleInitializeBuffer(Unknown Source)
    at com.databricks.client.hivecommon.api.HiveServer2BaseBuffer.initializeBuffer(Unknown Source)
    at com.databricks.client.hivecommon.api.RowsetBuffer.initializeBuffer(Unknown Source)
    at com.databricks.client.hivecommon.api.HS2Client.getRowSetInformation(Unknown Source)
    at com.databricks.client.hivecommon.api.HS2Client.fetchFromServer(Unknown Source)
    at com.databricks.client.spark.jdbc.DowloadableFetchClient.fetchNRows(Unknown Source)
    at com.databricks.client.hivecommon.api.HS2Client.fetchRows(Unknown Source)
    at com.databricks.client.hivecommon.dataengine.BackgroundFetcher.run(Unknown Source)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
    at java.base/java.lang.Thread.run(Thread.java:1583)
BillCM commented 11 months ago

Ah, looks like I need Java 11 from this closed ticket. I'll test that https://github.com/liquibase/liquibase-databricks/issues/64