search

apache / gravitino

World's most powerful open data catalog for building a high-performance, geo-distributed and federated metadata lake.
https://gravitino.apache.org
Apache License 2.0
997 stars 310 forks source link

[Bug report] Iceberg catalog: Failed to alter a decimal column's precision and scale #4114

Closed danhuawang closed 2 months ago

danhuawang commented 3 months ago

Version

main branch

Describe what's wrong

image

Error message and/or stacktrace

{
    "code": 1001,
    "type": "IllegalArgumentException",
    "message": "Failed to operate table(s) [ice_table_3] operation [ALTER] under schema [ice_schema_3], reason [Cannot change column type: c10: decimal(12, 2) -> decimal(20, 10)]",
    "stack": [
        "java.lang.IllegalArgumentException: Cannot change column type: c10: decimal(12, 2) -> decimal(20, 10)",
        "\tat org.apache.iceberg.relocated.com.google.common.base.Preconditions.checkArgument(Preconditions.java:453)",
        "\tat org.apache.iceberg.SchemaUpdate.updateColumn(SchemaUpdate.java:287)",
        "\tat com.datastrato.gravitino.catalog.lakehouse.iceberg.ops.IcebergTableOpsHelper.doUpdateColumnType(IcebergTableOpsHelper.java:171)",
        "\tat com.datastrato.gravitino.catalog.lakehouse.iceberg.ops.IcebergTableOpsHelper.alterTableColumn(IcebergTableOpsHelper.java:263)",
        "\tat com.datastrato.gravitino.catalog.lakehouse.iceberg.ops.IcebergTableOpsHelper.buildIcebergTableChanges(IcebergTableOpsHelper.java:319)",
        "\tat com.datastrato.gravitino.catalog.lakehouse.iceberg.IcebergCatalogOperations.internalUpdateTable(IcebergCatalogOperations.java:407)",
        "\tat com.datastrato.gravitino.catalog.lakehouse.iceberg.IcebergCatalogOperations.alterTable(IcebergCatalogOperations.java:399)",
        "\tat com.datastrato.gravitino.catalog.TableOperationDispatcher.lambda$alterTable$10(TableOperationDispatcher.java:236)",
        "\tat com.datastrato.gravitino.catalog.CatalogManager$CatalogWrapper.lambda$doWithTableOps$1(CatalogManager.java:127)",
        "\tat com.datastrato.gravitino.utils.IsolatedClassLoader.withClassLoader(IsolatedClassLoader.java:86)",
        "\tat com.datastrato.gravitino.catalog.CatalogManager$CatalogWrapper.doWithTableOps(CatalogManager.java:122)",
        "\tat com.datastrato.gravitino.catalog.TableOperationDispatcher.lambda$alterTable$11(TableOperationDispatcher.java:235)",
        "\tat com.datastrato.gravitino.catalog.OperationDispatcher.doWithCatalog(OperationDispatcher.java:135)",
        "\tat com.datastrato.gravitino.catalog.TableOperationDispatcher.alterTable(TableOperationDispatcher.java:232)",
        "\tat com.datastrato.gravitino.catalog.TableNormalizeDispatcher.alterTable(TableNormalizeDispatcher.java:90)",
        "\tat com.datastrato.gravitino.listener.TableEventDispatcher.alterTable(TableEventDispatcher.java:141)",
        "\tat com.datastrato.gravitino.server.web.rest.TableOperations.lambda$alterTable$6(TableOperations.java:201)",
        "\tat com.datastrato.gravitino.lock.TreeLockUtils.doWithTreeLock(TreeLockUtils.java:49)",
        "\tat com.datastrato.gravitino.server.web.rest.TableOperations.lambda$alterTable$7(TableOperations.java:198)",
        "\tat java.security.AccessController.doPrivileged(Native Method)",
        "\tat javax.security.auth.Subject.doAs(Subject.java:422)",
        "\tat com.datastrato.gravitino.utils.PrincipalUtils.doAs(PrincipalUtils.java:39)",
        "\tat com.datastrato.gravitino.server.web.Utils.doAs(Utils.java:135)",
        "\tat com.datastrato.gravitino.server.web.rest.TableOperations.alterTable(TableOperations.java:188)",
        "\tat sun.reflect.GeneratedMethodAccessor63.invoke(Unknown Source)",
        "\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)",
        "\tat java.lang.reflect.Method.invoke(Method.java:498)",
        "\tat org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)",
        "\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146)",
        "\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189)",
        "\tat org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)",
        "\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93)",
        "\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)",
        "\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)",
        "\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)",
        "\tat org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256)",
        "\tat org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)",
        "\tat org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)",
        "\tat org.glassfish.jersey.internal.Errors.process(Errors.java:292)",
        "\tat org.glassfish.jersey.internal.Errors.process(Errors.java:274)",
        "\tat org.glassfish.jersey.internal.Errors.process(Errors.java:244)",
        "\tat org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)",
        "\tat org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235)",
        "\tat org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)",
        "\tat org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)",
        "\tat org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)",
        "\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358)",
        "\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:311)",
        "\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)",
        "\tat org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)",
        "\tat org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656)",
        "\tat com.datastrato.gravitino.server.authentication.AuthenticationFilter.doFilter(AuthenticationFilter.java:73)",
        "\tat org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)",
        "\tat org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)",
        "\tat com.datastrato.gravitino.server.web.VersioningFilter.doFilter(VersioningFilter.java:123)",
        "\tat org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)",
        "\tat org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)",
        "\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552)",
        "\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)",
        "\tat org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)",
        "\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)",
        "\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)",
        "\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)",
        "\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)",
        "\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)",
        "\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)",
        "\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)",
        "\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)",
        "\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)",
        "\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)",
        "\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)",
        "\tat org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)",
        "\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)",
        "\tat org.eclipse.jetty.server.Server.handle(Server.java:516)",
        "\tat org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)",
        "\tat org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)",
        "\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)",
        "\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)",
        "\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)",
        "\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)",
        "\tat org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)",
        "\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)",
        "\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)",
        "\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)",
        "\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)",
        "\tat org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)",
        "\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)",
        "\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)",
        "\tat java.lang.Thread.run(Thread.java:750)"
    ]
}

How to reproduce

  1. Create table under iceberg catalog, hive backend request body: 
    {
"name": "decimal_test_alter_column",
"comment": "decimal_test_alter_column",
"columns": [
{
  "name": "id",
  "type": "integer",
  "comment": "integer column",
  "nullable": true
},
{
  "name": "amount",
  "type": "Decimal(10,2)",
  "comment": "Decimal column",
  "nullable": true
}]
}
  2. Update decimal column's precision and scale request body: 
    {
"updates": [
{
  "@type": "updateColumnType",
  "fieldName": [
    "amount"
  ],
  "newType": "decimal(20,10)"
}
]
}

Additional context

No response

FANNG1 commented 3 months ago

seems it's not a bug, but the different capabilities for different underlying catalogs.

danhuawang commented 3 months ago

Is it a limitation in Iceberg catalog?

FANNG1 commented 2 months ago

close it as it's not a bug for Iceberg catalog.