[BUG] ClassCastException on linux/amd64 build

[shantanu-dahiya]

Is this a possible security vulnerability?

• [X] This is NOT a possible security vulnerability

Describe the bug

Overview

Polaris server docker container built for the amd64 (x64) architecture errors out with code 500 on receiving a catalog creation request.

Error Message

java.lang.ClassCastException: Cannot cast java.util.ImmutableCollections$List12 to java.util.ImmutableCollections$ListN

Create Catalog Request Body

{
    "name": "catalog-name",
    "type": "INTERNAL",
    "readOnly": false,
    "storageConfigInfo": {
        "storageType": "AZURE",
        "tenantId": "<azure-tenant-id>"
    },
    "properties": {
        "default-base-location": "abfss://<path-to-catalog-directory>"
    }
}

Additional Facts

• Running it with or without persistence has no effect.
• The build for arm64 (same code) handles the create catalog request just fine (201 created).
• The last time I built a Polaris container for amd64 that didn't have this problem was on 13th September 2024. I can't find out precisely which commit it was. My guess is that a later commit has caused this bug.

Stack trace

ERROR [2024-10-23 03:32:30,681 - 79990 ] [pool-3-thread-8 - POST /api/management/v1/catalogs] [] o.a.p.s.e.IcebergExceptionMapper: Unhandled exception returning INTERNAL_SERVER_ERROR
java.lang.ClassCastException: Cannot cast java.util.ImmutableCollections$List12 to java.util.ImmutableCollections$ListN
    at java.base/java.lang.Class.cast(Class.java:4067)
    at org.apache.polaris.core.PolarisConfiguration.cast(PolarisConfiguration.java:55)
    at org.apache.polaris.core.PolarisConfigurationStore.tryCast(PolarisConfigurationStore.java:79)
    at org.apache.polaris.core.PolarisConfigurationStore.getConfiguration(PolarisConfigurationStore.java:95)
    at org.apache.polaris.service.admin.PolarisServiceImpl.validateStorageConfig(PolarisServiceImpl.java:121)
    at org.apache.polaris.service.admin.PolarisServiceImpl.createCatalog(PolarisServiceImpl.java:107)
    at org.apache.polaris.service.admin.api.PolarisCatalogsApi.createCatalog(PolarisCatalogsApi.java:139)
    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
    at java.base/java.lang.reflect.Method.invoke(Method.java:580)
    at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146)
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189)
    at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93)
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
    at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
    at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235)
    at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)
    at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:397)
    at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:349)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:379)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:312)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764)
    at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1665)
    at io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:36)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:46)
    at io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:40)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at org.apache.polaris.service.tracing.TracingFilter.doFilter(TracingFilter.java:94)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:314)
    at org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:267)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at org.apache.polaris.service.PolarisApplication$ContextResolverFilter.doFilter(PolarisApplication.java:401)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1381)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1303)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
    at io.dropwizard.metrics.jetty11.InstrumentedHandler.handle(InstrumentedHandler.java:313)
    at io.dropwizard.jetty.RoutingHandler.handle(RoutingHandler.java:52)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
    at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822)
    at io.dropwizard.jetty.ZipExceptionHandlingGzipHandler.handle(ZipExceptionHandlingGzipHandler.java:26)
    at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:46)
    at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:173)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
    at org.eclipse.jetty.server.Server.handle(Server.java:563)
    at org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598)
    at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:287)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
    at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
    at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421)
    at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390)
    at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277)
    at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199)
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
    at java.base/java.lang.Thread.run(Thread.java:1583)
INFO  [2024-10-23 03:32:30,682 - 79991 ] [pool-3-thread-8] [] i.o.e.l.LoggingSpanExporter: 'POST /api/management/v1/catalogs' : 8e20483d4e5a7d374f3964304afcc288 a79f2ecd53db1132 SERVER [tracer: /api/management/v1/catalogs:] AttributesMap{data={http.request.method=POST, server.address=localhost, url.path=/api/management/v1/catalogs, url.scheme=http, realm=default-realm}, capacity=128, totalAddedValues=5}
127.0.0.1 - root [23/Oct/2024:03:32:30 +0000] "POST /api/management/v1/catalogs HTTP/1.1" 500 152 "-" "PostmanRuntime/7.42.0" 5
INFO  [2024-10-23 03:32:31,581 - 80890 ] [pool-3-thread-7] [] o.a.p.s.tracing.TracingFilter: Started span with parent spanId="e8e122f36ce0820b" traceId="043c79bb96ca41a9b98b31b217bbc709" parentContext="{}"
INFO  [2024-10-23 03:32:31,582 - 80891 ] [pool-3-thread-7 - POST /api/management/v1/catalogs] [] o.a.p.s.a.TestInlineBearerTokenPolarisAuthenticator: Checking for existence of principal root in map {principal=root, password=9ef04654ccdcdcec1436b0661ccd0d90, realm=default-realm, role=ALL}
WARN  [2024-10-23 03:32:31,582 - 80891 ] [pool-3-thread-7 - POST /api/management/v1/catalogs] [] o.a.p.s.a.TestInlineBearerTokenPolarisAuthenticator: Failed to load secrets for principal root
INFO  [2024-10-23 03:32:31,584 - 80893 ] [pool-3-thread-7 - POST /api/management/v1/catalogs] [] o.a.p.s.e.IcebergExceptionMapper: Handling runtimeException Cannot cast java.util.ImmutableCollections$List12 to java.util.ImmutableCollections$ListN
ERROR [2024-10-23 03:32:31,584 - 80893 ] [pool-3-thread-7 - POST /api/management/v1/catalogs] [] o.a.p.s.e.IcebergExceptionMapper: Unhandled exception returning INTERNAL_SERVER_ERROR
java.lang.ClassCastException: Cannot cast java.util.ImmutableCollections$List12 to java.util.ImmutableCollections$ListN
    at java.base/java.lang.Class.cast(Class.java:4067)
    at org.apache.polaris.core.PolarisConfiguration.cast(PolarisConfiguration.java:55)
    at org.apache.polaris.core.PolarisConfigurationStore.tryCast(PolarisConfigurationStore.java:79)
    at org.apache.polaris.core.PolarisConfigurationStore.getConfiguration(PolarisConfigurationStore.java:95)
    at org.apache.polaris.service.admin.PolarisServiceImpl.validateStorageConfig(PolarisServiceImpl.java:121)
    at org.apache.polaris.service.admin.PolarisServiceImpl.createCatalog(PolarisServiceImpl.java:107)
    at org.apache.polaris.service.admin.api.PolarisCatalogsApi.createCatalog(PolarisCatalogsApi.java:139)
    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
    at java.base/java.lang.reflect.Method.invoke(Method.java:580)
    at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146)
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189)
    at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93)
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
    at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
    at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235)
    at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)
    at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:397)
    at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:349)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:379)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:312)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764)
    at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1665)
    at io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:36)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:46)
    at io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:40)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at org.apache.polaris.service.tracing.TracingFilter.doFilter(TracingFilter.java:94)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:314)
    at org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:267)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at org.apache.polaris.service.PolarisApplication$ContextResolverFilter.doFilter(PolarisApplication.java:401)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1381)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1303)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
    at io.dropwizard.metrics.jetty11.InstrumentedHandler.handle(InstrumentedHandler.java:313)
    at io.dropwizard.jetty.RoutingHandler.handle(RoutingHandler.java:52)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
    at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822)
    at io.dropwizard.jetty.ZipExceptionHandlingGzipHandler.handle(ZipExceptionHandlingGzipHandler.java:26)
    at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:46)
    at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:173)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
    at org.eclipse.jetty.server.Server.handle(Server.java:563)
    at org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598)
    at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:287)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
    at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
    at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421)
    at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390)
    at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277)
    at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199)
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
    at java.base/java.lang.Thread.run(Thread.java:1583)

To Reproduce

• Fetch latest main branch of apache/polaris
• Build the docker container for amd64 arch

  cd polaris
  docker build --platform linux/amd64 .

• Run Polaris server

  docker run -p 8181:8181 <image-id>

• Send a create catalog request with cURL

  curl --location 'http://localhost:8181/api/management/v1/catalogs' \
  --header 'Authorization: Bearer <token>' \
  --header 'Accept: application/json' \
  --header 'Content-Type: application/json' \
  --data-raw '{
    "name": "catalog-name",
    "type": "INTERNAL",
    "readOnly": false,
    "storageConfigInfo": {
        "storageType": "AZURE",
        "tenantId": "<azure-tenant-id>"
    },
    "properties": {
        "default-base-location": "abfss://<path-to-catalog-directory>"
    }
  }'

Actual Behavior

Polaris server returns error code 500 with the stack trace printed above.

Expected Behavior

Polaris server should create the catalog normally and return code 201 (created).

Additional context

No response

System information

Docker version: 27.1.1 JDK version: OpenJDK 21.0.4, Temurin 21.0.5, Microsoft OpenJDK 21 Machine: MacBook Pro M2 (arm64), Github runner ubuntu-latest that runs on x64.

[jbonofre]

It looks like a amd64 issue but weird as it's related to classloader. I gonna take a look. Did you try to run locally (not in docker) ?