search

apache / polaris

Apache Polaris, the interoperable, open source catalog for Apache Iceberg
https://polaris.apache.org/
Apache License 2.0
1.1k stars 119 forks source link

[BUG] Connection pool seems to be gone after sometime and Polaris doesn't create a new connection pool after that (possible deadlock) #107

Closed MonkeyCanCode closed 2 months ago

MonkeyCanCode commented 2 months ago

Is this a possible security vulnerability?

Describe the bug

We noticed after have Polaris running for sometime with Eclipse Link, the connection pool created by Polaris got tear down and Polaris is not trying to create a new connection pool after that.

To Reproduce

Have Polaris with Eclipse link and Postgres backend running for sometime (sent couple requests in to bring up the connection pool then have it sit idle for sometime), once the connection on the backend is all gone (somehow keepalive is not happening or it is working but potential deadlock blocked there), we will then be able to see the following:

Sample curl request:

curl --location 'http://localhost:8181/api/management/v1/principals' \
--header 'Authorization: Bearer principal:root;password:2c81396d261e1970c1527475e7ae7807;realm:default-realm;role:ALL'

Logs on the server side after submitted this request (stuck here until curl timed out):

DEBUG [2024-08-07 14:43:57,372 - 6224858] [pool-3-thread-9] [] i.p.s.c.DefaultContextResolver: Resolving RealmContext for method: GET, path: api/management/v1/principals, queryParams: {}, headers: {Authorization=Bearer principal:root;password:2c81396d261e1970c1527475e7ae7807;realm:default-realm;role:ALL, Accept=*/*, User-Agent=curl/8.7.1, Host=localhost:8181}
DEBUG [2024-08-07 14:43:57,372 - 6224858] [pool-3-thread-9] [] i.p.s.c.DefaultContextResolver: Resolving CallContext realmContext="default-realm" method="GET" path="api/management/v1/principals" queryParams="{}" headers="{Authorization=Bearer principal:root;password:2c81396d261e1970c1527475e7ae7807;realm:default-realm;role:ALL, Accept=*/*, User-Agent=curl/8.7.1, Host=localhost:8181}"
DEBUG [2024-08-07 14:43:57,372 - 6224858] [pool-3-thread-9] [] i.p.s.c.RealmEntityManagerFactory: Looking up PolarisEntityManager for realm default-realm
DEBUG [2024-08-07 14:43:57,373 - 6224859] [pool-3-thread-9] [] i.p.e.p.i.e.PolarisEclipseLinkMetaStoreSessionImpl: Create EclipseLink Meta Store Session for default-realm
INFO  [2024-08-07 14:43:57,373 - 6224859] [pool-3-thread-9] [] i.p.s.tracing.TracingFilter: Started span with parent spanId="d93e2ab7fa0e30bc" traceId="d56f42a0cd2df9acbff22f21cd7dc1be" parentContext="{}"
DEBUG [2024-08-07 14:43:57,373 - 6224859] [pool-3-thread-9 - GET /api/management/v1/principals] [] i.p.s.c.RealmEntityManagerFactory: Looking up PolarisEntityManager for realm default-realm
INFO  [2024-08-07 14:43:57,373 - 6224859] [pool-3-thread-9 - GET /api/management/v1/principals] [] i.p.s.a.TestInlineBearerTokenPolarisAuthenticator: Checking for existence of principal root in map {principal=root, password=2c81396d261e1970c1527475e7ae7807, realm=default-realm, role=ALL}

Check the backend database to see if there are any established connections and it shows none:

SELECT pid, age(clock_timestamp(), query_start), usename, query 
FROM pg_stat_activity 
WHERE query != '<IDLE>' AND query NOT ILIKE '%pg_stat_activity%' and application_name = 'PostgreSQL JDBC Driver'
ORDER BY query_start desc;

This leads me to think somehow the connection pool created by Polaris got teared down and Polaris is unable to recreate the connection pool. However, from the following steps, I verified connection pool recreation is for sure there under happy path:

  1. Redeploy polaris to clear the deadlock state
  2. Submitted a request to bring up the connection pool
  3. Restarted backend metastore
  4. Verified connection pool is gone on backend metastore
  5. Submitted a request to bring up the connection pool and got connection pool being gone. Then at the same time, Polaris will then try to bring up a new connection pool based on log messages
  6. Submitted another request and this time it goes through

With above steps, for sure the connection pooling recreation is working. However, when the poential deadlock came, the connection pooling recreation became no longer possible.

Actual Behavior

No response

Expected Behavior

No response

Additional context

Here are a couple of log lines that came before the shared logs above (generated by background thread I am assuming also indicated it is trying to keep connection alive but failed to do so:

[EL Fine]: sql: 2024-08-07 14:42:29.219--ServerSession(1501064238)--SELECT 1
[EL Info]: query: 2024-08-07 14:42:29.219--UnitOfWork(885977653)--Communication failure detected when attempting to perform read query outside of a transaction. Attempting to retry query. Error was: Exception [EclipseLink-4002] (Eclipse Persistence Services - 4.0.3.v202405220658): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: org.postgresql.util.PSQLException: This connection has been closed.
Error Code: 0
Call: SELECT PRINCIPALCLIENTID, MAINSECRET, PRINCIPALID, SECONDARYSECRET, VERSION FROM PRINCIPAL_SECRETS WHERE (PRINCIPALCLIENTID = ?)
    bind => [root]
Query: ReadObjectQuery(referenceClass=ModelPrincipalSecrets sql="SELECT PRINCIPALCLIENTID, MAINSECRET, PRINCIPALID, SECONDARYSECRET, VERSION FROM PRINCIPAL_SECRETS WHERE (PRINCIPALCLIENTID = ?)").
[EL Fine]: sql: 2024-08-07 14:42:29.219--ServerSession(1501064238)--SELECT 1
[EL Fine]: sql: 2024-08-07 14:42:29.219--ServerSession(1501064238)--SELECT 1
[EL Fine]: sql: 2024-08-07 14:42:29.219--ServerSession(1501064238)--SELECT 1

Here is a java trace that happened earlier as well (not caused by the curl command):

Call: SELECT PRINCIPALCLIENTID, MAINSECRET, PRINCIPALID, SECONDARYSECRET, VERSION FROM PRINCIPAL_SECRETS WHERE (PRINCIPALCLIENTID = ?)
    bind => [root]
Query: ReadObjectQuery(referenceClass=ModelPrincipalSecrets sql="SELECT PRINCIPALCLIENTID, MAINSECRET, PRINCIPALID, SECONDARYSECRET, VERSION FROM PRINCIPAL_SECRETS WHERE (PRINCIPALCLIENTID = ?)")
DEBUG [2024-08-07 14:20:22,032 - 4809518] [pool-3-thread-10 - GET /api/catalog/v1/config?warehouse=polaris] [] i.p.e.p.i.e.PolarisEclipseLinkMetaStoreSessionImpl: transaction rolled back: {}
jakarta.persistence.PersistenceException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 4.0.3.v202405220658): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend.
Error Code: 0
Call: SELECT PRINCIPALCLIENTID, MAINSECRET, PRINCIPALID, SECONDARYSECRET, VERSION FROM PRINCIPAL_SECRETS WHERE (PRINCIPALCLIENTID = ?)
    bind => [root]
Query: ReadObjectQuery(referenceClass=ModelPrincipalSecrets sql="SELECT PRINCIPALCLIENTID, MAINSECRET, PRINCIPALID, SECONDARYSECRET, VERSION FROM PRINCIPAL_SECRETS WHERE (PRINCIPALCLIENTID = ?)")
    at org.eclipse.persistence.internal.jpa.QueryImpl.getDetailedException(QueryImpl.java:392)
    at org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:265)
    at org.eclipse.persistence.internal.jpa.QueryImpl.getResultList(QueryImpl.java:475)
    at jakarta.persistence.TypedQuery.getResultStream(TypedQuery.java:87)
    at io.polaris.extension.persistence.impl.eclipselink.PolarisEclipseLinkStore.lookupPrincipalSecrets(PolarisEclipseLinkStore.java:385)
    at io.polaris.extension.persistence.impl.eclipselink.PolarisEclipseLinkMetaStoreSessionImpl.loadPrincipalSecrets(PolarisEclipseLinkMetaStoreSessionImpl.java:558)
    at io.polaris.core.persistence.PolarisMetaStoreManagerImpl.loadPrincipalSecrets(PolarisMetaStoreManagerImpl.java:972)
    at io.polaris.core.persistence.PolarisMetaStoreManagerImpl.lambda$loadPrincipalSecrets$7(PolarisMetaStoreManagerImpl.java:984)
    at io.polaris.extension.persistence.impl.eclipselink.PolarisEclipseLinkMetaStoreSessionImpl.runInTransaction(PolarisEclipseLinkMetaStoreSessionImpl.java:168)
    at io.polaris.core.persistence.PolarisMetaStoreManagerImpl.loadPrincipalSecrets(PolarisMetaStoreManagerImpl.java:984)
    at io.polaris.service.auth.TestInlineBearerTokenPolarisAuthenticator.authenticate(TestInlineBearerTokenPolarisAuthenticator.java:72)
    at io.polaris.service.auth.TestInlineBearerTokenPolarisAuthenticator.authenticate(TestInlineBearerTokenPolarisAuthenticator.java:45)
    at io.dropwizard.auth.AuthFilter.authenticate(AuthFilter.java:144)
    at io.dropwizard.auth.oauth.OAuthCredentialAuthFilter.filter(OAuthCredentialAuthFilter.java:37)
    at org.glassfish.jersey.server.ContainerFilteringStage.apply(ContainerFilteringStage.java:108)
    at org.glassfish.jersey.server.ContainerFilteringStage.apply(ContainerFilteringStage.java:44)
    at org.glassfish.jersey.process.internal.Stages.process(Stages.java:173)
    at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:248)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
    at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235)
    at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)
    at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
    at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:311)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764)
    at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1665)
    at io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:36)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:46)
    at io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:40)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at io.polaris.service.tracing.TracingFilter.doFilter(TracingFilter.java:91)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:314)
    at org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:267)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at io.polaris.service.PolarisApplication$ContextResolverFilter.doFilter(PolarisApplication.java:366)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1381)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1303)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
    at io.dropwizard.metrics.jetty11.InstrumentedHandler.handle(InstrumentedHandler.java:313)
    at io.dropwizard.jetty.RoutingHandler.handle(RoutingHandler.java:52)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
    at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822)
    at io.dropwizard.jetty.ZipExceptionHandlingGzipHandler.handle(ZipExceptionHandlingGzipHandler.java:26)
    at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:46)
    at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:173)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
    at org.eclipse.jetty.server.Server.handle(Server.java:563)
    at org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598)
    at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:287)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
    at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
    at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: org.eclipse.persistence.exceptions.DatabaseException:
Internal Exception: org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend.
Error Code: 0

No response

System information

No response

MonkeyCanCode commented 2 months ago

Here is the thread dump:

2024-08-07 14:49:43
Full thread dump OpenJDK 64-Bit Server VM (21+35-2513 mixed mode, sharing):

Threads class SMR info:
_java_thread_list=0x00007fe0840038b0, length=37, elements={
0x00007fe1780f9540, 0x00007fe1780fab90, 0x00007fe1780fc670, 0x00007fe1780fdcb0,
0x00007fe1780ff250, 0x00007fe178100d90, 0x00007fe178102450, 0x00007fe178110520,
0x00007fe178113cb0, 0x00007fe1789ddee0, 0x00007fe1789d6c10, 0x00007fe178c554e0,
0x00007fe178c561f0, 0x00007fe178c5d840, 0x00007fe178c5e9c0, 0x00007fe178c5fb10,
0x00007fe178c60c70, 0x00007fe178c61ea0, 0x00007fe178c630e0, 0x00007fe178c65570,
0x00007fe178c667c0, 0x00007fe178c67950, 0x00007fe178fa4c30, 0x00007fe178fa69a0,
0x00007fe178027e90, 0x00007fe0a40584f0, 0x00007fe09c7920a0, 0x00007fe088000e70,
0x00007fe09c6f3370, 0x00007fe0980cdbe0, 0x00007fe09c5442d0, 0x00007fe084001690,
0x00007fe06c001dd0, 0x00007fe084003a40, 0x00007fe0e800c650, 0x00007fe06c01d3a0,
0x00007fe06c023a90
}

"Reference Handler" #9 [15] daemon prio=10 os_prio=0 cpu=11.68ms elapsed=6571.53s tid=0x00007fe1780f9540 nid=15 waiting on condition  [0x00007fe15195d000]
   java.lang.Thread.State: RUNNABLE
    at java.lang.ref.Reference.waitForReferencePendingList(java.base@21/Native Method)
    at java.lang.ref.Reference.processPendingReferences(java.base@21/Reference.java:246)
    at java.lang.ref.Reference$ReferenceHandler.run(java.base@21/Reference.java:208)

"Finalizer" #10 [16] daemon prio=8 os_prio=0 cpu=59.30ms elapsed=6571.53s tid=0x00007fe1780fab90 nid=16 in Object.wait()  [0x00007fe15185c000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait0(java.base@21/Native Method)
    - waiting on <no object reference available>
    at java.lang.Object.wait(java.base@21/Object.java:366)
    at java.lang.Object.wait(java.base@21/Object.java:339)
    at java.lang.ref.NativeReferenceQueue.await(java.base@21/NativeReferenceQueue.java:48)
    at java.lang.ref.ReferenceQueue.remove0(java.base@21/ReferenceQueue.java:158)
    at java.lang.ref.NativeReferenceQueue.remove(java.base@21/NativeReferenceQueue.java:89)
    - locked <0x00000000c00000f0> (a java.lang.ref.NativeReferenceQueue$Lock)
    at java.lang.ref.Finalizer$FinalizerThread.run(java.base@21/Finalizer.java:173)

"Signal Dispatcher" #11 [17] daemon prio=9 os_prio=0 cpu=0.27ms elapsed=6571.53s tid=0x00007fe1780fc670 nid=17 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Service Thread" #12 [18] daemon prio=9 os_prio=0 cpu=23.83ms elapsed=6571.53s tid=0x00007fe1780fdcb0 nid=18 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Monitor Deflation Thread" #13 [19] daemon prio=9 os_prio=0 cpu=430.59ms elapsed=6571.53s tid=0x00007fe1780ff250 nid=19 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #14 [20] daemon prio=9 os_prio=0 cpu=25631.96ms elapsed=6571.53s tid=0x00007fe178100d90 nid=20 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   No compile task

"C1 CompilerThread0" #16 [21] daemon prio=9 os_prio=0 cpu=5045.00ms elapsed=6571.53s tid=0x00007fe178102450 nid=21 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   No compile task

"Notification Thread" #17 [22] daemon prio=9 os_prio=0 cpu=0.04ms elapsed=6571.52s tid=0x00007fe178110520 nid=22 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Common-Cleaner" #18 [23] daemon prio=8 os_prio=0 cpu=10.85ms elapsed=6571.52s tid=0x00007fe178113cb0 nid=23 waiting on condition  [0x00007fe151085000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@21/Native Method)
    - parking to wait for  <0x00000000c0000218> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(java.base@21/LockSupport.java:269)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21/AbstractQueuedSynchronizer.java:1847)
    at java.lang.ref.ReferenceQueue.await(java.base@21/ReferenceQueue.java:71)
    at java.lang.ref.ReferenceQueue.remove0(java.base@21/ReferenceQueue.java:143)
    at java.lang.ref.ReferenceQueue.remove(java.base@21/ReferenceQueue.java:218)
    at jdk.internal.ref.CleanerImpl.run(java.base@21/CleanerImpl.java:140)
    at java.lang.Thread.runWith(java.base@21/Thread.java:1596)
    at java.lang.Thread.run(java.base@21/Thread.java:1583)
    at jdk.internal.misc.InnocuousThread.run(java.base@21/InnocuousThread.java:186)

"AsyncAppender-Worker-async-console-appender" #24 [31] daemon prio=5 os_prio=0 cpu=1251.94ms elapsed=6570.49s tid=0x00007fe1789ddee0 nid=31 waiting on condition  [0x00007fe150b80000]
   java.lang.Thread.State: WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@21/Native Method)
    - parking to wait for  <0x00000000c053ba38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(java.base@21/LockSupport.java:371)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@21/AbstractQueuedSynchronizer.java:519)
    at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21/ForkJoinPool.java:3780)
    at java.util.concurrent.ForkJoinPool.managedBlock(java.base@21/ForkJoinPool.java:3725)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21/AbstractQueuedSynchronizer.java:1707)
    at java.util.concurrent.ArrayBlockingQueue.take(java.base@21/ArrayBlockingQueue.java:420)
    at ch.qos.logback.core.AsyncAppenderBase$Worker.run(AsyncAppenderBase.java:298)

"AsyncAppender-Worker-async-file-appender" #25 [32] daemon prio=5 os_prio=0 cpu=2379.54ms elapsed=6570.47s tid=0x00007fe1789d6c10 nid=32 waiting on condition  [0x00007fe150a7f000]
   java.lang.Thread.State: WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@21/Native Method)
    - parking to wait for  <0x00000000c0969ef0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(java.base@21/LockSupport.java:371)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@21/AbstractQueuedSynchronizer.java:519)
    at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21/ForkJoinPool.java:3780)
    at java.util.concurrent.ForkJoinPool.managedBlock(java.base@21/ForkJoinPool.java:3725)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21/AbstractQueuedSynchronizer.java:1707)
    at java.util.concurrent.ArrayBlockingQueue.take(java.base@21/ArrayBlockingQueue.java:420)
    at ch.qos.logback.core.AsyncAppenderBase$Worker.run(AsyncAppenderBase.java:298)

"AsyncAppender-Worker-async-console-appender" #26 [33] daemon prio=5 os_prio=0 cpu=91.66ms elapsed=6570.23s tid=0x00007fe178c554e0 nid=33 waiting on condition  [0x00007fe15097e000]
   java.lang.Thread.State: WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@21/Native Method)
    - parking to wait for  <0x00000000c053e288> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(java.base@21/LockSupport.java:371)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@21/AbstractQueuedSynchronizer.java:519)
    at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21/ForkJoinPool.java:3780)
    at java.util.concurrent.ForkJoinPool.managedBlock(java.base@21/ForkJoinPool.java:3725)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21/AbstractQueuedSynchronizer.java:1707)
    at java.util.concurrent.ArrayBlockingQueue.take(java.base@21/ArrayBlockingQueue.java:420)
    at ch.qos.logback.core.AsyncAppenderBase$Worker.run(AsyncAppenderBase.java:298)

"AsyncAppender-Worker-async-file-appender" #27 [34] daemon prio=5 os_prio=0 cpu=82.06ms elapsed=6570.23s tid=0x00007fe178c561f0 nid=34 waiting on condition  [0x00007fe15087d000]
   java.lang.Thread.State: WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@21/Native Method)
    - parking to wait for  <0x00000000c053bbd8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(java.base@21/LockSupport.java:371)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@21/AbstractQueuedSynchronizer.java:519)
    at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21/ForkJoinPool.java:3780)
    at java.util.concurrent.ForkJoinPool.managedBlock(java.base@21/ForkJoinPool.java:3725)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21/AbstractQueuedSynchronizer.java:1707)
    at java.util.concurrent.ArrayBlockingQueue.take(java.base@21/ArrayBlockingQueue.java:420)
    at ch.qos.logback.core.AsyncAppenderBase$Worker.run(AsyncAppenderBase.java:298)

"pool-3-thread-1 - GET /api/management/v1/principals" #28 [35] prio=5 os_prio=0 cpu=58.78ms elapsed=6570.22s tid=0x00007fe178c5d840 nid=35 waiting for monitor entry  [0x00007fe150779000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at org.eclipse.persistence.sessions.server.ConnectionPool.acquireConnection(ConnectionPool.java:130)
    - waiting to lock <0x00000000c0ac2788> (a org.eclipse.persistence.sessions.server.ConnectionPool)
    at org.eclipse.persistence.sessions.server.ServerSession.getAccessors(ServerSession.java:580)
    at org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:598)
    at org.eclipse.persistence.sessions.server.ClientSession.executeCall(ClientSession.java:263)
    at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:280)
    at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:266)
    at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.selectOneRow(DatasourceCallQueryMechanism.java:813)
    at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectOneRowFromTable(ExpressionQueryMechanism.java:2912)
    at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectOneRow(ExpressionQueryMechanism.java:2865)
    at org.eclipse.persistence.queries.ReadObjectQuery.executeObjectLevelReadQuery(ReadObjectQuery.java:563)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1236)
    at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:913)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1195)
    at org.eclipse.persistence.queries.ReadObjectQuery.execute(ReadObjectQuery.java:448)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1283)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:3025)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1841)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1823)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1788)
    at org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:263)
    at org.eclipse.persistence.internal.jpa.QueryImpl.getResultList(QueryImpl.java:475)
    at jakarta.persistence.TypedQuery.getResultStream(TypedQuery.java:87)
    at io.polaris.extension.persistence.impl.eclipselink.PolarisEclipseLinkStore.lookupPrincipalSecrets(PolarisEclipseLinkStore.java:385)
    at io.polaris.extension.persistence.impl.eclipselink.PolarisEclipseLinkMetaStoreSessionImpl.loadPrincipalSecrets(PolarisEclipseLinkMetaStoreSessionImpl.java:558)
    at io.polaris.core.persistence.PolarisMetaStoreManagerImpl.loadPrincipalSecrets(PolarisMetaStoreManagerImpl.java:972)
    at io.polaris.core.persistence.PolarisMetaStoreManagerImpl.lambda$loadPrincipalSecrets$7(PolarisMetaStoreManagerImpl.java:984)
    at io.polaris.core.persistence.PolarisMetaStoreManagerImpl$$Lambda/0x00007fe10474bd68.get(Unknown Source)
    at io.polaris.extension.persistence.impl.eclipselink.PolarisEclipseLinkMetaStoreSessionImpl.runInTransaction(PolarisEclipseLinkMetaStoreSessionImpl.java:168)
    at io.polaris.core.persistence.PolarisMetaStoreManagerImpl.loadPrincipalSecrets(PolarisMetaStoreManagerImpl.java:984)
    at io.polaris.service.auth.TestInlineBearerTokenPolarisAuthenticator.authenticate(TestInlineBearerTokenPolarisAuthenticator.java:72)
    at io.polaris.service.auth.TestInlineBearerTokenPolarisAuthenticator.authenticate(TestInlineBearerTokenPolarisAuthenticator.java:45)
    at io.dropwizard.auth.AuthFilter.authenticate(AuthFilter.java:144)
    at io.dropwizard.auth.oauth.OAuthCredentialAuthFilter.filter(OAuthCredentialAuthFilter.java:37)
    at org.glassfish.jersey.server.ContainerFilteringStage.apply(ContainerFilteringStage.java:108)
    at org.glassfish.jersey.server.ContainerFilteringStage.apply(ContainerFilteringStage.java:44)
    at org.glassfish.jersey.process.internal.Stages.process(Stages.java:173)
    at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:248)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
    at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235)
    at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)
    at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
    at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:311)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764)
    at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1665)
    at io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:36)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:46)
    at io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:40)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at io.polaris.service.tracing.TracingFilter.doFilter(TracingFilter.java:91)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:314)
    at org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:267)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at io.polaris.service.PolarisApplication$ContextResolverFilter.doFilter(PolarisApplication.java:366)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1381)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1303)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
    at io.dropwizard.metrics.jetty11.InstrumentedHandler.handle(InstrumentedHandler.java:313)
    at io.dropwizard.jetty.RoutingHandler.handle(RoutingHandler.java:52)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
    at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822)
    at io.dropwizard.jetty.ZipExceptionHandlingGzipHandler.handle(ZipExceptionHandlingGzipHandler.java:26)
    at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:46)
    at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:173)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
    at org.eclipse.jetty.server.Server.handle(Server.java:563)
    at org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598)
    at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:287)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
    at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
    at java.lang.Thread.runWith(java.base@21/Thread.java:1596)
    at java.lang.Thread.run(java.base@21/Thread.java:1583)

"pool-3-thread-2 - GET /api/management/v1/principals" #29 [36] prio=5 os_prio=0 cpu=67.84ms elapsed=6570.22s tid=0x00007fe178c5e9c0 nid=36 waiting for monitor entry  [0x00007fe150678000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at org.eclipse.persistence.sessions.server.ConnectionPool.acquireConnection(ConnectionPool.java:130)
    - waiting to lock <0x00000000c0ac2788> (a org.eclipse.persistence.sessions.server.ConnectionPool)
    at org.eclipse.persistence.sessions.server.ServerSession.getAccessors(ServerSession.java:580)
    at org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:598)
    at org.eclipse.persistence.sessions.server.ClientSession.executeCall(ClientSession.java:263)
    at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:280)
    at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:266)
    at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.selectOneRow(DatasourceCallQueryMechanism.java:813)
    at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectOneRowFromTable(ExpressionQueryMechanism.java:2912)
    at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectOneRow(ExpressionQueryMechanism.java:2865)
    at org.eclipse.persistence.queries.ReadObjectQuery.executeObjectLevelReadQuery(ReadObjectQuery.java:563)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1236)
    at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:913)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1195)
    at org.eclipse.persistence.queries.ReadObjectQuery.execute(ReadObjectQuery.java:448)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1283)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:3025)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1841)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1823)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1788)
    at org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:263)
    at org.eclipse.persistence.internal.jpa.QueryImpl.getResultList(QueryImpl.java:475)
    at jakarta.persistence.TypedQuery.getResultStream(TypedQuery.java:87)
    at io.polaris.extension.persistence.impl.eclipselink.PolarisEclipseLinkStore.lookupPrincipalSecrets(PolarisEclipseLinkStore.java:385)
    at io.polaris.extension.persistence.impl.eclipselink.PolarisEclipseLinkMetaStoreSessionImpl.loadPrincipalSecrets(PolarisEclipseLinkMetaStoreSessionImpl.java:558)
    at io.polaris.core.persistence.PolarisMetaStoreManagerImpl.loadPrincipalSecrets(PolarisMetaStoreManagerImpl.java:972)
    at io.polaris.core.persistence.PolarisMetaStoreManagerImpl.lambda$loadPrincipalSecrets$7(PolarisMetaStoreManagerImpl.java:984)
    at io.polaris.core.persistence.PolarisMetaStoreManagerImpl$$Lambda/0x00007fe10474bd68.get(Unknown Source)
    at io.polaris.extension.persistence.impl.eclipselink.PolarisEclipseLinkMetaStoreSessionImpl.runInTransaction(PolarisEclipseLinkMetaStoreSessionImpl.java:168)
    at io.polaris.core.persistence.PolarisMetaStoreManagerImpl.loadPrincipalSecrets(PolarisMetaStoreManagerImpl.java:984)
    at io.polaris.service.auth.TestInlineBearerTokenPolarisAuthenticator.authenticate(TestInlineBearerTokenPolarisAuthenticator.java:72)
    at io.polaris.service.auth.TestInlineBearerTokenPolarisAuthenticator.authenticate(TestInlineBearerTokenPolarisAuthenticator.java:45)
    at io.dropwizard.auth.AuthFilter.authenticate(AuthFilter.java:144)
    at io.dropwizard.auth.oauth.OAuthCredentialAuthFilter.filter(OAuthCredentialAuthFilter.java:37)
    at org.glassfish.jersey.server.ContainerFilteringStage.apply(ContainerFilteringStage.java:108)
    at org.glassfish.jersey.server.ContainerFilteringStage.apply(ContainerFilteringStage.java:44)
    at org.glassfish.jersey.process.internal.Stages.process(Stages.java:173)
    at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:248)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
    at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235)
    at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)
    at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
    at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:311)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764)
    at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1665)
    at io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:36)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:46)
    at io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:40)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at io.polaris.service.tracing.TracingFilter.doFilter(TracingFilter.java:91)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:314)
    at org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:267)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at io.polaris.service.PolarisApplication$ContextResolverFilter.doFilter(PolarisApplication.java:366)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1381)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1303)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
    at io.dropwizard.metrics.jetty11.InstrumentedHandler.handle(InstrumentedHandler.java:313)
    at io.dropwizard.jetty.RoutingHandler.handle(RoutingHandler.java:52)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
    at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822)
    at io.dropwizard.jetty.ZipExceptionHandlingGzipHandler.handle(ZipExceptionHandlingGzipHandler.java:26)
    at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:46)
    at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:173)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
    at org.eclipse.jetty.server.Server.handle(Server.java:563)
    at org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598)
    at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:287)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
    at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
    at java.lang.Thread.runWith(java.base@21/Thread.java:1596)
    at java.lang.Thread.run(java.base@21/Thread.java:1583)

"pool-3-thread-3-acceptor-0@70d8c228-application@1e58512c{HTTP/1.1, (http/1.1)}{0.0.0.0:8181}" #30 [37] prio=3 os_prio=0 cpu=3.82ms elapsed=6570.22s tid=0x00007fe178c5fb10 nid=37 runnable  [0x00007fe15057a000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.Net.accept(java.base@21/Native Method)
    at sun.nio.ch.ServerSocketChannelImpl.implAccept(java.base@21/ServerSocketChannelImpl.java:433)
    at sun.nio.ch.ServerSocketChannelImpl.accept(java.base@21/ServerSocketChannelImpl.java:399)
    at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:409)
    at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:748)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
    at java.lang.Thread.runWith(java.base@21/Thread.java:1596)
    at java.lang.Thread.run(java.base@21/Thread.java:1583)

"pool-3-thread-4" #31 [38] prio=5 os_prio=0 cpu=12339.31ms elapsed=6570.22s tid=0x00007fe178c60c70 nid=38 runnable  [0x00007fe150479000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPoll.wait(java.base@21/Native Method)
    at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@21/EPollSelectorImpl.java:121)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@21/SelectorImpl.java:130)
    - locked <0x00000000c323e120> (a sun.nio.ch.Util$2)
    - locked <0x00000000c323e0d0> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(java.base@21/SelectorImpl.java:147)
    at org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:181)
    at org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:188)
    at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:606)
    at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:542)
    at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:455)
    at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:248)
    at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199)
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
    at java.lang.Thread.runWith(java.base@21/Thread.java:1596)
    at java.lang.Thread.run(java.base@21/Thread.java:1583)

"pool-3-thread-5 - GET /api/management/v1/principals" #32 [39] prio=5 os_prio=0 cpu=1959.69ms elapsed=6570.22s tid=0x00007fe178c61ea0 nid=39 waiting for monitor entry  [0x00007fe150374000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at org.eclipse.persistence.sessions.server.ConnectionPool.acquireConnection(ConnectionPool.java:130)
    - waiting to lock <0x00000000c0ac2788> (a org.eclipse.persistence.sessions.server.ConnectionPool)
    at org.eclipse.persistence.sessions.server.ServerSession.getAccessors(ServerSession.java:580)
    at org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:598)
    at org.eclipse.persistence.sessions.server.ClientSession.executeCall(ClientSession.java:263)
    at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:280)
    at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:266)
    at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.selectOneRow(DatasourceCallQueryMechanism.java:813)
    at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectOneRowFromTable(ExpressionQueryMechanism.java:2912)
    at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectOneRow(ExpressionQueryMechanism.java:2865)
    at org.eclipse.persistence.queries.ReadObjectQuery.executeObjectLevelReadQuery(ReadObjectQuery.java:563)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1236)
    at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:913)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1195)
    at org.eclipse.persistence.queries.ReadObjectQuery.execute(ReadObjectQuery.java:448)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1283)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:3025)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1841)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1823)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1788)
    at org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:263)
    at org.eclipse.persistence.internal.jpa.QueryImpl.getResultList(QueryImpl.java:475)
    at jakarta.persistence.TypedQuery.getResultStream(TypedQuery.java:87)
    at io.polaris.extension.persistence.impl.eclipselink.PolarisEclipseLinkStore.lookupPrincipalSecrets(PolarisEclipseLinkStore.java:385)
    at io.polaris.extension.persistence.impl.eclipselink.PolarisEclipseLinkMetaStoreSessionImpl.loadPrincipalSecrets(PolarisEclipseLinkMetaStoreSessionImpl.java:558)
    at io.polaris.core.persistence.PolarisMetaStoreManagerImpl.loadPrincipalSecrets(PolarisMetaStoreManagerImpl.java:972)
    at io.polaris.core.persistence.PolarisMetaStoreManagerImpl.lambda$loadPrincipalSecrets$7(PolarisMetaStoreManagerImpl.java:984)
    at io.polaris.core.persistence.PolarisMetaStoreManagerImpl$$Lambda/0x00007fe10474bd68.get(Unknown Source)
    at io.polaris.extension.persistence.impl.eclipselink.PolarisEclipseLinkMetaStoreSessionImpl.runInTransaction(PolarisEclipseLinkMetaStoreSessionImpl.java:168)
    at io.polaris.core.persistence.PolarisMetaStoreManagerImpl.loadPrincipalSecrets(PolarisMetaStoreManagerImpl.java:984)
    at io.polaris.service.auth.TestInlineBearerTokenPolarisAuthenticator.authenticate(TestInlineBearerTokenPolarisAuthenticator.java:72)
    at io.polaris.service.auth.TestInlineBearerTokenPolarisAuthenticator.authenticate(TestInlineBearerTokenPolarisAuthenticator.java:45)
    at io.dropwizard.auth.AuthFilter.authenticate(AuthFilter.java:144)
    at io.dropwizard.auth.oauth.OAuthCredentialAuthFilter.filter(OAuthCredentialAuthFilter.java:37)
    at org.glassfish.jersey.server.ContainerFilteringStage.apply(ContainerFilteringStage.java:108)
    at org.glassfish.jersey.server.ContainerFilteringStage.apply(ContainerFilteringStage.java:44)
    at org.glassfish.jersey.process.internal.Stages.process(Stages.java:173)
    at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:248)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
    at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235)
    at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)
    at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
    at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:311)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764)
    at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1665)
    at io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:36)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:46)
    at io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:40)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at io.polaris.service.tracing.TracingFilter.doFilter(TracingFilter.java:91)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:314)
    at org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:267)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at io.polaris.service.PolarisApplication$ContextResolverFilter.doFilter(PolarisApplication.java:366)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1381)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1303)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
    at io.dropwizard.metrics.jetty11.InstrumentedHandler.handle(InstrumentedHandler.java:313)
    at io.dropwizard.jetty.RoutingHandler.handle(RoutingHandler.java:52)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
    at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822)
    at io.dropwizard.jetty.ZipExceptionHandlingGzipHandler.handle(ZipExceptionHandlingGzipHandler.java:26)
    at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:46)
    at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:173)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
    at org.eclipse.jetty.server.Server.handle(Server.java:563)
    at org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598)
    at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:287)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
    at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
    at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421)
    at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390)
    at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277)
    at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199)
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
    at java.lang.Thread.runWith(java.base@21/Thread.java:1596)
    at java.lang.Thread.run(java.base@21/Thread.java:1583)

"pool-3-thread-6" #33 [40] prio=5 os_prio=0 cpu=14128.56ms elapsed=6570.22s tid=0x00007fe178c630e0 nid=40 runnable  [0x00007fe150277000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPoll.wait(java.base@21/Native Method)
    at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@21/EPollSelectorImpl.java:121)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@21/SelectorImpl.java:130)
    - locked <0x00000000c323e810> (a sun.nio.ch.Util$2)
    - locked <0x00000000c323e7c0> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(java.base@21/SelectorImpl.java:147)
    at org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:181)
    at org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:188)
    at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:606)
    at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:542)
    at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:455)
    at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:248)
    at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199)
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
    at java.lang.Thread.runWith(java.base@21/Thread.java:1596)
    at java.lang.Thread.run(java.base@21/Thread.java:1583)

"pool-3-thread-8" #35 [42] prio=5 os_prio=0 cpu=7.76ms elapsed=6570.22s tid=0x00007fe178c65570 nid=42 waiting on condition  [0x00007fe0feffe000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@21/Native Method)
    - parking to wait for  <0x00000000c053bcd8> (a java.util.concurrent.SynchronousQueue$TransferStack)
    at java.util.concurrent.locks.LockSupport.parkNanos(java.base@21/LockSupport.java:269)
    at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@21/SynchronousQueue.java:401)
    at java.util.concurrent.SynchronousQueue.poll(java.base@21/SynchronousQueue.java:903)
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:325)
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:401)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
    at java.lang.Thread.runWith(java.base@21/Thread.java:1596)
    at java.lang.Thread.run(java.base@21/Thread.java:1583)

"pool-3-thread-9 - GET /api/management/v1/principals" #36 [43] prio=5 os_prio=0 cpu=12.62ms elapsed=6570.22s tid=0x00007fe178c667c0 nid=43 waiting for monitor entry  [0x00007fe0feefa000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at org.eclipse.persistence.sessions.server.ConnectionPool.acquireConnection(ConnectionPool.java:130)
    - waiting to lock <0x00000000c0ac2788> (a org.eclipse.persistence.sessions.server.ConnectionPool)
    at org.eclipse.persistence.sessions.server.ServerSession.getAccessors(ServerSession.java:580)
    at org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:598)
    at org.eclipse.persistence.sessions.server.ClientSession.executeCall(ClientSession.java:263)
    at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:280)
    at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:266)
    at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.selectOneRow(DatasourceCallQueryMechanism.java:813)
    at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectOneRowFromTable(ExpressionQueryMechanism.java:2912)
    at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectOneRow(ExpressionQueryMechanism.java:2865)
    at org.eclipse.persistence.queries.ReadObjectQuery.executeObjectLevelReadQuery(ReadObjectQuery.java:563)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1236)
    at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:913)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1195)
    at org.eclipse.persistence.queries.ReadObjectQuery.execute(ReadObjectQuery.java:448)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1283)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:3025)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1841)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1823)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1788)
    at org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:263)
    at org.eclipse.persistence.internal.jpa.QueryImpl.getResultList(QueryImpl.java:475)
    at jakarta.persistence.TypedQuery.getResultStream(TypedQuery.java:87)
    at io.polaris.extension.persistence.impl.eclipselink.PolarisEclipseLinkStore.lookupPrincipalSecrets(PolarisEclipseLinkStore.java:385)
    at io.polaris.extension.persistence.impl.eclipselink.PolarisEclipseLinkMetaStoreSessionImpl.loadPrincipalSecrets(PolarisEclipseLinkMetaStoreSessionImpl.java:558)
    at io.polaris.core.persistence.PolarisMetaStoreManagerImpl.loadPrincipalSecrets(PolarisMetaStoreManagerImpl.java:972)
    at io.polaris.core.persistence.PolarisMetaStoreManagerImpl.lambda$loadPrincipalSecrets$7(PolarisMetaStoreManagerImpl.java:984)
    at io.polaris.core.persistence.PolarisMetaStoreManagerImpl$$Lambda/0x00007fe10474bd68.get(Unknown Source)
    at io.polaris.extension.persistence.impl.eclipselink.PolarisEclipseLinkMetaStoreSessionImpl.runInTransaction(PolarisEclipseLinkMetaStoreSessionImpl.java:168)
    at io.polaris.core.persistence.PolarisMetaStoreManagerImpl.loadPrincipalSecrets(PolarisMetaStoreManagerImpl.java:984)
    at io.polaris.service.auth.TestInlineBearerTokenPolarisAuthenticator.authenticate(TestInlineBearerTokenPolarisAuthenticator.java:72)
    at io.polaris.service.auth.TestInlineBearerTokenPolarisAuthenticator.authenticate(TestInlineBearerTokenPolarisAuthenticator.java:45)
    at io.dropwizard.auth.AuthFilter.authenticate(AuthFilter.java:144)
    at io.dropwizard.auth.oauth.OAuthCredentialAuthFilter.filter(OAuthCredentialAuthFilter.java:37)
    at org.glassfish.jersey.server.ContainerFilteringStage.apply(ContainerFilteringStage.java:108)
    at org.glassfish.jersey.server.ContainerFilteringStage.apply(ContainerFilteringStage.java:44)
    at org.glassfish.jersey.process.internal.Stages.process(Stages.java:173)
    at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:248)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
    at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235)
    at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)
    at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
    at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:311)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764)
    at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1665)
    at io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:36)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:46)
    at io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:40)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at io.polaris.service.tracing.TracingFilter.doFilter(TracingFilter.java:91)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:314)
    at org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:267)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at io.polaris.service.PolarisApplication$ContextResolverFilter.doFilter(PolarisApplication.java:366)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1381)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1303)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
    at io.dropwizard.metrics.jetty11.InstrumentedHandler.handle(InstrumentedHandler.java:313)
    at io.dropwizard.jetty.RoutingHandler.handle(RoutingHandler.java:52)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
    at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822)
    at io.dropwizard.jetty.ZipExceptionHandlingGzipHandler.handle(ZipExceptionHandlingGzipHandler.java:26)
    at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:46)
    at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:173)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
    at org.eclipse.jetty.server.Server.handle(Server.java:563)
    at org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598)
    at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:287)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
    at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
    at java.lang.Thread.runWith(java.base@21/Thread.java:1596)
    at java.lang.Thread.run(java.base@21/Thread.java:1583)

"pool-3-thread-10 - GET /api/management/v1/principals" #37 [44] prio=5 os_prio=0 cpu=3102.88ms elapsed=6570.22s tid=0x00007fe178c67950 nid=44 runnable  [0x00007fe0fedf8000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.Net.poll(java.base@21/Native Method)
    at sun.nio.ch.NioSocketImpl.park(java.base@21/NioSocketImpl.java:191)
    at sun.nio.ch.NioSocketImpl.park(java.base@21/NioSocketImpl.java:201)
    at sun.nio.ch.NioSocketImpl.implRead(java.base@21/NioSocketImpl.java:309)
    at sun.nio.ch.NioSocketImpl.read(java.base@21/NioSocketImpl.java:346)
    at sun.nio.ch.NioSocketImpl$1.read(java.base@21/NioSocketImpl.java:796)
    at java.net.Socket$SocketInputStream.read(java.base@21/Socket.java:1099)
    at sun.security.ssl.SSLSocketInputRecord.read(java.base@21/SSLSocketInputRecord.java:489)
    at sun.security.ssl.SSLSocketInputRecord.readHeader(java.base@21/SSLSocketInputRecord.java:483)
    at sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(java.base@21/SSLSocketInputRecord.java:70)
    at sun.security.ssl.SSLSocketImpl.readApplicationRecord(java.base@21/SSLSocketImpl.java:1461)
    at sun.security.ssl.SSLSocketImpl$AppInputStream.read(java.base@21/SSLSocketImpl.java:1066)
    at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:162)
    at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:129)
    at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:114)
    at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:74)
    at org.postgresql.core.PGStream.receiveChar(PGStream.java:467)
    at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2166)
    at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:371)
    at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:502)
    at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:419)
    at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:194)
    at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:137)
    at org.eclipse.persistence.internal.databaseaccess.DatabasePlatform.wasFailureCommunicationBased(DatabasePlatform.java:3222)
    at org.eclipse.persistence.platform.server.ServerPlatformBase.wasFailureCommunicationBased(ServerPlatformBase.java:580)
    at org.eclipse.persistence.sessions.server.ConnectionPool.acquireConnection(ConnectionPool.java:175)
    - locked <0x00000000c0ac2788> (a org.eclipse.persistence.sessions.server.ConnectionPool)
    at org.eclipse.persistence.sessions.server.ServerSession.getAccessors(ServerSession.java:580)
    at org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:598)
    at org.eclipse.persistence.sessions.server.ClientSession.executeCall(ClientSession.java:263)
    at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:280)
    at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:266)
    at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.selectOneRow(DatasourceCallQueryMechanism.java:813)
    at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectOneRowFromTable(ExpressionQueryMechanism.java:2912)
    at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectOneRow(ExpressionQueryMechanism.java:2865)
    at org.eclipse.persistence.queries.ReadObjectQuery.executeObjectLevelReadQuery(ReadObjectQuery.java:563)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1236)
    at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:913)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1195)
    at org.eclipse.persistence.queries.ReadObjectQuery.execute(ReadObjectQuery.java:448)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1283)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:3025)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1841)
    at org.eclipse.persistence.internal.sessions.AbstractSession.retryQuery(AbstractSession.java:1912)
    at org.eclipse.persistence.sessions.server.ClientSession.retryQuery(ClientSession.java:704)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.retryQuery(UnitOfWorkImpl.java:5824)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1878)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1823)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1788)
    at org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:263)
    at org.eclipse.persistence.internal.jpa.QueryImpl.getResultList(QueryImpl.java:475)
    at jakarta.persistence.TypedQuery.getResultStream(TypedQuery.java:87)
    at io.polaris.extension.persistence.impl.eclipselink.PolarisEclipseLinkStore.lookupPrincipalSecrets(PolarisEclipseLinkStore.java:385)
    at io.polaris.extension.persistence.impl.eclipselink.PolarisEclipseLinkMetaStoreSessionImpl.loadPrincipalSecrets(PolarisEclipseLinkMetaStoreSessionImpl.java:558)
    at io.polaris.core.persistence.PolarisMetaStoreManagerImpl.loadPrincipalSecrets(PolarisMetaStoreManagerImpl.java:972)
    at io.polaris.core.persistence.PolarisMetaStoreManagerImpl.lambda$loadPrincipalSecrets$7(PolarisMetaStoreManagerImpl.java:984)
    at io.polaris.core.persistence.PolarisMetaStoreManagerImpl$$Lambda/0x00007fe10474bd68.get(Unknown Source)
    at io.polaris.extension.persistence.impl.eclipselink.PolarisEclipseLinkMetaStoreSessionImpl.runInTransaction(PolarisEclipseLinkMetaStoreSessionImpl.java:168)
    at io.polaris.core.persistence.PolarisMetaStoreManagerImpl.loadPrincipalSecrets(PolarisMetaStoreManagerImpl.java:984)
    at io.polaris.service.auth.TestInlineBearerTokenPolarisAuthenticator.authenticate(TestInlineBearerTokenPolarisAuthenticator.java:72)
    at io.polaris.service.auth.TestInlineBearerTokenPolarisAuthenticator.authenticate(TestInlineBearerTokenPolarisAuthenticator.java:45)
    at io.dropwizard.auth.AuthFilter.authenticate(AuthFilter.java:144)
    at io.dropwizard.auth.oauth.OAuthCredentialAuthFilter.filter(OAuthCredentialAuthFilter.java:37)
    at org.glassfish.jersey.server.ContainerFilteringStage.apply(ContainerFilteringStage.java:108)
    at org.glassfish.jersey.server.ContainerFilteringStage.apply(ContainerFilteringStage.java:44)
    at org.glassfish.jersey.process.internal.Stages.process(Stages.java:173)
    at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:248)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
    at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235)
    at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)
    at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
    at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:311)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764)
    at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1665)
    at io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:36)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:46)
    at io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:40)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at io.polaris.service.tracing.TracingFilter.doFilter(TracingFilter.java:91)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:314)
    at org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:267)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at io.polaris.service.PolarisApplication$ContextResolverFilter.doFilter(PolarisApplication.java:366)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1381)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1303)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
    at io.dropwizard.metrics.jetty11.InstrumentedHandler.handle(InstrumentedHandler.java:313)
    at io.dropwizard.jetty.RoutingHandler.handle(RoutingHandler.java:52)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
    at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822)
    at io.dropwizard.jetty.ZipExceptionHandlingGzipHandler.handle(ZipExceptionHandlingGzipHandler.java:26)
    at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:46)
    at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:173)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
    at org.eclipse.jetty.server.Server.handle(Server.java:563)
    at org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598)
    at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:287)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
    at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
    at java.lang.Thread.runWith(java.base@21/Thread.java:1596)
    at java.lang.Thread.run(java.base@21/Thread.java:1583)

"pool-4-thread-3-acceptor-0@1d6ca12f-admin@210308d5{HTTP/1.1, (http/1.1)}{0.0.0.0:8182}" #41 [47] prio=3 os_prio=0 cpu=1.41ms elapsed=6569.85s tid=0x00007fe178fa4c30 nid=47 runnable  [0x00007fe0feaf9000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.Net.accept(java.base@21/Native Method)
    at sun.nio.ch.ServerSocketChannelImpl.implAccept(java.base@21/ServerSocketChannelImpl.java:433)
    at sun.nio.ch.ServerSocketChannelImpl.accept(java.base@21/ServerSocketChannelImpl.java:399)
    at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:409)
    at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:748)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
    at java.lang.Thread.runWith(java.base@21/Thread.java:1596)
    at java.lang.Thread.run(java.base@21/Thread.java:1583)

"dw-awaiter" #42 [48] prio=5 os_prio=0 cpu=0.11ms elapsed=6569.85s tid=0x00007fe178fa69a0 nid=48 waiting on condition  [0x00007fe0fe9f8000]
   java.lang.Thread.State: WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@21/Native Method)
    - parking to wait for  <0x00000000c053c240> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(java.base@21/LockSupport.java:371)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@21/AbstractQueuedSynchronizer.java:519)
    at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21/ForkJoinPool.java:3780)
    at java.util.concurrent.ForkJoinPool.managedBlock(java.base@21/ForkJoinPool.java:3725)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21/AbstractQueuedSynchronizer.java:1707)
    at org.eclipse.jetty.util.thread.AutoLock$WithCondition.await(AutoLock.java:126)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.join(QueuedThreadPool.java:822)
    at org.eclipse.jetty.server.Server.join(Server.java:662)
    at io.dropwizard.core.cli.ServerCommand.lambda$run$0(ServerCommand.java:55)
    at io.dropwizard.core.cli.ServerCommand$$Lambda/0x00007fe104465420.run(Unknown Source)
    at java.lang.Thread.runWith(java.base@21/Thread.java:1596)
    at java.lang.Thread.run(java.base@21/Thread.java:1583)

"DestroyJavaVM" #43 [7] prio=5 os_prio=0 cpu=1648.09ms elapsed=6569.85s tid=0x00007fe178027e90 nid=7 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Scheduler-1741007954-1" #44 [49] prio=5 os_prio=0 cpu=27.57ms elapsed=6543.16s tid=0x00007fe0a40584f0 nid=49 waiting on condition  [0x00007fe150f84000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@21/Native Method)
    - parking to wait for  <0x00000000c323f260> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(java.base@21/LockSupport.java:269)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@21/AbstractQueuedSynchronizer.java:1758)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@21/ScheduledThreadPoolExecutor.java:1182)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@21/ScheduledThreadPoolExecutor.java:899)
    at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@21/ThreadPoolExecutor.java:1070)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21/ThreadPoolExecutor.java:1130)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21/ThreadPoolExecutor.java:642)
    at java.lang.Thread.runWith(java.base@21/Thread.java:1596)
    at java.lang.Thread.run(java.base@21/Thread.java:1583)

"PostgreSQL-JDBC-Cleaner" #45 [51] daemon prio=5 os_prio=0 cpu=11.76ms elapsed=6542.48s tid=0x00007fe09c7920a0 nid=51 waiting on condition  [0x00007fe0fe3d7000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@21/Native Method)
    - parking to wait for  <0x00000000c323f858> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(java.base@21/LockSupport.java:269)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21/AbstractQueuedSynchronizer.java:1847)
    at java.lang.ref.ReferenceQueue.await(java.base@21/ReferenceQueue.java:71)
    at java.lang.ref.ReferenceQueue.remove0(java.base@21/ReferenceQueue.java:143)
    at java.lang.ref.ReferenceQueue.remove(java.base@21/ReferenceQueue.java:218)
    at org.postgresql.util.LazyCleaner$1.run(LazyCleaner.java:129)
    at java.lang.Thread.runWith(java.base@21/Thread.java:1596)
    at java.lang.Thread.run(java.base@21/Thread.java:1583)

"pool-3-thread-11" #46 [52] prio=5 os_prio=0 cpu=6.59ms elapsed=6509.69s tid=0x00007fe088000e70 nid=52 waiting on condition  [0x00007fe0fe8f7000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@21/Native Method)
    - parking to wait for  <0x00000000c053e388> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(java.base@21/LockSupport.java:269)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@21/AbstractQueuedSynchronizer.java:1758)
    at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
    at java.lang.Thread.runWith(java.base@21/Thread.java:1596)
    at java.lang.Thread.run(java.base@21/Thread.java:1583)

"PostgreSQL-JDBC-SharedTimer-1" #47 [53] daemon prio=5 os_prio=0 cpu=251.21ms elapsed=6500.01s tid=0x00007fe09c6f3370 nid=53 in Object.wait()  [0x00007fe0fe2d6000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait0(java.base@21/Native Method)
    - waiting on <no object reference available>
    at java.lang.Object.wait(java.base@21/Object.java:366)
    at java.lang.Object.wait(java.base@21/Object.java:339)
    at java.util.TimerThread.mainLoop(java.base@21/Timer.java:537)
    - locked <0x00000000c2b9f790> (a java.util.TaskQueue)
    at java.util.TimerThread.run(java.base@21/Timer.java:516)

"org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner" #49 [57] daemon prio=5 os_prio=0 cpu=0.24ms elapsed=6498.70s tid=0x00007fe0980cdbe0 nid=57 waiting on condition  [0x00007fe0fdfd3000]
   java.lang.Thread.State: WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@21/Native Method)
    - parking to wait for  <0x00000000c28524d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(java.base@21/LockSupport.java:371)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@21/AbstractQueuedSynchronizer.java:519)
    at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21/ForkJoinPool.java:3780)
    at java.util.concurrent.ForkJoinPool.managedBlock(java.base@21/ForkJoinPool.java:3725)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21/AbstractQueuedSynchronizer.java:1707)
    at java.lang.ref.ReferenceQueue.await(java.base@21/ReferenceQueue.java:67)
    at java.lang.ref.ReferenceQueue.remove0(java.base@21/ReferenceQueue.java:158)
    at java.lang.ref.ReferenceQueue.remove(java.base@21/ReferenceQueue.java:234)
    at org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:4159)
    at java.lang.Thread.runWith(java.base@21/Thread.java:1596)
    at java.lang.Thread.run(java.base@21/Thread.java:1583)

"idle-connection-reaper" #100 [125] daemon prio=5 os_prio=0 cpu=10.38ms elapsed=5811.59s tid=0x00007fe09c5442d0 nid=125 waiting on condition  [0x00007fe0fded2000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep0(java.base@21/Native Method)
    at java.lang.Thread.sleep(java.base@21/Thread.java:509)
    at software.amazon.awssdk.http.apache.internal.conn.IdleConnectionReaper$ReaperTask.run(IdleConnectionReaper.java:151)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21/ThreadPoolExecutor.java:1144)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21/ThreadPoolExecutor.java:642)
    at java.lang.Thread.runWith(java.base@21/Thread.java:1596)
    at java.lang.Thread.run(java.base@21/Thread.java:1583)

"pool-4-thread-4" #101 [127] prio=5 os_prio=0 cpu=19.71ms elapsed=1297.92s tid=0x00007fe084001690 nid=127 waiting on condition  [0x00007fe0fe0d4000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@21/Native Method)
    - parking to wait for  <0x00000000c322f9d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(java.base@21/LockSupport.java:269)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@21/AbstractQueuedSynchronizer.java:1758)
    at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
    at java.lang.Thread.runWith(java.base@21/Thread.java:1596)
    at java.lang.Thread.run(java.base@21/Thread.java:1583)

"Scheduler-1351247295-1" #102 [128] prio=5 os_prio=0 cpu=0.45ms elapsed=1297.91s tid=0x00007fe06c001dd0 nid=128 waiting on condition  [0x00007fe150176000]
   java.lang.Thread.State: WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@21/Native Method)
    - parking to wait for  <0x00000000c0a083f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(java.base@21/LockSupport.java:371)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@21/AbstractQueuedSynchronizer.java:519)
    at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21/ForkJoinPool.java:3780)
    at java.util.concurrent.ForkJoinPool.managedBlock(java.base@21/ForkJoinPool.java:3725)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21/AbstractQueuedSynchronizer.java:1707)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@21/ScheduledThreadPoolExecutor.java:1170)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@21/ScheduledThreadPoolExecutor.java:899)
    at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@21/ThreadPoolExecutor.java:1070)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21/ThreadPoolExecutor.java:1130)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21/ThreadPoolExecutor.java:642)
    at java.lang.Thread.runWith(java.base@21/Thread.java:1596)
    at java.lang.Thread.run(java.base@21/Thread.java:1583)

"pool-4-thread-7" #105 [132] prio=5 os_prio=0 cpu=11.01ms elapsed=689.51s tid=0x00007fe084003a40 nid=132 runnable  [0x00007fe0fecfb000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPoll.wait(java.base@21/Native Method)
    at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@21/EPollSelectorImpl.java:121)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@21/SelectorImpl.java:130)
    - locked <0x00000000c323e370> (a sun.nio.ch.Util$2)
    - locked <0x00000000c323e320> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(java.base@21/SelectorImpl.java:147)
    at org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:181)
    at org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:188)
    at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:606)
    at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:542)
    at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:455)
    at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:248)
    at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199)
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
    at java.lang.Thread.runWith(java.base@21/Thread.java:1596)
    at java.lang.Thread.run(java.base@21/Thread.java:1583)

"Attach Listener" #108 [213] daemon prio=9 os_prio=0 cpu=0.51ms elapsed=525.21s tid=0x00007fe0e800c650 nid=213 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"pool-4-thread-10" #109 [214] prio=5 os_prio=0 cpu=9.54ms elapsed=194.35s tid=0x00007fe06c01d3a0 nid=214 runnable  [0x00007fe0fddd1000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPoll.wait(java.base@21/Native Method)
    at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@21/EPollSelectorImpl.java:121)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@21/SelectorImpl.java:130)
    - locked <0x00000000c323e6d8> (a sun.nio.ch.Util$2)
    - locked <0x00000000c323e688> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(java.base@21/SelectorImpl.java:147)
    at org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:181)
    at org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:188)
    at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:606)
    at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:542)
    at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:455)
    at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:248)
    at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199)
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
    at java.lang.Thread.runWith(java.base@21/Thread.java:1596)
    at java.lang.Thread.run(java.base@21/Thread.java:1583)

"pool-4-thread-12" #111 [216] prio=5 os_prio=0 cpu=1.30ms elapsed=193.18s tid=0x00007fe06c023a90 nid=216 waiting on condition  [0x00007fe0fdcd0000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@21/Native Method)
    - parking to wait for  <0x00000000c322f9d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(java.base@21/LockSupport.java:269)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@21/AbstractQueuedSynchronizer.java:1758)
    at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
    at java.lang.Thread.runWith(java.base@21/Thread.java:1596)
    at java.lang.Thread.run(java.base@21/Thread.java:1583)

"VM Thread" os_prio=0 cpu=258.58ms elapsed=6571.53s tid=0x00007fe1780e9ca0 nid=14 runnable

"GC Thread#0" os_prio=0 cpu=559.30ms elapsed=6571.54s tid=0x00007fe178062d00 nid=8 runnable

"GC Thread#1" os_prio=0 cpu=543.52ms elapsed=6571.27s tid=0x00007fe0f8006860 nid=25 runnable

"GC Thread#2" os_prio=0 cpu=562.83ms elapsed=6571.27s tid=0x00007fe0f800b860 nid=26 runnable

"GC Thread#3" os_prio=0 cpu=551.38ms elapsed=6571.01s tid=0x00007fe0f800ab50 nid=27 runnable

"G1 Main Marker" os_prio=0 cpu=13.26ms elapsed=6571.54s tid=0x00007fe17806c2f0 nid=9 runnable

"G1 Conc#0" os_prio=0 cpu=2381.05ms elapsed=6571.54s tid=0x00007fe17806d290 nid=10 runnable

"G1 Refine#0" os_prio=0 cpu=993.27ms elapsed=6571.54s tid=0x00007fe1780b4b10 nid=11 runnable

"G1 Service" os_prio=0 cpu=131.94ms elapsed=6571.54s tid=0x00007fe1780b5ac0 nid=12 runnable

"VM Periodic Task Thread" os_prio=0 cpu=1985.42ms elapsed=6571.54s tid=0x00007fe1780cf980 nid=13 waiting on condition

JNI global refs: 16, weak refs: 0

According to the thread dump, there are couple blocked threads and there seems to be a potential of deadlock.

MonkeyCanCode commented 2 months ago

This was not caused by Polaris nor backend but something else.