search

apache / pinot

Apache Pinot - A realtime distributed OLAP datastore
https://pinot.apache.org/
Apache License 2.0
5.26k stars 1.23k forks source link

Segments become Unavailable and fail to recover upon restarting server pods. #12773

Open rajatgupta063 opened 3 months ago

rajatgupta063 commented 3 months ago

I have a pinot cluster running on AWS EKS. My pinot cluster is continuously reading data from kinesis stream. The cluster has around 700GB of data with more than 12B records. Total number of segments ~ 140K

When I ran a sample query, it returned large number of rows probably more than 1B which resulted the pinot server to go out of memory as JVM heap was configured to be 10G. Post that noticing the following the issues in the cluster

  1. 99K Segments are unavailable and haven't recovered yet. (even post restarting the server pods)
  2. Broker pods are stuck in crash loop, seeing the following errors in logs.
  3. The pinot UI does not show any documents when I try to query, even the data from available segments (40K) is not returned.
  4. I have a real time for which I have configured that segment threshold rows to be 10M but I am seeing segments getting flushed with only 200K rows. What is the rationale behind this behavior?
2024-04-01T20:46:04.623-07:00   EventThread shut down for session: 0x3001644494f018a
    2024-04-01T20:46:04.623-07:00   EventThread shut down for session: 0x3001644494f018a
    2024-04-01T20:46:04.624-07:00   MBean HelixZkClient:Key=pinot-on-eks-cluster.Broker_pinot-broker-2.pinot-broker-headless.pinot.svc.cluster.local_8099,Type=PARTICIPANT has been un-registered.
    2024-04-01T20:46:04.624-07:00   MBean HelixZkClient:Key=pinot-on-eks-cluster.Broker_pinot-broker-2.pinot-broker-headless.pinot.svc.cluster.local_8099,Type=PARTICIPANT has been un-registered.
    2024-04-01T20:46:04.624-07:00   MBean HelixZkClient:Key=pinot-on-eks-cluster.Broker_pinot-broker-2.pinot-broker-headless.pinot.svc.cluster.local_8099,PATH=Root,Type=PARTICIPANT has been un-registered.
    2024-04-01T20:46:04.624-07:00   MBean HelixZkClient:Key=pinot-on-eks-cluster.Broker_pinot-broker-2.pinot-broker-headless.pinot.svc.cluster.local_8099,PATH=Root,Type=PARTICIPANT has been un-registered.
    2024-04-01T20:46:04.624-07:00   Closed zkclient with uid:6
    2024-04-01T20:46:04.624-07:00   Closed zkclient with uid:6
    2024-04-01T20:46:04.624-07:00   Cluster manager: Broker_pinot-broker-2.pinot-broker-headless.pinot.svc.cluster.local_8099 disconnected
    2024-04-01T20:46:04.624-07:00   Cluster manager: Broker_pinot-broker-2.pinot-broker-headless.pinot.svc.cluster.local_8099 disconnected
    2024-04-01T20:46:04.624-07:00   Stopping cluster change mediator
    2024-04-01T20:46:04.624-07:00   Stopping cluster change mediator
    2024-04-01T20:46:04.624-07:00   Stopping ClusterChangeMediator
    2024-04-01T20:46:04.624-07:00   Stopping ClusterChangeMediator
    2024-04-01T20:46:04.625-07:00   Wait for 10000ms before shutting down request handler to finish the pending queries
    2024-04-01T20:46:04.625-07:00   Wait for 10000ms before shutting down request handler to finish the pending queries
    2024-04-01T20:46:14.457-07:00   Caught exception while reading the service status
    2024-04-01T20:46:14.457-07:00   Caught exception while reading the service status
    2024-04-01T20:46:14.457-07:00   java.lang.IllegalStateException: ZkClient already closed!
    2024-04-01T20:46:14.457-07:00   java.lang.IllegalStateException: ZkClient already closed!
    2024-04-01T20:46:14.457-07:00   at org.apache.helix.zookeeper.zkclient.ZkClient.retryUntilConnected(ZkClient.java:2070) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.apache.helix.zookeeper.zkclient.ZkClient.retryUntilConnected(ZkClient.java:2070) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.apache.helix.zookeeper.zkclient.ZkClient.readData(ZkClient.java:2244) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.apache.helix.zookeeper.zkclient.ZkClient.readData(ZkClient.java:2244) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.apache.helix.zookeeper.zkclient.ZkClient.readData(ZkClient.java:2235) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.apache.helix.zookeeper.zkclient.ZkClient.readData(ZkClient.java:2235) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.apache.helix.manager.zk.ZkBaseDataAccessor.get(ZkBaseDataAccessor.java:495) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.apache.helix.manager.zk.ZkBaseDataAccessor.get(ZkBaseDataAccessor.java:495) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.apache.helix.manager.zk.ZKHelixDataAccessor.getProperty(ZKHelixDataAccessor.java:292) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.apache.helix.manager.zk.ZKHelixDataAccessor.getProperty(ZKHelixDataAccessor.java:292) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.apache.helix.manager.zk.ZKHelixAdmin.getResourceIdealState(ZKHelixAdmin.java:1255) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.apache.helix.manager.zk.ZKHelixAdmin.getResourceIdealState(ZKHelixAdmin.java:1255) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.apache.pinot.common.utils.ServiceStatus$IdealStateMatchServiceStatusCallback.getResourceIdealState(ServiceStatus.java:491) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.apache.pinot.common.utils.ServiceStatus$IdealStateMatchServiceStatusCallback.getResourceIdealState(ServiceStatus.java:491) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.apache.pinot.common.utils.ServiceStatus$IdealStateMatchServiceStatusCallback.evaluateResourceStatus(ServiceStatus.java:431) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.apache.pinot.common.utils.ServiceStatus$IdealStateMatchServiceStatusCallback.evaluateResourceStatus(ServiceStatus.java:431) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.apache.pinot.common.utils.ServiceStatus$IdealStateMatchServiceStatusCallback.getServiceStatus(ServiceStatus.java:374) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.apache.pinot.common.utils.ServiceStatus$IdealStateMatchServiceStatusCallback.getServiceStatus(ServiceStatus.java:374) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.apache.pinot.common.utils.ServiceStatus$IdealStateAndCurrentStateMatchServiceStatusCallback.getServiceStatus(ServiceStatus.java:500) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.apache.pinot.common.utils.ServiceStatus$IdealStateAndCurrentStateMatchServiceStatusCallback.getServiceStatus(ServiceStatus.java:500) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.apache.pinot.common.utils.ServiceStatus$MultipleCallbackServiceStatusCallback.getServiceStatus(ServiceStatus.java:179) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.apache.pinot.common.utils.ServiceStatus$MultipleCallbackServiceStatusCallback.getServiceStatus(ServiceStatus.java:179) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.apache.pinot.common.utils.ServiceStatus.getServiceStatus(ServiceStatus.java:84) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.apache.pinot.common.utils.ServiceStatus.getServiceStatus(ServiceStatus.java:84) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.apache.pinot.common.utils.ServiceStatus.getServiceStatus(ServiceStatus.java:76) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.apache.pinot.common.utils.ServiceStatus.getServiceStatus(ServiceStatus.java:76) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.apache.pinot.broker.api.resources.PinotBrokerHealthCheck.getBrokerHealth(PinotBrokerHealthCheck.java:78) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.apache.pinot.broker.api.resources.PinotBrokerHealthCheck.getBrokerHealth(PinotBrokerHealthCheck.java:78) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
    2024-04-01T20:46:14.457-07:00   at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
    2024-04-01T20:46:14.457-07:00   at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
    2024-04-01T20:46:14.457-07:00   at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
    2024-04-01T20:46:14.457-07:00   at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
    2024-04-01T20:46:14.457-07:00   at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
    2024-04-01T20:46:14.457-07:00   at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
    2024-04-01T20:46:14.457-07:00   at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
    2024-04-01T20:46:14.457-07:00   at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:134) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:134) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:177) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:177) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:81) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:81) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.glassfish.jersey.internal.Errors.process(Errors.java:244) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.glassfish.jersey.internal.Errors.process(Errors.java:244) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:356) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:356) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:200) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:200) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:569) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:569) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:549) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:549) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-28aec2e01cedfcfaf47e83f41f58f810ac153b31]
    2024-04-01T20:46:14.457-07:00   at java.base/java.lang.Thread.run(Thread.java:829) [?:?]
    2024-04-01T20:46:14.457-07:00   at java.base/java.lang.Thread.run(Thread.java:829) [?:?]
    2024-04-01T20:46:14.625-07:00   Shutting down request handler and broker admin application
    2024-04-01T20:46:14.625-07:00   Shutting down request handler and broker admin application
    2024-04-01T20:46:14.642-07:00   Shutting down GrpcMailboxServer
    2024-04-01T20:46:14.642-07:00   Shutting down GrpcMailboxServer
    2024-04-01T20:46:14.647-07:00   Shutting down http server
    2024-04-01T20:46:14.647-07:00   Shutting down http server
    2024-04-01T20:46:14.661-07:00   Shutting down executor service
    2024-04-01T20:46:14.661-07:00   Shutting down executor service
    2024-04-01T20:46:14.661-07:00   Disconnecting spectator Helix manager
    2024-04-01T20:46:14.661-07:00   Disconnecting spectator Helix manager
Screenshot 2024-04-01 at 5 10 15 PM
Jackie-Jiang commented 3 months ago

Regarding 2: Why is the same log printed twice? Is it possible that you have 2 brokers running on the same node? Regarding 4: You may take a look at #12058