search

pinpoint-apm / pinpoint

APM, (Application Performance Management) tool for large-scale distributed systems.
https://pinpoint-apm.gitbook.io/
Apache License 2.0
13.42k stars 3.75k forks source link

exceptionTrace is not activated. #11170

Open pan889 opened 4 months ago

pan889 commented 4 months ago

I am setting version 3.0.0. However, an error occurred while creating the exceptionTrace table in pinot.

image

image

It was confirmed that the schemaName was incorrect, and the table was applied after correction. pinot admin confirmed that the status was not normal as shown in the picture below.

image

image

intr3p1d commented 4 months ago

Thank you for pointing out the configuration error. To address this issue, I will modify the existing table and schema files.

Based on your feedback, it seems that changing the incorrect table name (exceptionTraceV1) to exceptionTrace resolved the error. Is that correct?

If there are any further issues, please let me know.

pan889 commented 4 months ago

I need some other help. When inserting a table with the applied schema, a segment error occurs as shown below. image All other tables are normal, but only exceptionTrace is occurring.

We are using version 1.0.0 of pinot and are operating in hybrid mode of QuickStartCluster.

image

donghun-cho commented 4 months ago

The segment might not be saved correctly due to the schema change. For more information https://docs.pinot.apache.org/users/tutorials/schema-evolution.

Try using the Reload Table Segments API to reload table segments. Or the Delete Table Segment API to delete segment exceptionTrace__0__0__20240624T0121Z.

These actions can be done with the Swagger Pinot Controller API.

image
pan889 commented 4 months ago

I've tried reloading segments, rebalance servers, and deleting and regenerating segments, but all I get is a message that the segment is not normal.

image

donghun-cho commented 4 months ago

I am not sure if the segment was deleted properly because the segment name is the same. Can you explain more about how you regenerate segments?

These are some solution with more details

delete segment

  1. deleting with api and check response image

  2. (optional) Use forceCommit /tables/{tableName}/forceCommit to create a new segment and check if it is created correctly. image

If you need to restore realtime segment

https://www.markhneedham.com/blog/2021/12/06/apache-pinot-copy-segment-new-table/

  1. download segment from deep storage(controller) or server
  2. Copy segment to offline table
pan889 commented 4 months ago

Thanks for your guide.

Like your guide, I proceeded as follows.

image image

However, segment errors continued to occur.

image image
donghun-cho commented 4 months ago

I've never seen a case like this before. I'll check pinot issue to see if there are any similar issues.

If you have any logs, please share them with me.

pan889 commented 4 months ago

my pinot verision : 1.0.0 cpu : aws r7g (Graviton) java version : 17 start script : apache-pinot-1.0.0-bin/bin/pinot-admin.sh QuickStart -type hybrid

This is the script used to insert the 3.0.0 version table. image

All tables except exceptionTrace are normal and data collection is working well. The log below is the message that appears when a segment is reloaded. It's a bit difficult to see because it's mixed in with other things.

2024/06/24 08:52:27.545 INFO [PinotHelixResourceManager] [grizzly-http-server-0] Sending reload message for segment: exceptionTrace0020240624T0630Z in table: exceptionTrace_REALTIME with forceDownload: false 2024/06/24 08:52:27.546 INFO [CriteriaEvaluator] [grizzly-http-server-0] Query returned 1 rows 2024/06/24 08:52:27.546 INFO [DefaultMessagingService] [grizzly-http-server-0] Send 1 messages with criteria instanceName=%resourceName=exceptionTrace_REALTIMEpartitionName=exceptionTrace0020240624T0630ZpartitionState= 2024/06/24 08:52:27.547 INFO [PinotHelixResourceManager] [grizzly-http-server-0] Sent 1 reload messages for segment: exceptionTrace0020240624T0630Z in table: exceptionTrace_REALTIME 2024/06/24 08:52:27.547 INFO [CallbackHandler] [ZkClient-EventThread-188-localhost:2123] 188 START: CallbackHandler 27, INVOKE /QuickStartCluster/INSTANCES/Server_10.220.207.212_7050/MESSAGES listener: org.apache.helix.controller.GenericHelixController@3b11deb6 type: CALLBACK 2024/06/24 08:52:27.547 INFO [CallbackHandler] [ZkClient-EventThread-188-localhost:2123] CallbackHandler 27 subscribing changes listener to path: /QuickStartCluster/INSTANCES/Server_10.220.207.212_7050/MESSAGES, callback type: CALLBACK, event types: [NodeChildrenChanged], listener: org.apache.helix.controller.GenericHelixController@3b11deb6, watchChild: false 2024/06/24 08:52:27.547 INFO [CallbackHandler] [ZkClient-EventThread-355-localhost:2123] 355 START: CallbackHandler 29, INVOKE /QuickStartCluster/INSTANCES/Server_10.220.207.212_7050/MESSAGES listener: org.apache.helix.messaging.handling.HelixTaskExecutor@5fd6d31f type: CALLBACK 2024/06/24 08:52:27.547 INFO [CallbackHandler] [ZkClient-EventThread-355-localhost:2123] CallbackHandler 29 subscribing changes listener to path: /QuickStartCluster/INSTANCES/Server_10.220.207.212_7050/MESSAGES, callback type: CALLBACK, event types: [NodeChildrenChanged], listener: org.apache.helix.messaging.handling.HelixTaskExecutor@5fd6d31f, watchChild: false 2024/06/24 08:52:27.548 INFO [ControllerResponseFilter] [grizzly-http-server-0] Handled request from 10.220.132.137 POST http://-/segments/exceptionTrace_REALTIME/exceptionTrace0020240624T0630Z/reload, content-type application/json; charset=UTF-8 status code 200 OK 2024/06/24 08:52:27.548 INFO [CallbackHandler] [ZkClient-EventThread-188-localhost:2123] CallbackHandler27, Subscribing to path: /QuickStartCluster/INSTANCES/Server_10.220.207.212_7050/MESSAGES took: 1 2024/06/24 08:52:27.548 INFO [GenericHelixController] [ZkClient-EventThread-188-localhost:2123] START: GenericClusterController.onMessage() for cluster QuickStartCluster 2024/06/24 08:52:27.548 INFO [CallbackHandler] [ZkClient-EventThread-355-localhost:2123] CallbackHandler29, Subscribing to path: /QuickStartCluster/INSTANCES/Server_10.220.207.212_7050/MESSAGES took: 0 2024/06/24 08:52:27.548 INFO [GenericHelixController] [ZkClient-EventThread-188-localhost:2123] END: GenericClusterController.onMessage() for cluster QuickStartCluster 2024/06/24 08:52:27.548 INFO [CallbackHandler] [ZkClient-EventThread-188-localhost:2123] 188 END:INVOKE CallbackHandler 27, /QuickStartCluster/INSTANCES/Server_10.220.207.212_7050/MESSAGES listener: org.apache.helix.controller.GenericHelixController@3b11deb6 type: CALLBACK Took: 1ms 2024/06/24 08:52:27.549 INFO [GenericHelixController] [HelixController-pipeline-default-QuickStartCluster-(2f323e54_DEFAULT)] Controller stopping periodical rebalance timer at period 9223372036854775807 2024/06/24 08:52:27.549 INFO [GenericHelixController] [HelixController-pipeline-task-QuickStartCluster-(2f323e54_TASK)] Controller stopping periodical rebalance timer at period 9223372036854775807 2024/06/24 08:52:27.549 INFO [GenericHelixController] [HelixController-pipeline-task-QuickStartCluster-(2f323e54_TASK)] START: Invoking TASK controller pipeline for cluster: QuickStartCluster. Event type: MessageChange, ID: 2f323e54_TASK. Event session ID: 10004f0fb490005 2024/06/24 08:52:27.549 INFO [GenericHelixController] [HelixController-pipeline-default-QuickStartCluster-(2f323e54_DEFAULT)] START: Invoking DEFAULT controller pipeline for cluster: QuickStartCluster. Event type: MessageChange, ID: 2f323e54_DEFAULT. Event session ID: 10004f0fb490005 2024/06/24 08:52:27.549 INFO [BaseControllerDataProvider] [HelixController-pipeline-default-QuickStartCluster-(2f323e54_DEFAULT)] Event 2f323e54_DEFAULT : No ideal state change for QuickStartCluster cluster, DEFAULT pipeline 2024/06/24 08:52:27.549 INFO [BaseControllerDataProvider] [HelixController-pipeline-task-QuickStartCluster-(2f323e54_TASK)] Event 2f323e54_TASK : No ideal state change for QuickStartCluster cluster, TASK pipeline 2024/06/24 08:52:27.549 INFO [BaseControllerDataProvider] [HelixController-pipeline-default-QuickStartCluster-(2f323e54_DEFAULT)] Event 2f323e54_DEFAULT : No live instance change for QuickStartCluster cluster, DEFAULT pipeline 2024/06/24 08:52:27.549 INFO [BaseControllerDataProvider] [HelixController-pipeline-task-QuickStartCluster-(2f323e54_TASK)] Event 2f323e54_TASK : No live instance change for QuickStartCluster cluster, TASK pipeline 2024/06/24 08:52:27.549 INFO [BaseControllerDataProvider] [HelixController-pipeline-task-QuickStartCluster-(2f323e54_TASK)] Event 2f323e54_TASK : No instance config change for QuickStartCluster cluster, TASK pipeline 2024/06/24 08:52:27.549 INFO [BaseControllerDataProvider] [HelixController-pipeline-default-QuickStartCluster-(2f323e54_DEFAULT)] Event 2f323e54_DEFAULT : No instance config change for QuickStartCluster cluster, DEFAULT pipeline 2024/06/24 08:52:27.549 INFO [BaseControllerDataProvider] [HelixController-pipeline-default-QuickStartCluster-(2f323e54_DEFAULT)] Event 2f323e54_DEFAULT : No resource config change for QuickStartCluster cluster, DEFAULT pipeline 2024/06/24 08:52:27.549 INFO [BaseControllerDataProvider] [HelixController-pipeline-task-QuickStartCluster-(2f323e54_TASK)] Event 2f323e54_TASK : No resource config change for QuickStartCluster cluster, TASK pipeline 2024/06/24 08:52:27.550 INFO [MessageLatencyMonitor] [ZkClient-EventThread-355-localhost:2123] The latency of message 8a579eec-c767-4459-9aa6-2edf5b9e45f9 is 5 ms 2024/06/24 08:52:27.551 INFO [PropertyCache] [HelixController-pipeline-default-QuickStartCluster-(2f323e54_DEFAULT)] Event QuickStartCluster::DEFAULT::2f323e54_DEFAULT : Refreshed 9 property StateModelDefinition took 2 ms. Selective: false 2024/06/24 08:52:27.551 INFO [HelixTaskExecutor] [ZkClient-EventThread-355-localhost:2123] Scheduling message 8a579eec-c767-4459-9aa6-2edf5b9e45f9: exceptionTrace_REALTIME:exceptionTrace0020240624T0630Z, null->null 2024/06/24 08:52:27.551 INFO [HelixTaskExecutor] [ZkClient-EventThread-355-localhost:2123] Submit task: 8a579eec-c767-4459-9aa6-2edf5b9e45f9 to pool: java.util.concurrent.ThreadPoolExecutor@50eeed6e[Running, pool size = 33, active threads = 0, queued tasks = 0, completed tasks = 33] 2024/06/24 08:52:27.551 INFO [HelixTaskExecutor] [ZkClient-EventThread-355-localhost:2123] Message: 8a579eec-c767-4459-9aa6-2edf5b9e45f9 handling task scheduled 2024/06/24 08:52:27.551 INFO [CallbackHandler] [ZkClient-EventThread-355-localhost:2123] 355 END:INVOKE CallbackHandler 29, /QuickStartCluster/INSTANCES/Server_10.220.207.212_7050/MESSAGES listener: org.apache.helix.messaging.handling.HelixTaskExecutor@5fd6d31f type: CALLBACK Took: 4ms 2024/06/24 08:52:27.552 INFO [PropertyCache] [HelixController-pipeline-task-QuickStartCluster-(2f323e54_TASK)] Event QuickStartCluster::TASK::2f323e54_TASK : Refreshed 9 property StateModelDefinition took 3 ms. Selective: false 2024/06/24 08:52:27.552 INFO [HelixTask] [HelixTaskExecutor-message_handle_thread_233] handling task: 8a579eec-c767-4459-9aa6-2edf5b9e45f9 begin, at: 1719219147552 2024/06/24 08:52:27.552 INFO [ConstraintItemBuilder] [HelixController-pipeline-default-QuickStartCluster-(2f323e54_DEFAULT)] overwrite existing constraint-value. old-value: null, new-value: 100000 2024/06/24 08:52:27.552 INFO [PropertyCache] [HelixController-pipeline-default-QuickStartCluster-(2f323e54_DEFAULT)] Event QuickStartCluster::DEFAULT::2f323e54_DEFAULT : Refreshed 1 property ClusterConstraint took 1 ms. Selective: false 2024/06/24 08:52:27.552 INFO [exceptionTrace_REALTIME-SegmentReloadMessageHandler] [HelixTaskExecutor-message_handle_thread_233] Handling message: ZnRecord=8a579eec-c767-4459-9aa6-2edf5b9e45f9, {CREATE_TIMESTAMP=1719219147545, EXECUTE_START_TIMESTAMP=1719219147552, MSG_ID=8a579eec-c767-4459-9aa6-2edf5b9e45f9, MSG_STATE=new, MSG_SUBTYPE=RELOAD_SEGMENT, MSG_TYPE=USER_DEFINE_MSG, PARTITION_NAME=exceptionTrace0020240624T0630Z, RESOURCE_NAME=exceptionTrace_REALTIME, RETRY_COUNT=0, SRC_CLUSTER=QuickStartCluster, SRC_INSTANCE_TYPE=PARTICIPANT, SRC_NAME=Controller_10.220.207.212_9000, TGT_NAME=Server_10.220.207.212_7050, TGT_SESSION_ID=10004f0fb490015, TIMEOUT=-1, forceDownload=false}{}{segmentNames=[exceptionTrace0020240624T0630Z]}, Stat=Stat {_version=0, _creationTime=1719219147546, _modifiedTime=1719219147546, _ephemeralOwner=0} 2024/06/24 08:52:27.552 INFO [HelixInstanceDataManager] [HelixTaskExecutor-message_handle_thread_233] Reloading multiple segments: [exceptionTrace0020240624T0630Z] in table: exceptionTrace_REALTIME 2024/06/24 08:52:27.552 WARN [HelixInstanceDataManager] [HelixTaskExecutor-message_handle_thread_233] Failed to get segment data manager for segments: [exceptionTrace00__20240624T0630Z] of table: org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager@467f8cc1, skipping reloading them 2024/06/24 08:52:27.552 INFO [ConstraintItemBuilder] [HelixController-pipeline-task-QuickStartCluster-(2f323e54_TASK)] overwrite existing constraint-value. old-value: null, new-value: 100000 2024/06/24 08:52:27.552 INFO [PropertyCache] [HelixController-pipeline-task-QuickStartCluster-(2f323e54_TASK)] Event QuickStartCluster::TASK::2f323e54_TASK : Refreshed 1 property ClusterConstraint took 0 ms. Selective: false 2024/06/24 08:52:27.553 INFO [InstanceMessagesCache] [HelixController-pipeline-default-QuickStartCluster-(2f323e54_DEFAULT)] START: InstanceMessagesCache.refresh() 2024/06/24 08:52:27.553 INFO [InstanceMessagesCache] [HelixController-pipeline-task-QuickStartCluster-(2f323e54_TASK)] START: InstanceMessagesCache.refresh() 2024/06/24 08:52:27.553 INFO [HelixInstanceDataManager] [HelixTaskExecutor-message_handle_thread_233] Reloaded segments with metadata in table: exceptionTrace_REALTIME. Duration: 1 2024/06/24 08:52:27.554 INFO [HelixTask] [HelixTaskExecutor-message_handle_thread_233] Message 8a579eec-c767-4459-9aa6-2edf5b9e45f9 completed. 2024/06/24 08:52:27.554 INFO [InstanceMessagesCache] [HelixController-pipeline-default-QuickStartCluster-(2f323e54_DEFAULT)] END: InstanceMessagesCache.refresh(), 1 of Messages read from ZooKeeper. took 1 ms. 2024/06/24 08:52:27.554 INFO [InstanceMessagesCache] [HelixController-pipeline-default-QuickStartCluster-(2f323e54_DEFAULT)] Start to refresh stale message cache 2024/06/24 08:52:27.555 WARN [ZkBaseDataAccessor] [HelixController-pipeline-task-QuickStartCluster-(2f323e54_TASK)] Fail to read record for paths: {/QuickStartCluster/INSTANCES/Server_10.220.207.212_7050/MESSAGES/8a579eec-c767-4459-9aa6-2edf5b9e45f9=-101} 2024/06/24 08:52:27.555 INFO [InstanceMessagesCache] [HelixController-pipeline-task-QuickStartCluster-(2f323e54_TASK)] END: InstanceMessagesCache.refresh(), 1 of Messages read from ZooKeeper. took 2 ms. 2024/06/24 08:52:27.555 INFO [InstanceMessagesCache] [HelixController-pipeline-task-QuickStartCluster-(2f323e54_TASK)] Start to refresh stale message cache 2024/06/24 08:52:27.555 INFO [HelixTask] [HelixTaskExecutor-message_handle_thread_233] Delete message 8a579eec-c767-4459-9aa6-2edf5b9e45f9 from zk! 2024/06/24 08:52:27.555 INFO [HelixTaskExecutor] [HelixTaskExecutor-message_handle_thread_233] message finished: 8a579eec-c767-4459-9aa6-2edf5b9e45f9, took 3 2024/06/24 08:52:27.555 INFO [HelixTask] [HelixTaskExecutor-message_handle_thread_233] Message: 8a579eec-c767-4459-9aa6-2edf5b9e45f9 (parent: null) handling task for exceptionTrace_REALTIME:exceptionTrace00__20240624T0630Z completed at: 1719219147555, results: true. FrameworkTime: 2 ms; HandlerTime: 1 ms. 2024/06/24 08:52:27.555 INFO [CallbackHandler] [ZkClient-EventThread-355-localhost:2123] 355 START: CallbackHandler 29, INVOKE /QuickStartCluster/INSTANCES/Server_10.220.207.212_7050/MESSAGES listener: org.apache.helix.messaging.handling.HelixTaskExecutor@5fd6d31f type: CALLBACK 2024/06/24 08:52:27.555 INFO [CallbackHandler] [ZkClient-EventThread-355-localhost:2123] CallbackHandler 29 subscribing changes listener to path: /QuickStartCluster/INSTANCES/Server_10.220.207.212_7050/MESSAGES, callback type: CALLBACK, event types: [NodeChildrenChanged], listener: org.apache.helix.messaging.handling.HelixTaskExecutor@5fd6d31f, watchChild: false 2024/06/24 08:52:27.555 INFO [CallbackHandler] [ZkClient-EventThread-188-localhost:2123] 188 START: CallbackHandler 27, INVOKE /QuickStartCluster/INSTANCES/Server_10.220.207.212_7050/MESSAGES listener: org.apache.helix.controller.GenericHelixController@3b11deb6 type: CALLBACK 2024/06/24 08:52:27.555 INFO [CallbackHandler] [ZkClient-EventThread-188-localhost:2123] CallbackHandler 27 subscribing changes listener to path: /QuickStartCluster/INSTANCES/Server_10.220.207.212_7050/MESSAGES, callback type: CALLBACK, event types: [NodeChildrenChanged], listener: org.apache.helix.controller.GenericHelixController@3b11deb6, watchChild: false 2024/06/24 08:52:27.555 INFO [CallbackHandler] [ZkClient-EventThread-355-localhost:2123] CallbackHandler29, Subscribing to path: /QuickStartCluster/INSTANCES/Server_10.220.207.212_7050/MESSAGES took: 0 2024/06/24 08:52:27.556 INFO [CallbackHandler] [ZkClient-EventThread-188-localhost:2123] CallbackHandler27, Subscribing to path: /QuickStartCluster/INSTANCES/Server_10.220.207.212_7050/MESSAGES took: 1 2024/06/24 08:52:27.556 INFO [GenericHelixController] [ZkClient-EventThread-188-localhost:2123] START: GenericClusterController.onMessage() for cluster QuickStartCluster 2024/06/24 08:52:27.556 INFO [HelixTaskExecutor] [ZkClient-EventThread-355-localhost:2123] No Messages to process 2024/06/24 08:52:27.556 INFO [CallbackHandler] [ZkClient-EventThread-355-localhost:2123] 355 END:INVOKE CallbackHandler 29, /QuickStartCluster/INSTANCES/Server_10.220.207.212_7050/MESSAGES listener: org.apache.helix.messaging.handling.HelixTaskExecutor@5fd6d31f type: CALLBACK Took: 1ms 2024/06/24 08:52:27.556 INFO [GenericHelixController] [ZkClient-EventThread-188-localhost:2123] END: GenericClusterController.onMessage() for cluster QuickStartCluster 2024/06/24 08:52:27.556 INFO [CallbackHandler] [ZkClient-EventThread-188-localhost:2123] 188 END:INVOKE CallbackHandler 27, /QuickStartCluster/INSTANCES/Server_10.220.207.212_7050/MESSAGES listener: org.apache.helix.controller.GenericHelixController@3b11deb6 type: CALLBACK Took: 1ms 2024/06/24 08:52:27.557 INFO [AbstractDataCache] [HelixController-pipeline-default-QuickStartCluster-(2f323e54_DEFAULT)] Event QuickStartCluster::DEFAULT::2f323e54_DEFAULT : 0 properties refreshed from ZK. 2024/06/24 08:52:27.557 INFO [ParticipantStateCache] [HelixController-pipeline-default-QuickStartCluster-(2f323e54_DEFAULT)] Event QuickStartCluster::DEFAULT::2f323e54_DEFAULT : END: participantStateCache.refresh() for cluster QuickStartCluster, started at : 1719219147554, took 3 ms 2024/06/24 08:52:27.557 INFO [InstanceMessagesCache] [HelixController-pipeline-default-QuickStartCluster-(2f323e54_DEFAULT)] END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms. 2024/06/24 08:52:27.557 INFO [ResourceControllerDataProvider] [HelixController-pipeline-default-QuickStartCluster-(2f323e54_DEFAULT)] Event 2f323e54_DEFAULT : No customized state config change for QuickStartCluster cluster, DEFAULT pipeline 2024/06/24 08:52:27.557 INFO [AbstractDataCache] [HelixController-pipeline-default-QuickStartCluster-(2f323e54_DEFAULT)] Event QuickStartCluster::DEFAULT::2f323e54_DEFAULT : 0 properties refreshed from ZK. 2024/06/24 08:52:27.557 INFO [ParticipantStateCache] [HelixController-pipeline-default-QuickStartCluster-(2f323e54_DEFAULT)] Event QuickStartCluster::DEFAULT::2f323e54_DEFAULT : END: participantStateCache.refresh() for cluster QuickStartCluster, started at : 1719219147557, took 0 ms 2024/06/24 08:52:27.557 INFO [ResourceControllerDataProvider] [HelixController-pipeline-default-QuickStartCluster-(2f323e54_DEFAULT)] Event 2f323e54_DEFAULT : END: ResourceControllerDataProvider.refresh() for cluster QuickStartCluster, started at 1719219147549 took 8 for DEFAULT pipeline 2024/06/24 08:52:27.557 INFO [Pipeline] [HelixController-pipeline-default-QuickStartCluster-(2f323e54_DEFAULT)] END ReadClusterDataStage for DEFAULT pipeline for cluster QuickStartCluster. took: 8 ms for event 2f323e54_DEFAULT 2024/06/24 08:52:27.557 INFO [Pipeline] [HelixController-pipeline-default-QuickStartCluster-(2f323e54_DEFAULT)] END ResourceComputationStage for DEFAULT pipeline for cluster QuickStartCluster. took: 0 ms for event 2f323e54_DEFAULT 2024/06/24 08:52:27.557 INFO [Pipeline] [HelixController-pipeline-default-QuickStartCluster-(2f323e54_DEFAULT)] END ResourceValidationStage for DEFAULT pipeline for cluster QuickStartCluster. took: 0 ms for event 2f323e54_DEFAULT 2024/06/24 08:52:27.557 INFO [Pipeline] [HelixController-pipeline-default-QuickStartCluster-(2f323e54_DEFAULT)] END CurrentStateComputationStage for DEFAULT pipeline for cluster QuickStartCluster. took: 0 ms for event 2f323e54_DEFAULT 2024/06/24 08:52:27.557 INFO [Pipeline] [HelixController-pipeline-default-QuickStartCluster-(2f323e54_DEFAULT)] END CustomizedStateComputationStage for DEFAULT pipeline for cluster QuickStartCluster. took: 0 ms for event 2f323e54_DEFAULT 2024/06/24 08:52:27.557 INFO [Pipeline] [HelixController-pipeline-default-QuickStartCluster-(2f323e54_DEFAULT)] END TopStateHandoffReportStage for DEFAULT pipeline for cluster QuickStartCluster. took: 0 ms for event 2f323e54_DEFAULT 2024/06/24 08:52:27.557 INFO [WagedRebalancer] [HelixController-pipeline-default-QuickStartCluster-(2f323e54_DEFAULT)] Start computing new ideal states for resources: [] 2024/06/24 08:52:27.558 INFO [WagedRebalancer] [HelixController-pipeline-default-QuickStartCluster-(2f323e54_DEFAULT)] Start calculating the new best possible assignment. 2024/06/24 08:52:27.558 INFO [WagedRebalancer] [HelixController-pipeline-default-QuickStartCluster-(2f323e54_DEFAULT)] Start calculating for an assignment with algorithm ConstraintBasedAlgorithm 2024/06/24 08:52:27.558 INFO [WagedRebalancer] [HelixController-pipeline-default-QuickStartCluster-(2f323e54_DEFAULT)] Finish calculating an assignment with algorithm ConstraintBasedAlgorithm. Took: 0 ms. 2024/06/24 08:52:27.558 INFO [WagedRebalancer] [HelixController-pipeline-default-QuickStartCluster-(2f323e54_DEFAULT)] Finish calculating the new best possible assignment. 2024/06/24 08:52:27.558 INFO [WagedRebalancer] [HelixController-pipeline-default-QuickStartCluster-(2f323e54_DEFAULT)] Finish computing new ideal states for resources: [] 2024/06/24 08:52:27.558 INFO [Pipeline] [HelixController-pipeline-default-QuickStartCluster-(2f323e54_DEFAULT)] END BestPossibleStateCalcStage for DEFAULT pipeline for cluster QuickStartCluster. took: 1 ms for event 2f323e54_DEFAULT 2024/06/24 08:52:27.558 INFO [Pipeline] [HelixController-pipeline-default-QuickStartCluster-(2f323e54_DEFAULT)] END MessageGenerationPhase for DEFAULT pipeline for cluster QuickStartCluster. took: 0 ms for event 2f323e54_DEFAULT 2024/06/24 08:52:27.558 INFO [Pipeline] [HelixController-pipeline-default-QuickStartCluster-(2f323e54_DEFAULT)] END MessageSelectionStage for DEFAULT pipeline for cluster QuickStartCluster. took: 0 ms for event 2f323e54_DEFAULT 2024/06/24 08:52:27.558 INFO [StateTransitionThrottleController] [HelixController-pipeline-default-QuickStartCluster-(2f323e54_DEFAULT)] No throttle config is set! 2024/06/24 08:52:27.558 INFO [Pipeline] [HelixController-pipeline-default-QuickStartCluster-(2f323e54_DEFAULT)] END IntermediateStateCalcStage for DEFAULT pipeline for cluster QuickStartCluster. took: 0 ms for event 2f323e54_DEFAULT 2024/06/24 08:52:27.558 INFO [Pipeline] [HelixController-pipeline-default-QuickStartCluster-(2f323e54_DEFAULT)] END MessageThrottleStage for DEFAULT pipeline for cluster QuickStartCluster. took: 0 ms for event 2f323e54_DEFAULT 2024/06/24 08:52:27.558 INFO [AbstractAsyncBaseStage] [MaintenanceRecoveryWorker-QuickStartCluster] START AsyncProcess: DEFAULT::MaintenanceRecoveryStage 2024/06/24 08:52:27.558 INFO [AbstractAsyncBaseStage] [MaintenanceRecoveryWorker-QuickStartCluster] END AsyncProcess: DEFAULT::MaintenanceRecoveryStage, took 0 ms 2024/06/24 08:52:27.558 INFO [AbstractAsyncBaseStage] [HelixController-pipeline-default-QuickStartCluster-(2f323e54_DEFAULT)] Submitted asynchronous DEFAULT::MaintenanceRecoveryStage task to worker 2024/06/24 08:52:27.558 INFO [Pipeline] [HelixController-pipeline-default-QuickStartCluster-(2f323e54_DEFAULT)] END MaintenanceRecoveryStage for DEFAULT pipeline for cluster QuickStartCluster. took: 0 ms for event 2f323e54_DEFAULT 2024/06/24 08:52:27.558 INFO [Pipeline] [HelixController-pipeline-default-QuickStartCluster-(2f323e54_DEFAULT)] END ResourceMessageDispatchStage for DEFAULT pipeline for cluster QuickStartCluster. took: 0 ms for event 2f323e54_DEFAULT 2024/06/24 08:52:27.558 INFO [AbstractAsyncBaseStage] [HelixController-pipeline-default-QuickStartCluster-(2f323e54_DEFAULT)] Submitted asynchronous DEFAULT::PersistAssignmentStage task to worker 2024/06/24 08:52:27.558 INFO [Pipeline] [HelixController-pipeline-default-QuickStartCluster-(2f323e54_DEFAULT)] END PersistAssignmentStage for DEFAULT pipeline for cluster QuickStartCluster. took: 0 ms for event 2f323e54_DEFAULT 2024/06/24 08:52:27.558 INFO [AbstractAsyncBaseStage] [PersistAssignmentWorker-QuickStartCluster] START AsyncProcess: DEFAULT::PersistAssignmentStage 2024/06/24 08:52:27.559 INFO [AbstractAsyncBaseStage] [TargetExternalViewCalcWorker-QuickStartCluster] START AsyncProcess: DEFAULT::TargetExteralViewCalcStage 2024/06/24 08:52:27.559 INFO [AbstractAsyncBaseStage] [PersistAssignmentWorker-QuickStartCluster] END AsyncProcess: DEFAULT::PersistAssignmentStage, took 1 ms 2024/06/24 08:52:27.559 INFO [AbstractAsyncBaseStage] [TargetExternalViewCalcWorker-QuickStartCluster] END AsyncProcess: DEFAULT::TargetExteralViewCalcStage, took 0 ms 2024/06/24 08:52:27.559 INFO [AbstractAsyncBaseStage] [HelixController-pipeline-default-QuickStartCluster-(2f323e54_DEFAULT)] Submitted asynchronous DEFAULT::TargetExteralViewCalcStage task to worker 2024/06/24 08:52:27.559 INFO [Pipeline] [HelixController-pipeline-default-QuickStartCluster-(2f323e54_DEFAULT)] END TargetExteralViewCalcStage for DEFAULT pipeline for cluster QuickStartCluster. took: 1 ms for event 2f323e54_DEFAULT 2024/06/24 08:52:27.559 INFO [GenericHelixController] [HelixController-pipeline-default-QuickStartCluster-(2f323e54_DEFAULT)] END: Invoking DEFAULT controller pipeline for event MessageChange::2f323e54_DEFAULT for cluster QuickStartCluster, took 10 ms 2024/06/24 08:52:27.559 INFO [GenericHelixController] [HelixController-pipeline-default-QuickStartCluster-(2f323e54_DEFAULT)] Callback time for event: MessageChange took: 1 ms InQueue time for event: MessageChange took: 1 ms TotalProcessed time for event: MessageChange took: 10 ms 2024/06/24 08:52:27.559 INFO [GenericHelixController] [HelixController-pipeline-default-QuickStartCluster-(0ba2c2af_DEFAULT)] Controller stopping periodical rebalance timer at period 9223372036854775807 2024/06/24 08:52:27.559 INFO [GenericHelixController] [HelixController-pipeline-default-QuickStartCluster-(0ba2c2af_DEFAULT)] START: Invoking DEFAULT controller pipeline for cluster: QuickStartCluster. Event type: MessageChange, ID: 0ba2c2af_DEFAULT. Event session ID: 10004f0fb490005 2024/06/24 08:52:27.559 INFO [BaseControllerDataProvider] [HelixController-pipeline-default-QuickStartCluster-(0ba2c2af_DEFAULT)] Event 0ba2c2af_DEFAULT : No ideal state change for QuickStartCluster cluster, DEFAULT pipeline 2024/06/24 08:52:27.559 INFO [BaseControllerDataProvider] [HelixController-pipeline-default-QuickStartCluster-(0ba2c2af_DEFAULT)] Event 0ba2c2af_DEFAULT : No live instance change for QuickStartCluster cluster, DEFAULT pipeline 2024/06/24 08:52:27.559 INFO [BaseControllerDataProvider] [HelixController-pipeline-default-QuickStartCluster-(0ba2c2af_DEFAULT)] Event 0ba2c2af_DEFAULT : No instance config change for QuickStartCluster cluster, DEFAULT pipeline 2024/06/24 08:52:27.559 INFO [BaseControllerDataProvider] [HelixController-pipeline-default-QuickStartCluster-(0ba2c2af_DEFAULT)] Event 0ba2c2af_DEFAULT : No resource config change for QuickStartCluster cluster, DEFAULT pipeline 2024/06/24 08:52:27.560 INFO [AbstractDataCache] [HelixController-pipeline-task-QuickStartCluster-(2f323e54_TASK)] Event QuickStartCluster::TASK::2f323e54_TASK : 0 properties refreshed from ZK. 2024/06/24 08:52:27.560 INFO [ParticipantStateCache] [HelixController-pipeline-task-QuickStartCluster-(2f323e54_TASK)] Event QuickStartCluster::TASK::2f323e54_TASK : END: participantStateCache.refresh() for cluster QuickStartCluster, started at : 1719219147555, took 5 ms 2024/06/24 08:52:27.560 INFO [InstanceMessagesCache] [HelixController-pipeline-task-QuickStartCluster-(2f323e54_TASK)] END: updateRelayMessages(), 0 of valid relay messages in cache, took 0 ms. 2024/06/24 08:52:27.560 INFO [PropertyCache] [HelixController-pipeline-default-QuickStartCluster-(0ba2c2af_DEFAULT)] Event QuickStartCluster::DEFAULT::0ba2c2af_DEFAULT : Refreshed 9 property StateModelDefinition took 1 ms. Selective: false 2024/06/24 08:52:27.561 INFO [ConstraintItemBuilder] [HelixController-pipeline-default-QuickStartCluster-(0ba2c2af_DEFAULT)] overwrite existing constraint-value. old-value: null, new-value: 100000 2024/06/24 08:52:27.561 INFO [PropertyCache] [HelixController-pipeline-default-QuickStartCluster-(0ba2c2af_DEFAULT)] Event QuickStartCluster::DEFAULT::0ba2c2af_DEFAULT : Refreshed 1 property ClusterConstraint took 1 ms. Selective: false 2024/06/24 08:52:27.561 INFO [AbstractDataCache] [HelixController-pipeline-task-QuickStartCluster-(2f323e54_TASK)] Event QuickStartCluster::TASK::2f323e54_TASK : 0 properties refreshed from ZK. 2024/06/24 08:52:27.561 INFO [ParticipantStateCache] [HelixController-pipeline-task-QuickStartCluster-(2f323e54_TASK)] Event QuickStartCluster::TASK::2f323e54_TASK : END: participantStateCache.refresh() for cluster QuickStartCluster, started at : 1719219147560, took 1 ms 2024/06/24 08:52:27.562 INFO [InstanceMessagesCache] [HelixController-pipeline-default-QuickStartCluster-(0ba2c2af_DEFAULT)] START: InstanceMessagesCache.refresh() 2024/06/24 08:52:27.572 INFO [WorkflowControllerDataProvider] [HelixController-pipeline-task-QuickStartCluster-(2f323e54_TASK)] Event 2f323e54_TASK : END: WorkflowControllerDataProvider.refresh() for cluster QuickStartCluster, started at 1719219147549 took 23 for TASK pipeline 2024/06/24 08:52:27.572 INFO [Pipeline] [HelixController-pipeline-task-QuickStartCluster-(2f323e54_TASK)] END ReadClusterDataStage for TASK pipeline for cluster QuickStartCluster. took: 23 ms for event 2f323e54_TASK 2024/06/24 08:52:27.572 INFO [Pipeline] [HelixController-pipeline-task-QuickStartCluster-(2f323e54_TASK)] END ResourceComputationStage for TASK pipeline for cluster QuickStartCluster. took: 0 ms for event 2f323e54_TASK 2024/06/24 08:52:27.572 INFO [Pipeline] [HelixController-pipeline-task-QuickStartCluster-(2f323e54_TASK)] END ResourceValidationStage for TASK pipeline for cluster QuickStartCluster. took: 0 ms for event 2f323e54_TASK 2024/06/24 08:52:27.573 INFO [Pipeline] [HelixController-pipeline-task-QuickStartCluster-(2f323e54_TASK)] END CurrentStateComputationStage for TASK pipeline for cluster QuickStartCluster. took: 1 ms for event 2f323e54_TASK 2024/06/24 08:52:27.573 INFO [AssignableInstanceManager] [HelixController-pipeline-task-QuickStartCluster-(2f323e54_TASK)] AssignableInstanceManager built AssignableInstances from scratch based on CurrentState. 2024/06/24 08:52:27.573 INFO [AssignableInstanceManager] [HelixController-pipeline-task-QuickStartCluster-(2f323e54_TASK)] Current quota capacity: {"Broker_10.220.207.212_8000":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}},"Server_10.220.207.212_7050":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}},"Minion_10.220.207.212_6000":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}},"Controller_10.220.207.212_9000":{"TASK_EXEC_THREAD":{"DEFAULT":"0/40"}}} 2024/06/24 08:52:27.573 INFO [ControllerResponseFilter] [grizzly-http-server-1] Handled request from 10.220.132.137 GET http://-/tables/exceptionTrace_REALTIME/externalview, content-type null status code 200 OK 2024/06/24 08:52:27.574 INFO [Pipeline] [HelixController-pipeline-task-QuickStartCluster-(2f323e54_TASK)] END TaskSchedulingStage for TASK pipeline for cluster QuickStartCluster. took: 1 ms for event 2f323e54_TASK 2024/06/24 08:52:27.574 INFO [TaskPersistDataStage] [HelixController-pipeline-task-QuickStartCluster-(2f323e54_TASK)] START TaskPersistDataStage.process() 2024/06/24 08:52:27.574 INFO [TaskPersistDataStage] [HelixController-pipeline-task-QuickStartCluster-(2f323e54_TASK)] END TaskPersistDataStage.process() for cluster QuickStartCluster took 0 ms 2024/06/24 08:52:27.574 INFO [Pipeline] [HelixController-pipeline-task-QuickStartCluster-(2f323e54_TASK)] END TaskPersistDataStage for TASK pipeline for cluster QuickStartCluster. took: 0 ms for event 2f323e54_TASK 2024/06/24 08:52:27.574 INFO [AbstractAsyncBaseStage] [TaskJobPurgeWorker-QuickStartCluster] START AsyncProcess: TASK::TaskGarbageCollectionStage 2024/06/24 08:52:27.574 INFO [AbstractAsyncBaseStage] [TaskJobPurgeWorker-QuickStartCluster] END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms 2024/06/24 08:52:27.574 INFO [AbstractAsyncBaseStage] [HelixController-pipeline-task-QuickStartCluster-(2f323e54_TASK)] Submitted asynchronous TASK::TaskGarbageCollectionStage task to worker 2024/06/24 08:52:27.574 INFO [Pipeline] [HelixController-pipeline-task-QuickStartCluster-(2f323e54_TASK)] END TaskGarbageCollectionStage for TASK pipeline for cluster QuickStartCluster. took: 0 ms for event 2f323e54_TASK 2024/06/24 08:52:27.574 INFO [InstanceMessagesCache] [HelixController-pipeline-default-QuickStartCluster-(0ba2c2af_DEFAULT)] END: InstanceMessagesCache.refresh(), 0 of Messages read from ZooKeeper. took 12 ms. 2024/06/24 08:52:27.574 INFO [InstanceMessagesCache] [HelixController-pipeline-default-QuickStartCluster-(0ba2c2af_DEFAULT)] Start to refresh stale message cache 2024/06/24 08:52:27.574 INFO [Pipeline] [HelixController-pipeline-task-QuickStartCluster-(2f323e54_TASK)] END MessageGenerationPhase for TASK pipeline for cluster QuickStartCluster. took: 0 ms for event 2f323e54_TASK 2024/06/24 08:52:27.574 INFO [Pipeline] [HelixController-pipeline-task-QuickStartCluster-(2f323e54_TASK)] END TaskMessageDispatchStage for TASK pipeline for cluster QuickStartCluster. took: 0 ms for event 2f323e54_TASK 2024/06/24 08:52:27.574 INFO [GenericHelixController] [HelixController-pipeline-task-QuickStartCluster-(2f323e54_TASK)] END: Invoking TASK controller pipeline for event MessageChange::2f323e54_TASK for cluster QuickStartCluster, took 25 ms 2024/06/24 08:52:27.574 INFO [GenericHelixController] [HelixController-pipeline-task-QuickStartCluster-(0ba2c2af_TASK)] Controller stopping periodical rebalance timer at period 9223372036854775807 2024/06/24 08:52:27.574 INFO [GenericHelixController] [HelixController-pipeline-task-QuickStartCluster-(0ba2c2af_TASK)] START: Invoking TASK controller pipeline for cluster: QuickStartCluster. Event type: MessageChange, ID: 0ba2c2af_TASK. Event session ID: 10004f0fb490005 2024/06/24 08:52:27.574 INFO [BaseControllerDataProvider] [HelixController-pipeline-task-QuickStartCluster-(0ba2c2af_TASK)] Event 0ba2c2af_TASK : No ideal state change for QuickStartCluster cluster, TASK pipeline 2024/06/24 08:52:27.574 INFO [BaseControllerDataProvider] [HelixController-pipeline-task-QuickStartCluster-(0ba2c2af_TASK)] Event 0ba2c2af_TASK : No live instance change for QuickStartCluster cluster, TASK pipeline 2024/06/24 08:52:27.574 INFO [BaseControllerDataProvider] [HelixController-pipeline-task-QuickStartCluster-(0ba2c2af_TASK)] Event 0ba2c2af_TASK : No instance config change for QuickStartCluster cluster, TASK pipeline 2024/06/24 08:52:27.574 INFO [BaseControllerDataProvider] [HelixController-pipeline-task-QuickStartCluster-(0ba2c2af_TASK)] Event 0ba2c2af_TASK : No resource config change for QuickStartCluster cluster, TASK pipeline 2024/06/24 08:52:27.575 ERROR [CompletionServiceHelper] [grizzly-http-server-3] Server: Server_10.220.207.212_7050 returned error: 404 2024/06/24 08:52:27.575 WARN [CompletionServiceHelper] [grizzly-http-server-3] Finished reading information for table: exceptionTrace_REALTIME with 0/1 server responses 2024/06/24 08:52:27.575 INFO [ControllerResponseFilter] [grizzly-http-server-3] Handled request from 10.220.132.137 GET http://-/segments/exceptionTrace_REALTIME/exceptionTrace__0__0__20240624T0630Z/metadata?columns=*, content-type null status code 200 OK 2024/06/24 08:52:27.576 INFO [DebugResource] [grizzly-http-server-2] Reading segments debug info from servers: [Server_10.220.207.212_7050] for table: exceptionTrace_REALTIME 2024/06/24 08:52:27.577 INFO [CompletionServiceHelper] [grizzly-http-server-2] Finished reading information for table: exceptionTrace_REALTIME

pan889 commented 4 months ago

While testing, I checked the log below. Would this help?

2024/06/24 09:36:40.188 INFO [HelixInstanceDataManager] [HelixTaskExecutor-message_handle_thread_71] Adding segment: exceptionTraceV10020240624T0936Z to table: exceptionTraceV1_REALTIME 2024/06/24 09:36:40.190 INFO [BaseTableDataManager] [HelixTaskExecutor-message_handle_thread_71] Default time: 0 does not comply with format: 1:MILLISECONDS:EPOCH, using creation time: 1719221800160 as the default time for table: exceptionTraceV1_REALTIME 2024/06/24 09:36:40.190 INFO [LLRealtimeSegmentDataManager_exceptionTraceV10020240624T0936Z] [HelixTaskExecutor-message_handle_thread_71] Setting sorted column name: applicationName from RealtimeDataResourceZKMetadata for segment exceptionTraceV100__20240624T0936Z 2024/06/24 09:36:40.190 ERROR [HelixStateTransitionHandler] [HelixTaskExecutor-message_handle_thread_71] Exception while executing a state transition task exceptionTraceV100__20240624T0936Z java.lang.reflect.InvocationTargetException: null at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?] at org.apache.helix.messaging.handling.HelixStateTransitionHandler.invoke(HelixStateTransitionHandler.java:350) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79] at org.apache.helix.messaging.handling.HelixStateTransitionHandler.handleMessage(HelixStateTransitionHandler.java:278) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79] at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:97) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79] at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:49) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79] at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?] at java.lang.Thread.run(Thread.java:840) [?:?] Caused by: java.lang.UnsatisfiedLinkError: 'void com.yscope.clp.compressorfrontend.MessageEncoder.setVariableHandlingRuleVersions(byte[], int, byte[], int)' at com.yscope.clp.compressorfrontend.MessageEncoder.setVariableHandlingRuleVersions(Native Method) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79] at com.yscope.clp.compressorfrontend.MessageEncoder.(MessageEncoder.java:36) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79] at org.apache.pinot.plugin.inputformat.clplog.CLPLogRecordExtractor.init(CLPLogRecordExtractor.java:91) ~[pinot-clp-log-1.0.0-shaded.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79] at org.apache.pinot.plugin.inputformat.clplog.CLPLogMessageDecoder.init(CLPLogMessageDecoder.java:61) ~[pinot-clp-log-1.0.0-shaded.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79] at org.apache.pinot.spi.stream.StreamDecoderProvider.create(StreamDecoderProvider.java:45) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79] at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.(LLRealtimeSegmentDataManager.java:1433) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79] at org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager.addSegment(RealtimeTableDataManager.java:444) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79] at org.apache.pinot.server.starter.helix.HelixInstanceDataManager.addRealtimeSegment(HelixInstanceDataManager.java:230) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79] at org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeConsumingFromOffline(SegmentOnlineOfflineStateModelFactory.java:80) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79] ... 12 more 2024/06/24 09:36:40.191 ERROR [StateModel] [HelixTaskExecutor-message_handle_thread_71] Default rollback method invoked on error. Error Code: ERROR 2024/06/24 09:36:40.192 ERROR [HelixTask] [HelixTaskExecutor-message_handle_thread_71] Message execution failed. msgId: 26fa39a8-716d-4668-a2b9-c1fedc702192, errorMsg: java.lang.reflect.InvocationTargetException

donghun-cho commented 4 months ago

exceptionTrace table uses CLP encoding. I am not sure CLP library support Graviton(ARM64).

reference: Pinot doesn't provide official binaries for M1 or M2 Mac systems. https://docs.pinot.apache.org/basics/getting-started/running-pinot-locally#m1-and-m2-mac-support

pan889 commented 4 months ago

Thanks for your help. As you said, I changed it to x86 and applied it, and it worked fine.

Perhaps CLP does not support arm64 architecture.

intr3p1d commented 4 months ago

I'm glad to hear that the issue has been resolved. Here are the details related to the issue you raised:

Apache Pinot utilizes the clp-ffi-java library internally for its CLP-related functionalities. This library currently supports only the linux/amd64 OS X/x86-64 architecture, which is the root cause of the problem you are experiencing.

According to the issue linked below, it appears that there are plans to support binaries for multiple platforms, including aarch64, in the near future: https://github.com/y-scope/clp-ffi-java/issues/46

intr3p1d commented 3 months ago

It seems that Apache Pinot is also planning to support various platform, including aarch64.

https://github.com/apache/pinot/pull/13648