apache / hertzbeat

Apache HertzBeat(incubating) is a real-time monitoring system with agentless, performance cluster, prometheus-compatible, custom monitoring and status page building capabilities.
https://hertzbeat.apache.org/
Apache License 2.0
5.72k stars 990 forks source link

[BUG] <title>Oracle monitoring time stopped #710

Closed zqr10159 closed 1 year ago

zqr10159 commented 1 year ago

Is there an existing issue for this?

Current Behavior

在2023-03-09 19:48:52秒后,oracle监控时间停止,其他监控如mysql,centos正常,停止后状态为正常,未发送告警。重新启用监控后恢复正常。

Expected Behavior

No response

Steps To Reproduce

No response

Environment

HertzBeat version(1.25):

Debug logs

2023-03-09 19:48:24 [228887525460224-windows-disk-4527] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Failed, Run 2ms, All 2ms] Reason: SNMP error No such name on index 1 2023-03-09 19:48:24 [228887525460224-windows-network-4529] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 1ms, All 1ms]. 2023-03-09 19:48:25 [228776751884544-mysql-basic-5505] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 4ms, All 5ms]. 2023-03-09 19:48:25 [228776751884544-mysql-innodb-5510] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 3ms, All 3ms]. 2023-03-09 19:48:25 [228776751884544-mysql-cache-5510] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 7ms, All 8ms]. 2023-03-09 19:48:25 [228776751884544-mysql-performance-5510] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 9ms, All 10ms]. 2023-03-09 19:48:25 [228776751884544-mysql-status-5510] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 12ms, All 12ms]. 2023-03-09 19:48:25 [228776751884544-mysql-table_lock-5522] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 4ms, All 4ms]. 2023-03-09 19:48:25 [228776751884544-mysql-process_state-5523] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 5ms, All 6ms]. 2023-03-09 19:48:25 [228776751884544-mysql-handler-5522] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 8ms, All 8ms]. 2023-03-09 19:48:25 [228776751884544-mysql-slow_sql-5522] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 9ms, All 9ms]. 2023-03-09 19:48:25 [warehouse-persistent-data-storage] INFO com.usthe.warehouse.store.HistoryJpaDatabaseDataStorage - [warehouse jpa] flush metrics data 228776751884544 is null, ignore. 2023-03-09 19:48:25 [228776751884544-mysql-tmp-5522] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 11ms, All 11ms]. 2023-03-09 19:48:25 [228776751884544-mysql-thread-5522] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 13ms, All 13ms]. 2023-03-09 19:48:25 [228776751884544-mysql-sort-5522] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 15ms, All 15ms]. 2023-03-09 19:48:25 [228776751884544-mysql-select_type-5522] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 17ms, All 17ms]. 2023-03-09 19:48:25 [228776751884544-mysql-connection-5522] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 20ms, All 20ms]. 2023-03-09 19:48:52 [228776362760448-oracle-basic-2505] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 0ms, All 1ms]. 2023-03-09 19:48:52 [228776362760448-oracle-process-2507] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 1ms, All 1ms]. 2023-03-09 19:48:52 [228776362760448-oracle-basic-2505] ERROR com.usthe.collector.dispatch.WorkerPool - workerExecutor has uncaughtException. 2023-03-09 19:48:52 [228776362760448-oracle-basic-2505] ERROR com.usthe.collector.dispatch.WorkerPool - null java.util.ConcurrentModificationException: null at java.base/java.util.HashMap$HashIterator.nextNode(Unknown Source) at java.base/java.util.HashMap$KeyIterator.next(Unknown Source) at java.base/java.lang.Iterable.forEach(Unknown Source) at com.usthe.collector.dispatch.CommonDispatcher.dispatchCollectData(CommonDispatcher.java:245) at com.usthe.collector.dispatch.MetricsCollect.run(MetricsCollect.java:166) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) 2023-03-09 19:48:52 [228776362760448-oracle-cache_hit_ratio-2507] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 65ms, All 66ms]. 2023-03-09 19:48:52 [228776362760448-oracle-background_sessions-2509] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 65ms, All 68ms]. 2023-03-09 19:48:52 [228776362760448-oracle-total_sessions-2509] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 67ms, All 70ms]. 2023-03-09 19:48:53 [228776362760448-oracle-wait-2509] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 519ms, All 519ms]. 2023-03-09 19:49:16 [228886747273472-centos-basic-6504] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 25ms, All 25ms]. 2023-03-09 19:49:17 [228886747273472-centos-cpu-6529] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 1074ms, All 1074ms]. 2023-03-09 19:49:17 [228886747273472-centos-memory-7603] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 58ms, All 58ms]. 2023-03-09 19:49:17 [228886747273472-centos-disk-7662] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 68ms, All 68ms]. 2023-03-09 19:49:17 [228886747273472-centos-interface-7731] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 55ms, All 56ms]. 2023-03-09 19:49:17 [228886747273472-centos-disk_free-7787] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 56ms, All 57ms]. 2023-03-09 19:49:17 [228886747273472-centos-top_cpu_process-7844] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 73ms, All 73ms]. 2023-03-09 19:49:17 [228886747273472-centos-top_mem_process-7919] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 67ms, All 68ms]. 2023-03-09 19:49:36 [228779520764160-ftp-basic-6505] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 3ms, All 3ms]. 2023-03-09 19:50:00 [228776645368064-oracle-basic-0505] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 1ms, All 1ms]. 2023-03-09 19:50:00 [228776645368064-oracle-connection-0507] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 4ms, All 4ms]. 2023-03-09 19:50:00 [228776645368064-oracle-cache_hit_ratio-0507] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 63ms, All 63ms]. 2023-03-09 19:50:00 [228776645368064-oracle-background_sessions-0507] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 65ms, All 65ms]. 2023-03-09 19:50:00 [228776645368064-oracle-active_sessions-0507] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 67ms, All 67ms]. 2023-03-09 19:50:06 [228885960734976-mysql-basic-6505] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 2ms, All 2ms]. 2023-03-09 19:50:06 [228885960734976-mysql-performance-6509] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 1ms, All 1ms]. 2023-03-09 19:50:06 [228885960734976-mysql-innodb-6509] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 2ms, All 2ms]. 2023-03-09 19:50:06 [228885960734976-mysql-status-6509] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 3ms, All 3ms]. 2023-03-09 19:50:06 [228885960734976-mysql-cache-6509] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 3ms, All 3ms]. 2023-03-09 19:50:06 [228885960734976-mysql-handler-6513] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 0ms, All 0ms]. 2023-03-09 19:50:06 [228885960734976-mysql-select_type-6513] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 1ms, All 1ms]. 2023-03-09 19:50:06 [228885960734976-mysql-slow_sql-6513] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 1ms, All 1ms]. 2023-03-09 19:50:06 [warehouse-persistent-data-storage] INFO com.usthe.warehouse.store.HistoryJpaDatabaseDataStorage - [warehouse jpa] flush metrics data 228885960734976 is null, ignore. 2023-03-09 19:50:06 [228885960734976-mysql-sort-6513] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 2ms, All 2ms]. 2023-03-09 19:50:06 [228885960734976-mysql-thread-6513] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 3ms, All 3ms]. 2023-03-09 19:50:06 [228885960734976-mysql-table_lock-6513] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 3ms, All 3ms]. 2023-03-09 19:50:06 [228885960734976-mysql-tmp-6513] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 4ms, All 4ms]. 2023-03-09 19:50:06 [228885960734976-mysql-process_state-6513] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 5ms, All 5ms]. 2023-03-09 19:50:06 [228885960734976-mysql-connection-6513] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 7ms, All 7ms]. 2023-03-09 19:50:06 [228776645368064-oracle-percentage-0507] WARN com.usthe.collector.collect.database.JdbcCommonCollect - Jdbc sql error: ORA-01013: 用户请求取消当前的操作 , code: 1013. 2023-03-09 19:50:06 [228776645368064-oracle-performance-0507] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 6183ms, All 6183ms]. 2023-03-09 19:50:07 [228776645368064-oracle-wait-0507] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 6706ms, All 6706ms]. 2023-03-09 19:50:07 [228776645368064-oracle-process-0507] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 6706ms, All 6706ms]. 2023-03-09 19:50:07 [228776645368064-oracle-tablespace-0507] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 6718ms, All 6718ms]. 2023-03-09 19:50:07 [228776645368064-oracle-transaction-0508] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 6771ms, All 6772ms]. 2023-03-09 19:50:07 [228776645368064-oracle-mem_stats-0508] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 6779ms, All 6780ms]. 2023-03-09 19:50:07 [228776645368064-oracle-cpu_stats-0508] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 6786ms, All 6787ms]. 2023-03-09 19:50:07 [228776645368064-oracle-total_sessions-0507] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 6787ms, All 6787ms]. 2023-03-09 19:50:07 [228776645368064-oracle-percentage-0507] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Failed, Run 6787ms, All 6787ms] Reason: Query Error: ORA-01013: 用户请求取消当前的操作 Code: 1013 2023-03-09 19:50:25 [228887525460224-windows-system-5505] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 5ms, All 5ms]. 2023-03-09 19:50:25 [228887525460224-windows-cpu-5511] WARN com.usthe.collector.collect.snmp.SnmpCollectImpl - [snmp collect] error: SNMP error No such name on index 1 java.util.concurrent.ExecutionException: org.snmp4j.fluent.SnmpCompletableFuture$SnmpErrorException: SNMP error No such name on index 1 at java.base/java.util.concurrent.CompletableFuture.reportGet(Unknown Source) at java.base/java.util.concurrent.CompletableFuture.get(Unknown Source) at com.usthe.collector.collect.snmp.SnmpCollectImpl.collect(SnmpCollectImpl.java:114) at com.usthe.collector.dispatch.MetricsCollect.run(MetricsCollect.java:146) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.snmp4j.fluent.SnmpCompletableFuture$SnmpErrorException: SNMP error No such name on index 1 at org.snmp4j.fluent.SnmpCompletableFuture.onResponse(SnmpCompletableFuture.java:57) at org.snmp4j.Snmp.processPdu(Snmp.java:1261) at org.snmp4j.MessageDispatcherImpl.fireProcessPdu(MessageDispatcherImpl.java:767) at org.snmp4j.MessageDispatcherImpl.dispatchMessage(MessageDispatcherImpl.java:425) at org.snmp4j.MessageDispatcherImpl.processMessage(MessageDispatcherImpl.java:499) at org.snmp4j.MessageDispatcherImpl.processMessage(MessageDispatcherImpl.java:463) at org.snmp4j.util.MultiThreadedMessageDispatcher$MessageTask.run(MultiThreadedMessageDispatcher.java:226) at org.snmp4j.util.ThreadPool$TaskManager.run(ThreadPool.java:312) 2023-03-09 19:50:25 [228887525460224-windows-cpu-5511] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Failed, Run 12ms, All 12ms] Reason: SNMP error No such name on index 1 2023-03-09 19:50:25 [228887525460224-windows-memory-5523] WARN com.usthe.collector.collect.snmp.SnmpCollectImpl - [snmp collect] error: SNMP error No such name on index 2 java.util.concurrent.ExecutionException: org.snmp4j.fluent.SnmpCompletableFuture$SnmpErrorException: SNMP error No such name on index 2 at java.base/java.util.concurrent.CompletableFuture.reportGet(Unknown Source) at java.base/java.util.concurrent.CompletableFuture.get(Unknown Source) at com.usthe.collector.collect.snmp.SnmpCollectImpl.collect(SnmpCollectImpl.java:114) at com.usthe.collector.dispatch.MetricsCollect.run(MetricsCollect.java:146) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.snmp4j.fluent.SnmpCompletableFuture$SnmpErrorException: SNMP error No such name on index 2 at org.snmp4j.fluent.SnmpCompletableFuture.onResponse(SnmpCompletableFuture.java:57) at org.snmp4j.Snmp.processPdu(Snmp.java:1261) at org.snmp4j.MessageDispatcherImpl.fireProcessPdu(MessageDispatcherImpl.java:767) at org.snmp4j.MessageDispatcherImpl.dispatchMessage(MessageDispatcherImpl.java:425) at org.snmp4j.MessageDispatcherImpl.processMessage(MessageDispatcherImpl.java:499) at org.snmp4j.MessageDispatcherImpl.processMessage(MessageDispatcherImpl.java:463) at org.snmp4j.util.MultiThreadedMessageDispatcher$MessageTask.run(MultiThreadedMessageDispatcher.java:226) at org.snmp4j.util.ThreadPool$TaskManager.run(ThreadPool.java:312) 2023-03-09 19:50:25 [228887525460224-windows-memory-5523] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Failed, Run 4ms, All 4ms] Reason: SNMP error No such name on index 2 2023-03-09 19:50:25 [228887525460224-windows-disk-5528] WARN com.usthe.collector.collect.snmp.SnmpCollectImpl - [snmp collect] error: SNMP error No such name on index 1 java.util.concurrent.ExecutionException: org.snmp4j.fluent.SnmpCompletableFuture$SnmpErrorException: SNMP error No such name on index 1 at java.base/java.util.concurrent.CompletableFuture.reportGet(Unknown Source) at java.base/java.util.concurrent.CompletableFuture.get(Unknown Source) at com.usthe.collector.collect.snmp.SnmpCollectImpl.collect(SnmpCollectImpl.java:114) at com.usthe.collector.dispatch.MetricsCollect.run(MetricsCollect.java:146) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.snmp4j.fluent.SnmpCompletableFuture$SnmpErrorException: SNMP error No such name on index 1 at org.snmp4j.fluent.SnmpCompletableFuture.onResponse(SnmpCompletableFuture.java:57) at org.snmp4j.Snmp.processPdu(Snmp.java:1261) at org.snmp4j.MessageDispatcherImpl.fireProcessPdu(MessageDispatcherImpl.java:767) at org.snmp4j.MessageDispatcherImpl.dispatchMessage(MessageDispatcherImpl.java:425) at org.snmp4j.MessageDispatcherImpl.processMessage(MessageDispatcherImpl.java:499) at org.snmp4j.MessageDispatcherImpl.processMessage(MessageDispatcherImpl.java:463) at org.snmp4j.util.MultiThreadedMessageDispatcher$MessageTask.run(MultiThreadedMessageDispatcher.java:226) at org.snmp4j.util.ThreadPool$TaskManager.run(ThreadPool.java:312) 2023-03-09 19:50:25 [228887525460224-windows-disk-5528] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Failed, Run 1ms, All 2ms] Reason: SNMP error No such name on index 1 2023-03-09 19:50:25 [228887525460224-windows-network-5529] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 1ms, All 1ms]. 2023-03-09 19:50:26 [228776751884544-mysql-basic-6505] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 4ms, All 4ms]. 2023-03-09 19:50:26 [228776751884544-mysql-status-6510] INFO com.usthe.collector.dispatch.MetricsCollect - [Collect Success, Run 3ms, All 3ms].

Anything else?

@E}WQ0UPMEH09RN0IDHQMTY FL{3N$NF_@43RY_L%89I}YP

tomsun28 commented 1 year ago

fix by v1.3.0

hertzbeat commented 1 year ago

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿


fix by v1.3.0