Open jdcoats opened 2 weeks ago
then at the end when the smoke clears
2024/06/20 17:57:22 - CMDPHP SQL Backtrace: (/plugins/mactrack/mactrack_scanner.php[241]:db_process_remove(), /plugins/mactrack/lib/mactrack_functions.php[2216]:db_execute_prepared())
2024/06/20 17:57:22 - CMDPHP ERROR: A DB Exec Failed!, Error: MySQL server has gone away
2024/06/20 17:57:22 - CMDPHP SQL Backtrace: (/plugins/mactrack/mactrack_scanner.php[240]:db_update_device_status(), /plugins/mactrack/lib/mactrack_functions.php[2247]:db_execute_prepared())
2024/06/20 17:57:22 - CMDPHP ERROR: A DB Exec Failed!, Error: MySQL server has gone away
2024/06/20 17:57:22 - CMDPHP SQL Backtrace: (/plugins/mactrack/mactrack_scanner.php[199]:get_standard_arp_table(), /plugins/mactrack/lib/mactrack_functions.php[482]:db_execute_prepared())
2024/06/20 17:57:22 - CMDPHP ERROR: A DB Exec Failed!, Error: MySQL server has gone away
2024/06/20 17:57:22 - CMDPHP SQL Backtrace: (/plugins/mactrack/mactrack_scanner.php[199]:get_standard_arp_table(), /plugins/mactrack/lib/mactrack_functions.php[471]:db_execute_prepared())
mysql never really goes away! its up the whole time and everything else is working properly
Run this script as root...
while true; do
clear
uptime
mysql -e "SELECT time, info FROM information_schema.processlist WHERE info!='NULL'
ORDER BY time DESC LIMIT 20"
sleep 2
done
Just watch it when maintenance is running.
we will see if i get the same today around the same time. Note MAINT completed at 17:24 and this happened at 17:57 . I ran maint again manually to see if it triggered this and it did not.
Dan,
You can tee that output to a file while watching it or just see it and then catch it tomorrow.
Larry
@jdcoats, any feedback yet?
@xxconn, once we know the blocking queries, we can do one or more of the following.
register_process_start()
functionregister_process_start()
functionItems 1. and 4. should be done regardless.
Larry
Sadly, I do not have a reproduce environment as I'm not in the network space any longer.
I have not been able to capture the data yet. I will try again this afternoon
didn't happen again yesterday, still working on it.
Anything in the processlist that ran more than a few seconds?
my slow query log from the last time it happened. DSSTATS are commonly the pig in my queries though
INSERT INTO data_source_stats_hourly (local_data_id, rrd_name, average, peak) SELECT local_data_id, rrd_name, AVG(`value`), MAX(`value`) FROM data_source_stats_hourly_cache WHERE `value` IS NOT NULL GROUP BY local_data_id, rrd_name ON D>
# Time: 240624 18:08:04
# User@Host: cacti123[cacti123] @ localhost []
# Thread_id: 102356680 Schema: cacti QC_hit: No
# Query_time: 9.819825 Lock_time: 0.000072 Rows_sent: 0 Rows_examined: 8936906
# Rows_affected: 193292 Bytes_sent: 62
SET timestamp=1719266884;
INSERT INTO data_source_stats_hourly (local_data_id, rrd_name, average, peak) SELECT local_data_id, rrd_name, AVG(`value`), MAX(`value`) FROM data_source_stats_hourly_cache WHERE `value` IS NOT NULL GROUP BY local_data_id, rrd_name ON D>
# Time: 240624 18:09:10
# User@Host: cacti123[cacti123] @ localhost []
# Thread_id: 102357160 Schema: cacti QC_hit: No
# Query_time: 10.571106 Lock_time: 0.000055 Rows_sent: 0 Rows_examined: 8936858
# Rows_affected: 193322 Bytes_sent: 62
SET timestamp=1719266950;
INSERT INTO data_source_stats_hourly (local_data_id, rrd_name, average, peak) SELECT local_data_id, rrd_name, AVG(`value`), MAX(`value`) FROM data_source_stats_hourly_cache WHERE `value` IS NOT NULL GROUP BY local_data_id, rrd_name ON D>
# Time: 240624 18:10:03
# User@Host: cacti123[cacti123] @ localhost []
# Thread_id: 102357458 Schema: cacti QC_hit: No
# Query_time: 9.877193 Lock_time: 0.000049 Rows_sent: 0 Rows_examined: 8936858
# Rows_affected: 193404 Bytes_sent: 62
SET timestamp=1719267003;
INSERT INTO data_source_stats_hourly (local_data_id, rrd_name, average, peak) SELECT local_data_id, rrd_name, AVG(`value`), MAX(`value`) FROM data_source_stats_hourly_cache WHERE `value` IS NOT NULL GROUP BY local_data_id, rrd_name ON D>
# Time: 240624 18:11:10
# User@Host: cacti123[cacti123] @ localhost []
# Thread_id: 102357753 Schema: cacti QC_hit: No
# Query_time: 10.956210 Lock_time: 0.000078 Rows_sent: 0 Rows_examined: 8936834
# Rows_affected: 194458 Bytes_sent: 62
SET timestamp=1719267070;
INSERT INTO data_source_stats_hourly (local_data_id, rrd_name, average, peak) SELECT local_data_id, rrd_name, AVG(`value`), MAX(`value`) FROM data_source_stats_hourly_cache WHERE `value` IS NOT NULL GROUP BY local_data_id, rrd_name ON D>
# Time: 240624 18:12:06
# User@Host: cacti123[cacti123] @ localhost []
# Thread_id: 102358066 Schema: cacti QC_hit: No
# Query_time: 10.437556 Lock_time: 0.000061 Rows_sent: 0 Rows_examined: 8944797
# Rows_affected: 194574 Bytes_sent: 62
SET timestamp=1719267126;
INSERT INTO data_source_stats_hourly (local_data_id, rrd_name, average, peak) SELECT local_data_id, rrd_name, AVG(`value`), MAX(`value`) FROM data_source_stats_hourly_cache WHERE `value` IS NOT NULL GROUP BY local_data_id, rrd_name ON D>
# Time: 240624 18:13:11
# User@Host: cacti123[cacti123] @ localhost []
# Thread_id: 102358368 Schema: cacti QC_hit: No
# Query_time: 10.101875 Lock_time: 0.000052 Rows_sent: 0 Rows_examined: 8944770
# Rows_affected: 193875 Bytes_sent: 62
SET timestamp=1719267191;
INSERT INTO data_source_stats_hourly (local_data_id, rrd_name, average, peak) SELECT local_data_id, rrd_name, AVG(`value`), MAX(`value`) FROM data_source_stats_hourly_cache WHERE `value` IS NOT NULL GROUP BY local_data_id, rrd_name ON D>
# Time: 240624 18:14:05
# User@Host: cacti123[cacti123] @ localhost []
# Thread_id: 102358548 Schema: cacti QC_hit: No
# Query_time: 9.652266 Lock_time: 0.000057 Rows_sent: 0 Rows_examined: 8944731
# Rows_affected: 194130 Bytes_sent: 62
SET timestamp=1719267245;
INSERT INTO data_source_stats_hourly (local_data_id, rrd_name, average, peak) SELECT local_data_id, rrd_name, AVG(`value`), MAX(`value`) FROM data_source_stats_hourly_cache WHERE `value` IS NOT NULL GROUP BY local_data_id, rrd_name ON D>
# Time: 240624 18:15:10
# User@Host: cacti123[cacti123] @ localhost []
# Thread_id: 102359452 Schema: cacti QC_hit: No
# Query_time: 9.791056 Lock_time: 0.000055 Rows_sent: 0 Rows_examined: 8944707
# Rows_affected: 194559 Bytes_sent: 62
SET timestamp=1719267310;
timeinfo
3INSERT INTO data_source_stats_hourly (local_data_id, rrd_name, average, peak) SELECT local_data_id, rrd_name, AVG(`value`), MAX(`value`) FROM data_source_stats_hourly_cache WHERE `value` IS NOT NULL GROUP BY local_data_id, rrd_name ON DUPLICATE KEY UPDATE average=VALUES(average), peak=VALUES(peak)
0SELECT time, info FROM information_schema.processlist WHERE info!='NULL'\nORDER BY time DESC LIMIT 20
0INSERT INTO data_source_stats_hourly_cache (local_data_id, rrd_name, time, `value`) VALUES (242, '5min_cpu', '2024-06-26 16:43:05', 44), (243, 'traffic_out', '2024-06-26 16:43:05', 272.5), (243, 'traffic_in', '2024-06-26 16:43:05', 1075.3), (252, '5min_cpu', '2024-06-26 16:43:05', 50), (253, 'traffic_out', '2024-06-26 16:43:05', 198.5), (253, 'traffic_in', '2024-06-26 16:43:05', 1044.8833333333), (268, '5min_cpu', '2024-06-26 16:43:05', 16), (269, 'traffic_out', '2024-06-26 16:43:05', 203.2), (269, 'traffic_in', '2024-06-26 16:43:05', 881.93333333333), (274, '5min_cpu', '2024-06-26 16:43:05', 22), (275, 'traffic_in', '2024-06-26 16:43:05', 885.2), (275, 'traffic_out', '2024-06-26 16:43:05', 409.26666666667), (276, '5min_cpu', '2024-06-26 16:43:05', 25), (277, 'traffic_in', '2024-06-26 16:43:05', 888.46666666667), (277, 'traffic_out', '2024-06-26 16:43:05', 331.66666666667), (278, '5min_cpu', '2024-06-26 16:43:05', 19), (279, 'traffic_in', '2024-06-26 16:43:05', 886.83333333333), (279, 'traffic_out', '2024-06-26 16:43:05', 365.78333333333), (294, '5min_cpu', '2024-06-26 16:43:05', 53), (295, 'traffic_out', '2024-06-26 16:43:05', 513.91666666667), (295, 'traffic_in', '2024-06-26 16:43:05', 1012), (300, '5min_cpu', '2024-06-26 16:43:05', 18), (301, 'traffic_in', '2024-06-26 16:43:05', 883.56666666667), (301, 'traffic_out', '2024-06-26 16:43:05', 283.23333333333), (326, '5min_cpu', '2024-06-26 16:43:05', 44), (327, 'traffic_out', '2024-06-26 16:43:05', 246.71666666667), (327, 'traffic_in', '2024-06-26 16:43:05', 985.18333333333), (374, '5min_cpu', '2024-06-26 16:43:05', 51), (375, 'traffic_out', '2024-06-26 16:43:05', 205.61666666667), (375, 'traffic_in', '2024-06-26 16:43:05', 980.65), (378, '5min_cpu', '2024-06-26 16:43:05', 58), (379, 'traffic_in', '2024-06-26 16:43:05', 1216.1166666667), (379, 'traffic_out', '2024-06-26 16:43:05', 583.56666666667), (409, '5min_cpu', '2024-06-26 16:43:05', 49), (410, 'traffic_out', '2024-06-26 16:43:05', 186.51666666667), (410, 'traffic_in', '2024-06-26 16:43:05', 1031.6333333333), (411, '5min_cpu', '2024-06-26 16:43:05', 35), (412, 'traffic_out', '2024-06-26 16:43:05', 205.46666666667), (412, 'traffic_in', '2024-06-26 16:43:05', 1071.1166666667), (417, '5min_cpu', '2024-06-26 16:43:05', 18), (418, 'traffic_out', '2024-06-26 16:43:05', 247.65), (418, 'traffic_in', '2024-06-26 16:43:05', 896.63333333333), (1459, 'traffic_out', '2024-06-26 16:43:05', 404.86666666667), (1459, 'traffic_in', '2024-06-26 16:43:05', 868.86666666667), (1460, '5min_cpu', '2024-06-26 16:43:05', 3), (1473, 'traffic_out', '2024-06-26 16:43:05', 434.81666666667), (1473, 'traffic_in', '2024-06-26 16:43:05', 867.23333333333), (1474, '5min_cpu', '2024-06-26 16:43:05', 3), (1478, 'traffic_in', '2024-06-26 16:43:05', 909.7), (1478, 'traffic_out', '2024-06-26 16:43:05', 826.21666666667), (1479, '5min_cpu', '2024-06-26 16:43:05', 2), (1485, 'traffic_out', '2024-06-26 16:43:05', 384.56666666667), (1485, 'traffic_in', '2024-06-26 16:43:05', 865.6), (1486, '5min_cpu', '2024-06-26 16:43:05', 3), (1496, 'traffic_out', '2024-06-26 16:43:05', 492.83333333333), (1496, 'traffic_in', '2024-06-26 16:43:05', 912.96666666667), (1499, '5min_cpu', '2024-06-26 16:43:05', 3), (1510, 'traffic_out', '2024-06-26 16:43:05', 266.05), (1510, 'traffic_in', '2024-06-26 16:43:05', 865.6), (1511, '5min_cpu', '2024-06-26 16:43:05', 2), (1549, 'traffic_out', '2024-06-26 16:43:05', 464.78333333333), (1549, 'traffic_in', '2024-06-26 16:43:05', 911.33333333333), (1551, '5min_cpu', '2024-06-26 16:43:05', 3), (1610, 'traffic_out', '2024-06-26 16:43:05', 2478.4333333333), (1610, 'traffic_in', '2024-06-26 16:43:05', 867.23333333333), (1611, '5min_cpu', '2024-06-26 16:43:05', 2), (1667, 'traffic_in', '2024-06-26 16:43:05', 868.86666666667), (1667, 'traffic_out', '2024-06-26 16:43:05', 1026.25), (1668, '5min_cpu', '2024-06-26 16:43:05', 3), (1720, 'traffic_out', '2024-06-26 16:43:05', 325.43333333333), (1720, 'traffic_in', '2024-06-26 16:43:05', 909.7), (1721, '5min_cpu', '2024-06-26 16:43:05', 2), (2471, 'traffic_out', '2024-06-26 16:43:05', 13154.2), (2471, 'traffic_in', '2024-06-26 16:43:05', 1207.5333333333), (2472, 'traffic_in', '2024-06-26 16:43:05', 35.3), (2472, 'traffic_out', '2024-06-26 16:43:05', 2655.15), (2473, 'traffic_out', '2024-06-26 16:43:05', 12183.766666667), (2473, 'traffic_in', '2024-06-26 16:43:05', 754.98333333333), (2474, 'traffic_out', '2024-06-26 16:43:05', 0), (2474, 'traffic_in', '2024-06-26 16:43:05', 0), (2475, 'traffic_out', '2024-06-26 16:43:05', 2636.85), (2475, 'traffic_in', '2024-06-26 16:43:05', 2.1333333333333), (2476, 'traffic_out', '2024-06-26 16:43:05', 12664.333333333), (2476, 'traffic_in', '2024-06-26 16:43:05', 889.2), (2477, 'traffic_out', '2024-06-26 16:43:05', 2655.3833333333), (2477, 'traffic_in', '2024-06-26 16:43:05', 34.266666666667), (2478, 'traffic_in', '2024-06-26 16:43:05', 825.03333333333), (2478, 'traffic_out', '2024-06-26 16:43:05', 11851.516666667), (2479, 'traffic_out', '2024-06-26 16:43:05', 11748.566666667), (2479, 'traffic_in', '2024-06-26 16:43:05', 222.18333333333), (2480, 'traffic_out', '2024-06-26 16:43:05', 0), (2480, 'traffic_in', '2024-06-26 16:43:05', 0), (2481, 'traffic_out', '2024-06-26 16:43:05', 2646.2833333333), (2481, 'traffic_in', '2024-06-26 16:43:05', 24.583333333333), (2482, 'traffic_out', '2024-06-26 16:43:05', 2664.25), (2482, 'traffic_in', '2024-06-26 16:43:05', 44.55), (2483, 'traffic_out', '2024-06-26 16:43:05', 23033.216666667), (2483, 'traffic_in', '2024-06-26 16:43:05', 11579.75) ON DUPLICATE KEY UPDATE `value`=VALUES(`value`)
16:43:09 up 105 days, 3:04, 7 users, load average: 7.42, 6.69, 5.58
timeinfo
5INSERT INTO data_source_stats_hourly (local_data_id, rrd_name, average, peak) SELECT local_data_id, rrd_name, AVG(`value`), MAX(`value`) FROM data_source_stats_hourly_cache WHERE `value` IS NOT NULL GROUP BY local_data_id, rrd_name ON DUPLICATE KEY UPDATE average=VALUES(average), peak=VALUES(peak)
2INSERT INTO data_source_stats_hourly_cache (local_data_id, rrd_name, time, `value`) VALUES (242, '5min_cpu', '2024-06-26 16:43:05', 44), (243, 'traffic_out', '2024-06-26 16:43:05', 272.5), (243, 'traffic_in', '2024-06-26 16:43:05', 1075.3), (252, '5min_cpu', '2024-06-26 16:43:05', 50), (253, 'traffic_out', '2024-06-26 16:43:05', 198.5), (253, 'traffic_in', '2024-06-26 16:43:05', 1044.8833333333), (268, '5min_cpu', '2024-06-26 16:43:05', 16), (269, 'traffic_out', '2024-06-26 16:43:05', 203.2), (269, 'traffic_in', '2024-06-26 16:43:05', 881.93333333333), (274, '5min_cpu', '2024-06-26 16:43:05', 22), (275, 'traffic_in', '2024-06-26 16:43:05', 885.2), (275, 'traffic_out', '2024-06-26 16:43:05', 409.26666666667), (276, '5min_cpu', '2024-06-26 16:43:05', 25), (277, 'traffic_in', '2024-06-26 16:43:05', 888.46666666667), (277, 'traffic_out', '2024-06-26 16:43:05', 331.66666666667), (278, '5min_cpu', '2024-06-26 16:43:05', 19), (279, 'traffic_in', '2024-06-26 16:43:05', 886.83333333333), (279, 'traffic_out', '2024-06-26 16:43:05', 365.78333333333), (294, '5min_cpu', '2024-06-26 16:43:05', 53), (295, 'traffic_out', '2024-06-26 16:43:05', 513.91666666667), (295, 'traffic_in', '2024-06-26 16:43:05', 1012), (300, '5min_cpu', '2024-06-26 16:43:05', 18), (301, 'traffic_in', '2024-06-26 16:43:05', 883.56666666667), (301, 'traffic_out', '2024-06-26 16:43:05', 283.23333333333), (326, '5min_cpu', '2024-06-26 16:43:05', 44), (327, 'traffic_out', '2024-06-26 16:43:05', 246.71666666667), (327, 'traffic_in', '2024-06-26 16:43:05', 985.18333333333), (374, '5min_cpu', '2024-06-26 16:43:05', 51), (375, 'traffic_out', '2024-06-26 16:43:05', 205.61666666667), (375, 'traffic_in', '2024-06-26 16:43:05', 980.65), (378, '5min_cpu', '2024-06-26 16:43:05', 58), (379, 'traffic_in', '2024-06-26 16:43:05', 1216.1166666667), (379, 'traffic_out', '2024-06-26 16:43:05', 583.56666666667), (409, '5min_cpu', '2024-06-26 16:43:05', 49), (410, 'traffic_out', '2024-06-26 16:43:05', 186.51666666667), (410, 'traffic_in', '2024-06-26 16:43:05', 1031.6333333333), (411, '5min_cpu', '2024-06-26 16:43:05', 35), (412, 'traffic_out', '2024-06-26 16:43:05', 205.46666666667), (412, 'traffic_in', '2024-06-26 16:43:05', 1071.1166666667), (417, '5min_cpu', '2024-06-26 16:43:05', 18), (418, 'traffic_out', '2024-06-26 16:43:05', 247.65), (418, 'traffic_in', '2024-06-26 16:43:05', 896.63333333333), (1459, 'traffic_out', '2024-06-26 16:43:05', 404.86666666667), (1459, 'traffic_in', '2024-06-26 16:43:05', 868.86666666667), (1460, '5min_cpu', '2024-06-26 16:43:05', 3), (1473, 'traffic_out', '2024-06-26 16:43:05', 434.81666666667), (1473, 'traffic_in', '2024-06-26 16:43:05', 867.23333333333), (1474, '5min_cpu', '2024-06-26 16:43:05', 3), (1478, 'traffic_in', '2024-06-26 16:43:05', 909.7), (1478, 'traffic_out', '2024-06-26 16:43:05', 826.21666666667), (1479, '5min_cpu', '2024-06-26 16:43:05', 2), (1485, 'traffic_out', '2024-06-26 16:43:05', 384.56666666667), (1485, 'traffic_in', '2024-06-26 16:43:05', 865.6), (1486, '5min_cpu', '2024-06-26 16:43:05', 3), (1496, 'traffic_out', '2024-06-26 16:43:05', 492.83333333333), (1496, 'traffic_in', '2024-06-26 16:43:05', 912.96666666667), (1499, '5min_cpu', '2024-06-26 16:43:05', 3), (1510, 'traffic_out', '2024-06-26 16:43:05', 266.05), (1510, 'traffic_in', '2024-06-26 16:43:05', 865.6), (1511, '5min_cpu', '2024-06-26 16:43:05', 2), (1549, 'traffic_out', '2024-06-26 16:43:05', 464.78333333333), (1549, 'traffic_in', '2024-06-26 16:43:05', 911.33333333333), (1551, '5min_cpu', '2024-06-26 16:43:05', 3), (1610, 'traffic_out', '2024-06-26 16:43:05', 2478.4333333333), (1610, 'traffic_in', '2024-06-26 16:43:05', 867.23333333333), (1611, '5min_cpu', '2024-06-26 16:43:05', 2), (1667, 'traffic_in', '2024-06-26 16:43:05', 868.86666666667), (1667, 'traffic_out', '2024-06-26 16:43:05', 1026.25), (1668, '5min_cpu', '2024-06-26 16:43:05', 3), (1720, 'traffic_out', '2024-06-26 16:43:05', 325.43333333333), (1720, 'traffic_in', '2024-06-26 16:43:05', 909.7), (1721, '5min_cpu', '2024-06-26 16:43:05', 2), (2471, 'traffic_out', '2024-06-26 16:43:05', 13154.2), (2471, 'traffic_in', '2024-06-26 16:43:05', 1207.5333333333), (2472, 'traffic_in', '2024-06-26 16:43:05', 35.3), (2472, 'traffic_out', '2024-06-26 16:43:05', 2655.15), (2473, 'traffic_out', '2024-06-26 16:43:05', 12183.766666667), (2473, 'traffic_in', '2024-06-26 16:43:05', 754.98333333333), (2474, 'traffic_out', '2024-06-26 16:43:05', 0), (2474, 'traffic_in', '2024-06-26 16:43:05', 0), (2475, 'traffic_out', '2024-06-26 16:43:05', 2636.85), (2475, 'traffic_in', '2024-06-26 16:43:05', 2.1333333333333), (2476, 'traffic_out', '2024-06-26 16:43:05', 12664.333333333), (2476, 'traffic_in', '2024-06-26 16:43:05', 889.2), (2477, 'traffic_out', '2024-06-26 16:43:05', 2655.3833333333), (2477, 'traffic_in', '2024-06-26 16:43:05', 34.266666666667), (2478, 'traffic_in', '2024-06-26 16:43:05', 825.03333333333), (2478, 'traffic_out', '2024-06-26 16:43:05', 11851.516666667), (2479, 'traffic_out', '2024-06-26 16:43:05', 11748.566666667), (2479, 'traffic_in', '2024-06-26 16:43:05', 222.18333333333), (2480, 'traffic_out', '2024-06-26 16:43:05', 0), (2480, 'traffic_in', '2024-06-26 16:43:05', 0), (2481, 'traffic_out', '2024-06-26 16:43:05', 2646.2833333333), (2481, 'traffic_in', '2024-06-26 16:43:05', 24.583333333333), (2482, 'traffic_out', '2024-06-26 16:43:05', 2664.25), (2482, 'traffic_in', '2024-06-26 16:43:05', 44.55), (2483, 'traffic_out', '2024-06-26 16:43:05', 23033.216666667), (2483, 'traffic_in', '2024-06-26 16:43:05', 11579.75) ON DUPLICATE KEY UPDATE `value`=VALUES(`value`)
0SELECT time, info FROM information_schema.processlist WHERE info!='NULL'\nORDER BY time DESC LIMIT 20
16:43:11 up 105 days, 3:04, 7 users, load average: 7.42, 6.69, 5.58
timeinfo
7INSERT INTO data_source_stats_hourly (local_data_id, rrd_name, average, peak) SELECT local_data_id, rrd_name, AVG(`value`), MAX(`value`) FROM data_source_stats_hourly_cache WHERE `value` IS NOT NULL GROUP BY local_data_id, rrd_name ON DUPLICATE KEY UPDATE average=VALUES(average), peak=VALUES(peak)
4INSERT INTO data_source_stats_hourly_cache (local_data_id, rrd_name, time, `value`) VALUES (242, '5min_cpu', '2024-06-26 16:43:05', 44), (243, 'traffic_out', '2024-06-26 16:43:05', 272.5), (243, 'traffic_in', '2024-06-26 16:43:05', 1075.3), (252, '5min_cpu', '2024-06-26 16:43:05', 50), (253, 'traffic_out', '2024-06-26 16:43:05', 198.5), (253, 'traffic_in', '2024-06-26 16:43:05', 1044.8833333333), (268, '5min_cpu', '2024-06-26 16:43:05', 16), (269, 'traffic_out', '2024-06-26 16:43:05', 203.2), (269, 'traffic_in', '2024-06-26 16:43:05', 881.93333333333), (274, '5min_cpu', '2024-06-26 16:43:05', 22), (275, 'traffic_in', '2024-06-26 16:43:05', 885.2), (275, 'traffic_out', '2024-06-26 16:43:05', 409.26666666667), (276, '5min_cpu', '2024-06-26 16:43:05', 25), (277, 'traffic_in', '2024-06-26 16:43:05', 888.46666666667), (277, 'traffic_out', '2024-06-26 16:43:05', 331.66666666667), (278, '5min_cpu', '2024-06-26 16:43:05', 19), (279, 'traffic_in', '2024-06-26 16:43:05', 886.83333333333), (279, 'traffic_out', '2024-06-26 16:43:05', 365.78333333333), (294, '5min_cpu', '2024-06-26 16:43:05', 53), (295, 'traffic_out', '2024-06-26 16:43:05', 513.91666666667), (295, 'traffic_in', '2024-06-26 16:43:05', 1012), (300, '5min_cpu', '2024-06-26 16:43:05', 18), (301, 'traffic_in', '2024-06-26 16:43:05', 883.56666666667), (301, 'traffic_out', '2024-06-26 16:43:05', 283.23333333333), (326, '5min_cpu', '2024-06-26 16:43:05', 44), (327, 'traffic_out', '2024-06-26 16:43:05', 246.71666666667), (327, 'traffic_in', '2024-06-26 16:43:05', 985.18333333333), (374, '5min_cpu', '2024-06-26 16:43:05', 51), (375, 'traffic_out', '2024-06-26 16:43:05', 205.61666666667), (375, 'traffic_in', '2024-06-26 16:43:05', 980.65), (378, '5min_cpu', '2024-06-26 16:43:05', 58), (379, 'traffic_in', '2024-06-26 16:43:05', 1216.1166666667), (379, 'traffic_out', '2024-06-26 16:43:05', 583.56666666667), (409, '5min_cpu', '2024-06-26 16:43:05', 49), (410, 'traffic_out', '2024-06-26 16:43:05', 186.51666666667), (410, 'traffic_in', '2024-06-26 16:43:05', 1031.6333333333), (411, '5min_cpu', '2024-06-26 16:43:05', 35), (412, 'traffic_out', '2024-06-26 16:43:05', 205.46666666667), (412, 'traffic_in', '2024-06-26 16:43:05', 1071.1166666667), (417, '5min_cpu', '2024-06-26 16:43:05', 18), (418, 'traffic_out', '2024-06-26 16:43:05', 247.65), (418, 'traffic_in', '2024-06-26 16:43:05', 896.63333333333), (1459, 'traffic_out', '2024-06-26 16:43:05', 404.86666666667), (1459, 'traffic_in', '2024-06-26 16:43:05', 868.86666666667), (1460, '5min_cpu', '2024-06-26 16:43:05', 3), (1473, 'traffic_out', '2024-06-26 16:43:05', 434.81666666667), (1473, 'traffic_in', '2024-06-26 16:43:05', 867.23333333333), (1474, '5min_cpu', '2024-06-26 16:43:05', 3), (1478, 'traffic_in', '2024-06-26 16:43:05', 909.7), (1478, 'traffic_out', '2024-06-26 16:43:05', 826.21666666667), (1479, '5min_cpu', '2024-06-26 16:43:05', 2), (1485, 'traffic_out', '2024-06-26 16:43:05', 384.56666666667), (1485, 'traffic_in', '2024-06-26 16:43:05', 865.6), (1486, '5min_cpu', '2024-06-26 16:43:05', 3), (1496, 'traffic_out', '2024-06-26 16:43:05', 492.83333333333), (1496, 'traffic_in', '2024-06-26 16:43:05', 912.96666666667), (1499, '5min_cpu', '2024-06-26 16:43:05', 3), (1510, 'traffic_out', '2024-06-26 16:43:05', 266.05), (1510, 'traffic_in', '2024-06-26 16:43:05', 865.6), (1511, '5min_cpu', '2024-06-26 16:43:05', 2), (1549, 'traffic_out', '2024-06-26 16:43:05', 464.78333333333), (1549, 'traffic_in', '2024-06-26 16:43:05', 911.33333333333), (1551, '5min_cpu', '2024-06-26 16:43:05', 3), (1610, 'traffic_out', '2024-06-26 16:43:05', 2478.4333333333), (1610, 'traffic_in', '2024-06-26 16:43:05', 867.23333333333), (1611, '5min_cpu', '2024-06-26 16:43:05', 2), (1667, 'traffic_in', '2024-06-26 16:43:05', 868.86666666667), (1667, 'traffic_out', '2024-06-26 16:43:05', 1026.25), (1668, '5min_cpu', '2024-06-26 16:43:05', 3), (1720, 'traffic_out', '2024-06-26 16:43:05', 325.43333333333), (1720, 'traffic_in', '2024-06-26 16:43:05', 909.7), (1721, '5min_cpu', '2024-06-26 16:43:05', 2), (2471, 'traffic_out', '2024-06-26 16:43:05', 13154.2), (2471, 'traffic_in', '2024-06-26 16:43:05', 1207.5333333333), (2472, 'traffic_in', '2024-06-26 16:43:05', 35.3), (2472, 'traffic_out', '2024-06-26 16:43:05', 2655.15), (2473, 'traffic_out', '2024-06-26 16:43:05', 12183.766666667), (2473, 'traffic_in', '2024-06-26 16:43:05', 754.98333333333), (2474, 'traffic_out', '2024-06-26 16:43:05', 0), (2474, 'traffic_in', '2024-06-26 16:43:05', 0), (2475, 'traffic_out', '2024-06-26 16:43:05', 2636.85), (2475, 'traffic_in', '2024-06-26 16:43:05', 2.1333333333333), (2476, 'traffic_out', '2024-06-26 16:43:05', 12664.333333333), (2476, 'traffic_in', '2024-06-26 16:43:05', 889.2), (2477, 'traffic_out', '2024-06-26 16:43:05', 2655.3833333333), (2477, 'traffic_in', '2024-06-26 16:43:05', 34.266666666667), (2478, 'traffic_in', '2024-06-26 16:43:05', 825.03333333333), (2478, 'traffic_out', '2024-06-26 16:43:05', 11851.516666667), (2479, 'traffic_out', '2024-06-26 16:43:05', 11748.566666667), (2479, 'traffic_in', '2024-06-26 16:43:05', 222.18333333333), (2480, 'traffic_out', '2024-06-26 16:43:05', 0), (2480, 'traffic_in', '2024-06-26 16:43:05', 0), (2481, 'traffic_out', '2024-06-26 16:43:05', 2646.2833333333), (2481, 'traffic_in', '2024-06-26 16:43:05', 24.583333333333), (2482, 'traffic_out', '2024-06-26 16:43:05', 2664.25), (2482, 'traffic_in', '2024-06-26 16:43:05', 44.55), (2483, 'traffic_out', '2024-06-26 16:43:05', 23033.216666667), (2483, 'traffic_in', '2024-06-26 16:43:05', 11579.75) ON DUPLICATE KEY UPDATE `value`=VALUES(`value`)
0SELECT time, info FROM information_schema.processlist WHERE info!='NULL'\nORDER BY time DESC LIMIT 20
16:43:13 up 105 days, 3:04, 7 users, load average: 6.91, 6.60, 5.55
timeinfo
9INSERT INTO data_source_stats_hourly (local_data_id, rrd_name, average, peak) SELECT local_data_id, rrd_name, AVG(`value`), MAX(`value`) FROM data_source_stats_hourly_cache WHERE `value` IS NOT NULL GROUP BY local_data_id, rrd_name ON DUPLICATE KEY UPDATE average=VALUES(average), peak=VALUES(peak)
6INSERT INTO data_source_stats_hourly_cache (local_data_id, rrd_name, time, `value`) VALUES (242, '5min_cpu', '2024-06-26 16:43:05', 44), (243, 'traffic_out', '2024-06-26 16:43:05', 272.5), (243, 'traffic_in', '2024-06-26 16:43:05', 1075.3), (252, '5min_cpu', '2024-06-26 16:43:05', 50), (253, 'traffic_out', '2024-06-26 16:43:05', 198.5), (253, 'traffic_in', '2024-06-26 16:43:05', 1044.8833333333), (268, '5min_cpu', '2024-06-26 16:43:05', 16), (269, 'traffic_out', '2024-06-26 16:43:05', 203.2), (269, 'traffic_in', '2024-06-26 16:43:05', 881.93333333333), (274, '5min_cpu', '2024-06-26 16:43:05', 22), (275, 'traffic_in', '2024-06-26 16:43:05', 885.2), (275, 'traffic_out', '2024-06-26 16:43:05', 409.26666666667), (276, '5min_cpu', '2024-06-26 16:43:05', 25), (277, 'traffic_in', '2024-06-26 16:43:05', 888.46666666667), (277, 'traffic_out', '2024-06-26 16:43:05', 331.66666666667), (278, '5min_cpu', '2024-06-26 16:43:05', 19), (279, 'traffic_in', '2024-06-26 16:43:05', 886.83333333333), (279, 'traffic_out', '2024-06-26 16:43:05', 365.78333333333), (294, '5min_cpu', '2024-06-26 16:43:05', 53), (295, 'traffic_out', '2024-06-26 16:43:05', 513.91666666667), (295, 'traffic_in', '2024-06-26 16:43:05', 1012), (300, '5min_cpu', '2024-06-26 16:43:05', 18), (301, 'traffic_in', '2024-06-26 16:43:05', 883.56666666667), (301, 'traffic_out', '2024-06-26 16:43:05', 283.23333333333), (326, '5min_cpu', '2024-06-26 16:43:05', 44), (327, 'traffic_out', '2024-06-26 16:43:05', 246.71666666667), (327, 'traffic_in', '2024-06-26 16:43:05', 985.18333333333), (374, '5min_cpu', '2024-06-26 16:43:05', 51), (375, 'traffic_out', '2024-06-26 16:43:05', 205.61666666667), (375, 'traffic_in', '2024-06-26 16:43:05', 980.65), (378, '5min_cpu', '2024-06-26 16:43:05', 58), (379, 'traffic_in', '2024-06-26 16:43:05', 1216.1166666667), (379, 'traffic_out', '2024-06-26 16:43:05', 583.56666666667), (409, '5min_cpu', '2024-06-26 16:43:05', 49), (410, 'traffic_out', '2024-06-26 16:43:05', 186.51666666667), (410, 'traffic_in', '2024-06-26 16:43:05', 1031.6333333333), (411, '5min_cpu', '2024-06-26 16:43:05', 35), (412, 'traffic_out', '2024-06-26 16:43:05', 205.46666666667), (412, 'traffic_in', '2024-06-26 16:43:05', 1071.1166666667), (417, '5min_cpu', '2024-06-26 16:43:05', 18), (418, 'traffic_out', '2024-06-26 16:43:05', 247.65), (418, 'traffic_in', '2024-06-26 16:43:05', 896.63333333333), (1459, 'traffic_out', '2024-06-26 16:43:05', 404.86666666667), (1459, 'traffic_in', '2024-06-26 16:43:05', 868.86666666667), (1460, '5min_cpu', '2024-06-26 16:43:05', 3), (1473, 'traffic_out', '2024-06-26 16:43:05', 434.81666666667), (1473, 'traffic_in', '2024-06-26 16:43:05', 867.23333333333), (1474, '5min_cpu', '2024-06-26 16:43:05', 3), (1478, 'traffic_in', '2024-06-26 16:43:05', 909.7), (1478, 'traffic_out', '2024-06-26 16:43:05', 826.21666666667), (1479, '5min_cpu', '2024-06-26 16:43:05', 2), (1485, 'traffic_out', '2024-06-26 16:43:05', 384.56666666667), (1485, 'traffic_in', '2024-06-26 16:43:05', 865.6), (1486, '5min_cpu', '2024-06-26 16:43:05', 3), (1496, 'traffic_out', '2024-06-26 16:43:05', 492.83333333333), (1496, 'traffic_in', '2024-06-26 16:43:05', 912.96666666667), (1499, '5min_cpu', '2024-06-26 16:43:05', 3), (1510, 'traffic_out', '2024-06-26 16:43:05', 266.05), (1510, 'traffic_in', '2024-06-26 16:43:05', 865.6), (1511, '5min_cpu', '2024-06-26 16:43:05', 2), (1549, 'traffic_out', '2024-06-26 16:43:05', 464.78333333333), (1549, 'traffic_in', '2024-06-26 16:43:05', 911.33333333333), (1551, '5min_cpu', '2024-06-26 16:43:05', 3), (1610, 'traffic_out', '2024-06-26 16:43:05', 2478.4333333333), (1610, 'traffic_in', '2024-06-26 16:43:05', 867.23333333333), (1611, '5min_cpu', '2024-06-26 16:43:05', 2), (1667, 'traffic_in', '2024-06-26 16:43:05', 868.86666666667), (1667, 'traffic_out', '2024-06-26 16:43:05', 1026.25), (1668, '5min_cpu', '2024-06-26 16:43:05', 3), (1720, 'traffic_out', '2024-06-26 16:43:05', 325.43333333333), (1720, 'traffic_in', '2024-06-26 16:43:05', 909.7), (1721, '5min_cpu', '2024-06-26 16:43:05', 2), (2471, 'traffic_out', '2024-06-26 16:43:05', 13154.2), (2471, 'traffic_in', '2024-06-26 16:43:05', 1207.5333333333), (2472, 'traffic_in', '2024-06-26 16:43:05', 35.3), (2472, 'traffic_out', '2024-06-26 16:43:05', 2655.15), (2473, 'traffic_out', '2024-06-26 16:43:05', 12183.766666667), (2473, 'traffic_in', '2024-06-26 16:43:05', 754.98333333333), (2474, 'traffic_out', '2024-06-26 16:43:05', 0), (2474, 'traffic_in', '2024-06-26 16:43:05', 0), (2475, 'traffic_out', '2024-06-26 16:43:05', 2636.85), (2475, 'traffic_in', '2024-06-26 16:43:05', 2.1333333333333), (2476, 'traffic_out', '2024-06-26 16:43:05', 12664.333333333), (2476, 'traffic_in', '2024-06-26 16:43:05', 889.2), (2477, 'traffic_out', '2024-06-26 16:43:05', 2655.3833333333), (2477, 'traffic_in', '2024-06-26 16:43:05', 34.266666666667), (2478, 'traffic_in', '2024-06-26 16:43:05', 825.03333333333), (2478, 'traffic_out', '2024-06-26 16:43:05', 11851.516666667), (2479, 'traffic_out', '2024-06-26 16:43:05', 11748.566666667), (2479, 'traffic_in', '2024-06-26 16:43:05', 222.18333333333), (2480, 'traffic_out', '2024-06-26 16:43:05', 0), (2480, 'traffic_in', '2024-06-26 16:43:05', 0), (2481, 'traffic_out', '2024-06-26 16:43:05', 2646.2833333333), (2481, 'traffic_in', '2024-06-26 16:43:05', 24.583333333333), (2482, 'traffic_out', '2024-06-26 16:43:05', 2664.25), (2482, 'traffic_in', '2024-06-26 16:43:05', 44.55), (2483, 'traffic_out', '2024-06-26 16:43:05', 23033.216666667), (2483, 'traffic_in', '2024-06-26 16:43:05', 11579.75) ON DUPLICATE KEY UPDATE `value`=VALUES(`value`)
0SELECT time, info FROM information_schema.processlist WHERE info!='NULL'\nORDER BY time DESC LIMIT 20
16:43:15 up 105 days, 3:04, 7 users, load average: 6.91, 6.60, 5.55
@jdcoats, Is that table memory or some other engine?
data_source_stats_hourly is Innodb and data_source_stats_hourly_cache is memory
MariaDB [cacti]> show table status where name = 'data_source_stats_hourly';
+--------------------------+--------+---------+------------+--------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+------------+--------------------+----------+--------------------+---------+------------------+-----------+
| Name | Engine | Version | Row_format | Rows | Avg_row_length | Data_length | Max_data_length | Index_length | Data_free | Auto_increment | Create_time | Update_time | Check_time | Collation | Checksum | Create_options | Comment | Max_index_length | Temporary |
+--------------------------+--------+---------+------------+--------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+------------+--------------------+----------+--------------------+---------+------------------+-----------+
| data_source_stats_hourly | InnoDB | 10 | Dynamic | 132039 | 91 | 12075008 | 0 | 0 | 4194304 | NULL | 2024-03-11 08:34:31 | 2024-07-01 08:48:21 | NULL | utf8mb4_unicode_ci | NULL | row_format=DYNAMIC | | 0 | N |
+--------------------------+--------+---------+------------+--------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+------------+--------------------+----------+--------------------+---------+------------------+-----------+
1 row in set (0.001 sec)
MariaDB [cacti]> show table status where name = 'data_source_stats_hourly_cache';
+--------------------------------+--------+---------+------------+---------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+--------------------+----------+--------------------+---------+------------------+-----------+
| Name | Engine | Version | Row_format | Rows | Avg_row_length | Data_length | Max_data_length | Index_length | Data_free | Auto_increment | Create_time | Update_time | Check_time | Collation | Checksum | Create_options | Comment | Max_index_length | Temporary |
+--------------------------------+--------+---------+------------+---------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+--------------------+----------+--------------------+---------+------------------+-----------+
| data_source_stats_hourly_cache | MEMORY | 10 | Fixed | 8606526 | 94 | 911019872 | 2727884606 | 537720704 | 82777810 | NULL | 2024-06-28 16:00:19 | NULL | NULL | utf8mb4_unicode_ci | NULL | row_format=DYNAMIC | | 0 | N |
+--------------------------------+--------+---------+------------+---------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+--------------------+----------+--------------------+---------+------------------+-----------+
Not sure why this started again this afternoon after dbmaint ran but here it is
timeline