Cacti / plugin_mactrack

Mactrack Plugin for Cacti
GNU General Public License v2.0
12 stars 24 forks source link

Mactrack leaves resolver process running after each cycle. #198

Closed jdcoats closed 4 months ago

jdcoats commented 10 months ago
~ # ps aux| grep mact
root      4641 23.7  0.0 119976 51472 ?        R    05:03  37:34 /usr/bin/php -q /var/www/localhost/htdocs/cacti/plugins/mactrack/poller_mactrack.php
root      4721  0.0  0.0 113576 44848 ?        S    01:03   0:02 /usr/bin/php -q /var/www/localhost/htdocs/cacti/plugins/mactrack/mactrack_resolver.php
root      5180  0.0  0.0 113576 44832 ?        S    05:03   0:01 /usr/bin/php -q /var/www/localhost/htdocs/cacti/plugins/mactrack/mactrack_resolver.php
root     24506  0.0  0.0 113576 44816 ?        S    Aug17   0:04 /usr/bin/php -q /var/www/localhost/htdocs/cacti/plugins/mactrack/mactrack_resolver.php
root     28943  0.0  0.0 113576 44728 ?        S    Aug17   0:06 /usr/bin/php -q /var/www/localhost/htdocs/cacti/plugins/mactrack/mactrack_resolver.php
jdcoats commented 10 months ago

image

jdcoats commented 10 months ago

when running in debug they just print

2023/08/18 07:41:20 - MACTRACK DEBUG: No IP's require resolving this pass
2023/08/18 07:41:17 - MACTRACK DEBUG: No IP's require resolving this pass
2023/08/18 07:41:14 - MACTRACK DEBUG: No IP's require resolving this pass
2023/08/18 07:41:11 - MACTRACK DEBUG: No IP's require resolving this pass
2023/08/18 07:41:08 - MACTRACK DEBUG: No IP's require resolving this pass
2023/08/18 07:41:05 - MACTRACK DEBUG: No IP's require resolving this pass
2023/08/18 07:41:02 - MACTRACK DEBUG: No IP's require resolving this pass
2023/08/18 07:40:59 - MACTRACK DEBUG: No IP's require resolving this pass
2023/08/18 07:40:56 - MACTRACK DEBUG: No IP's require resolving this pass
2023/08/18 07:40:53 - MACTRACK DEBUG: No IP's require resolving this pass
2023/08/18 07:40:50 - MACTRACK DEBUG: No IP's require resolving this pass
2023/08/18 07:40:47 - MACTRACK DEBUG: No IP's require resolving this pass
2023/08/18 07:40:44 - MACTRACK DEBUG: No IP's require resolving this pass
2023/08/18 07:40:41 - MACTRACK DEBUG: No IP's require resolving this pass
2023/08/18 07:40:38 - MACTRACK DEBUG: No IP's require resolving this pass
2023/08/18 07:40:35 - MACTRACK DEBUG: No IP's require resolving this pass
2023/08/18 07:40:32 - MACTRACK DEBUG: No IP's require resolving this pass
2023/08/18 07:40:29 - MACTRACK DEBUG: No IP's require resolving this pass
daangel27 commented 10 months ago

Hello,

I've got the same issue: apache 41655 0.0 0.5 222080 41556 ? S Aug11 1:55 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 81258 0.0 0.5 222080 41452 ? S Aug11 1:53 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 117058 0.0 0.5 222080 41424 ? S Aug11 1:51 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 154367 0.0 0.5 222080 41472 ? S Aug12 1:50 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 190172 0.0 0.5 222080 41536 ? S Aug12 1:47 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 226033 0.0 0.5 222080 41584 ? S Aug12 1:45 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 261085 0.0 0.5 222080 41236 ? S Aug12 1:43 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 296878 0.0 0.5 222080 41496 ? S Aug12 1:41 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 332640 0.0 0.5 222080 41408 ? S Aug12 1:39 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 370276 0.0 0.5 222080 41396 ? S Aug13 1:37 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 405333 0.0 0.5 222080 41576 ? S Aug13 1:34 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 441123 0.0 0.5 222080 41440 ? S Aug13 1:33 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 476279 0.0 0.5 222080 41592 ? S Aug13 1:32 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 511373 0.0 0.5 222080 41500 ? S Aug13 1:29 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 547114 0.0 0.5 222080 41672 ? S Aug13 1:28 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 584491 0.0 0.5 222080 41528 ? S Aug14 1:25 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 620417 0.0 0.5 222080 41456 ? S Aug14 1:23 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 656545 0.0 0.5 222080 41476 ? S Aug14 1:22 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 692574 0.0 0.5 222080 41320 ? S Aug14 1:18 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 727683 0.0 0.5 222080 41416 ? S Aug14 1:17 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 763457 0.0 0.5 222080 41424 ? S Aug14 1:16 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 800856 0.0 0.5 222080 41356 ? S Aug15 1:13 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 836045 0.0 0.5 222080 41392 ? S Aug15 1:12 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 871951 0.0 0.5 222080 41492 ? S Aug15 1:10 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 907989 0.0 0.5 222080 41516 ? S Aug15 1:08 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 943717 0.0 0.5 222080 41400 ? S Aug15 1:05 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 979524 0.0 0.5 222080 41468 ? S Aug15 1:03 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 1017041 0.0 0.5 222080 41468 ? S Aug16 1:01 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 1052325 0.0 0.5 222080 41364 ? S Aug16 1:00 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 1091408 0.0 0.5 222080 41528 ? S Aug16 0:58 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 1128182 0.0 0.5 222080 41564 ? S Aug16 0:56 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 1165883 0.0 0.5 222080 41580 ? S Aug16 0:54 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 1200968 0.0 0.5 222080 41548 ? S Aug16 0:51 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 1238408 0.0 0.5 222080 41324 ? S Aug17 0:50 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 1274330 0.0 0.5 222080 41292 ? S Aug17 0:49 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 1310285 0.0 0.5 222080 41648 ? S Aug17 0:46 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 1346136 0.0 0.5 222080 41332 ? S Aug17 0:44 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 1381954 0.0 0.5 222080 41464 ? S Aug17 0:43 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 1417793 0.0 0.5 222080 41372 ? S Aug17 0:40 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 1455171 0.0 0.5 222080 41324 ? S Aug18 0:39 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 1491247 0.0 0.5 222080 41348 ? S Aug18 0:36 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 1527232 0.0 0.5 222080 41432 ? S Aug18 0:35 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 1562244 0.0 0.5 222080 41360 ? S Aug18 0:33 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 1598166 0.0 0.5 222080 41400 ? S Aug18 0:30 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 1635484 0.0 0.5 222080 41448 ? S Aug19 0:28 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 1670718 0.0 0.5 222080 41516 ? S Aug19 0:26 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 1706708 0.0 0.5 222080 41468 ? S Aug19 0:25 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 1742525 0.0 0.5 222080 41308 ? S Aug19 0:23 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 1777672 0.0 0.5 222080 41448 ? S Aug19 0:21 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 1813427 0.0 0.5 222080 41652 ? S Aug19 0:19 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 1850982 0.0 0.5 222080 41308 ? S Aug20 0:17 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 1886869 0.0 0.5 222080 41460 ? S Aug20 0:16 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 1922035 0.0 0.5 222080 41632 ? S Aug20 0:13 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 1958175 0.0 0.5 222080 41356 ? S Aug20 0:12 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 1994069 0.0 0.5 222080 41460 ? S Aug20 0:10 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 2029134 0.0 0.5 222080 41448 ? S Aug20 0:09 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 2066539 0.0 0.5 222080 41472 ? S 01:00 0:06 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 2102374 0.0 0.5 222080 41500 ? S 05:05 0:04 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 2137754 0.0 0.5 222080 41488 ? S 09:05 0:03 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php apache 2173686 0.0 0.5 222812 42228 ? S 13:10 0:01 /bin/php -q /usr/share/cacti/plugins/mactrack/poller_mactrack.php apache 2173748 0.0 0.5 222080 41384 ? S 13:10 0:01 /bin/php -q /usr/share/cacti/plugins/mactrack/mactrack_resolver.php

Below mactrack settings:

image

This cause memory usage issue:

image
TheWitness commented 10 months ago

It's a good point. The resolver right now has no process control in it. The resolver can be real slow in a big install as it have to do a lookup on all those IP Addresses, and if you don't have a local DNS cache, it can take quite a bit of time to resolve 20-30k IP addresses over 30 minutes.

jdcoats commented 7 months ago

even after @xmacan added process control it still leave abended processes

root      1457  0.0  0.0 134824 64608 ?        S    02:03   0:08 /usr/bin/php -q /var/www/localhost/htdocs/cacti/plugins/mactrack/mactrack_resolver.php
root      2055  0.0  0.0 134824 64892 ?        S    Dec02   0:49 /usr/bin/php -q /var/www/localhost/htdocs/cacti/plugins/mactrack/mactrack_resolver.php
root      3427  0.0  0.0 134824 64824 ?        S    Dec04   0:21 /usr/bin/php -q /var/www/localhost/htdocs/cacti/plugins/mactrack/mactrack_resolver.php
root      3453  0.0  0.0 134824 64804 ?        S    Dec05   0:13 /usr/bin/php -q /var/www/localhost/htdocs/cacti/plugins/mactrack/mactrack_resolver.php
root      4952  0.0  0.0 134824 64896 ?        S    Dec02   0:52 /usr/bin/php -q /var/www/localhost/htdocs/cacti/plugins/mactrack/mactrack_resolver.php
root      5824  0.0  0.0 134824 64960 ?        S    Dec03   0:33 /usr/bin/php -q /var/www/localhost/htdocs/cacti/plugins/mactrack/mactrack_resolver.php
root      6396  0.0  0.0 134824 64884 ?        S    Dec05   0:17 /usr/bin/php -q /var/www/localhost/htdocs/cacti/plugins/mactrack/mactrack_resolver.php
root      7437 25.6  0.0 122792 52592 ?        S    14:06  61:52 /usr/bin/php -q /var/www/localhost/htdocs/cacti/plugins/mactrack/poller_mactrack.php
root      8226  0.0  0.0 134824 62880 ?        S    14:06   0:02 /usr/bin/php -q /var/www/localhost/htdocs/cacti/plugins/mactrack/mactrack_resolver.php
root      8873  0.0  0.0 134824 64988 ?        S    Dec03   0:37 /usr/bin/php -q /var/www/localhost/htdocs/cacti/plugins/mactrack/mactrack_resolver.php
root     10676  0.0  0.0 134824 65040 ?        S    Dec02   0:54 /usr/bin/php -q /var/www/localhost/htdocs/cacti/plugins/mactrack/mactrack_resolver.php
root     12783  0.0  0.0 134824 64940 ?        S    Dec02   0:46 /usr/bin/php -q /var/www/localhost/htdocs/cacti/plugins/mactrack/mactrack_resolver.php
root     13181  0.0  0.0 134824 64960 ?        S    Dec04   0:28 /usr/bin/php -q /var/www/localhost/htdocs/cacti/plugins/mactrack/mactrack_resolver.php
root     13235  0.0  0.0 134824 64784 ?        S    Dec01   0:58 /usr/bin/php -q /var/www/localhost/htdocs/cacti/plugins/mactrack/mactrack_resolver.php
root     13504  0.0  0.0 134824 64748 ?        S    Dec04   0:26 /usr/bin/php -q /var/www/localhost/htdocs/cacti/plugins/mactrack/mactrack_resolver.php
root     15065  0.0  0.0 134824 64760 ?        S    Dec05   0:12 /usr/bin/php -q /var/www/localhost/htdocs/cacti/plugins/mactrack/mactrack_resolver.php
root     15874  0.0  0.0 134824 64828 ?        S    Dec03   0:43 /usr/bin/php -q /var/www/localhost/htdocs/cacti/plugins/mactrack/mactrack_resolver.php
root     16694  0.0  0.0 134824 64716 ?        S    Dec05   0:09 /usr/bin/php -q /var/www/localhost/htdocs/cacti/plugins/mactrack/mactrack_resolver.php
root     18680  0.0  0.0 134824 64596 ?        S    Dec05   0:15 /usr/bin/php -q /var/www/localhost/htdocs/cacti/plugins/mactrack/mactrack_resolver.php
root     19345  0.1  0.0 116392 45732 ?        S    18:06   0:00 /usr/bin/php -q /var/www/localhost/htdocs/cacti/plugins/mactrack/poller_mactrack.php
root     19707  0.0  0.0 134824 64880 ?        S    Dec02   0:50 /usr/bin/php -q /var/www/localhost/htdocs/cacti/plugins/mactrack/mactrack_resolver.php
root     21284  0.0  0.0   6716  2228 pts/6    S+   18:07   0:00 grep --colour=auto mact
root     22867  0.0  0.0 134824 64620 ?        S    06:04   0:06 /usr/bin/php -q /var/www/localhost/htdocs/cacti/plugins/mactrack/mactrack_resolver.php
root     24171  0.0  0.0 134824 63952 ?        S    10:05   0:04 /usr/bin/php -q /var/www/localhost/htdocs/cacti/plugins/mactrack/mactrack_resolver.php
root     24207  0.0  0.0 134824 64912 ?        S    Dec03   0:35 /usr/bin/php -q /var/www/localhost/htdocs/cacti/plugins/mactrack/mactrack_resolver.php
root     24483  0.0  0.0 134824 64844 ?        S    Dec04   0:24 /usr/bin/php -q /var/www/localhost/htdocs/cacti/plugins/mactrack/mactrack_resolver.php
root     24498  0.0  0.0 134824 64968 ?        S    Dec04   0:31 /usr/bin/php -q /var/www/localhost/htdocs/cacti/plugins/mactrack/mactrack_resolver.php
root     25009  0.0  0.0 134824 64936 ?        S    Dec04   0:18 /usr/bin/php -q /var/www/localhost/htdocs/cacti/plugins/mactrack/mactrack_resolver.php
root     25911  0.0  0.0 134824 64876 ?        S    Dec03   0:39 /usr/bin/php -q /var/www/localhost/htdocs/cacti/plugins/mactrack/mactrack_resolver.php
root     28284  0.0  0.0 134824 65012 ?        S    Dec02   0:57 /usr/bin/php -q /var/www/localhost/htdocs/cacti/plugins/mactrack/mactrack_resolver.php
root     30125  0.0  0.0 134824 64972 ?        S    Dec01   1:00 /usr/bin/php -q /var/www/localhost/htdocs/cacti/plugins/mactrack/mactrack_resolver.php
root     30943  0.0  0.0 134824 64996 ?        S    Dec03   0:44 /usr/bin/php -q /var/www/localhost/htdocs/cacti/plugins/mactrack/mactrack_resolver.php
root     31804  0.0  0.0 134824 64672 ?        S    Dec05   0:11 /usr/bin/php -q /var/www/localhost/htdocs/cacti/plugins/mactrack/mactrack_resolver.php
xmacan commented 7 months ago

please show SQL query result "select * from mac_track_processes;"

jdcoats commented 7 months ago
MariaDB [cacti]> select * from mac_track_processes;
+-----------+------------+---------+---------------------+
| device_id | process_id | status  | start_date          |
+-----------+------------+---------+---------------------+
|         0 |       1777 | Running | 2023-12-08 06:14:43 |
+-----------+------------+---------+---------------------+
1 row in set (0.000 sec)
xmacan commented 7 months ago

and ps aux | grep 1777

jdcoats commented 7 months ago
# ps aux | grep 1777
root      1777  0.0  0.0 132776 62240 ?        S    06:14   0:01 /usr/bin/php -q /var/www/localhost/htdocs/cacti/plugins/mactrack/mactrack_resolver.php
root      4956  0.0  0.0   6716  2228 pts/8    S+   08:28   0:00 grep --colour=auto 1777
xmacan commented 7 months ago

ok, thank you, I will look at it

xmacan commented 4 months ago

@jdcoats Do you still have this problem? If yes, please show me again: ps ax | grep mactrack select from mac_track_processes; select from processes;

jdcoats commented 4 months ago

There is still one stuck from yesterday but its not piles of them like before.

# ps aux | grep mactr
root     13301  0.0  0.0   6584  2228 pts/9    S+   15:23   0:00 grep --colour=auto mactr
root     25697  0.0  0.0 132776 62936 ?        S    Mar04   0:07 /usr/bin/php -q /var/www/localhost/htdocs/cacti/plugins/mactrack/mactrack_resolver.php
MariaDB [cacti]> select * from mac_track_processes;
Empty set (0.000 sec)

MariaDB [cacti]> select * from processes;
+---------+-------+-------------------+----------+--------+---------+---------------------+---------------------+
| id      | pid   | tasktype          | taskname | taskid | timeout | started             | last_update         |
+---------+-------+-------------------+----------+--------+---------+---------------------+---------------------+
| 2029231 | 25697 | mactrack_resolver | master   |      0 |   86400 | 2024-03-04 19:57:30 | 2024-03-04 19:57:30 |
| 2037613 |  3940 | intropage         | master   |      1 |    1800 | 2024-03-05 15:26:32 | 2024-03-05 15:26:32 |
+---------+-------+-------------------+----------+--------+---------+---------------------+---------------------+
2 rows in set (0.030 sec)
jdcoats commented 4 months ago

that's the 86400 timeout working i guess :)