NagiosEnterprises / ncpa

Nagios Cross-Platform Agent
Other
177 stars 95 forks source link

Linux NCPA Agent: UNKNOWN: An error occured connecting to API. (HTTP error: '500 INTERNAL SERVER ERROR') #969

Closed ongsunglau closed 1 year ago

ongsunglau commented 1 year ago

image

All services is working fine, I do able to pull the output also but did not show anything related to ncpa services.

When i check in netstat it's LISTEN. At Nagios XI server also can reach to the destination IP/Port too.

[root@nagios root]# /usr/local/nagios/libexec/check_ncpa.py -H serverIP -t nagiosadmin -P 5693 --list { "root": { "processes": [], "plugins": [], "system": { "node": "server", "uptime": [ 8523323.0, "s" ], "system": "Linux", "machine": "x86_64", "version": "#1 SMP Tue Jun 28 03:02:21 EDT 2022", "time": 1692235430.615638, "release": "4.18.0-372.16.1.el8_6.x86_64", "timezone": [ "+08", "+08" ],

[root@server~]# telnet localhost 5693 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. ^CConnection closed by foreign host. [root@server~]# netstat -plunt | grep 5693 tcp6 0 0 :::5693 :::* LISTEN 3988578/ncpa_listen [root@server~]#

ericloyd commented 1 year ago

What happens if you run the command line version as the nagios user, like the Nagios software would?

ongsunglau commented 1 year ago

Hi Eric,

Tried the command line in NagiosXI, it's able to show, correct me if wrong as what you are expecting?

[root@server root]# /usr/local/nagios/libexec/check_ncpa.py -H serverIP -t nagiosadmin -P 5693 --list { "root": { "processes": [], "plugins": [], "system": { "node": "server", "uptime": [ 8547598.0, "s" ], "system": "Linux", "machine": "x86_64", "version": "#1 SMP Tue Jun 28 03:02:21 EDT 2022", "time": 1692259705.344861, "release": "4.18.0-372.16.1.el8_6.x86_64", "timezone": [ "+08", "+08" ], "processor": "x86_64", "agent_version": "2.4.1" }, "interface": { "lo": { "dropin": [ 0, "packets" ], "bytes_recv": [ 0.01, "GiB" ], "packets_recv": [ 98778, "packets" ], "dropout": [ 0, "packets" ], "bytes_sent": [ 0.01, "GiB" ], "errout": [ 0, "errors" ], "errin": [ 0, "errors" ], "packets_sent": [ 98778, "packets" ] }, "ens192": { "dropin": [ 0, "packets" ], "bytes_recv": [ 168.24, "GiB" ], "packets_recv": [ 100655075, "packets" ], "dropout": [ 0, "packets" ], "bytes_sent": [ 150.57, "GiB" ], "errout": [ 0, "errors" ], "errin": [ 0, "errors" ], "packets_sent": [ 75515487, "packets" ] } }, "user": { "count": [ 1, "users" ], "list": [ [ "root" ], "users" ] }, "memory": { "swap": { "used": [ 0.08, "GiB" ], "percent": [ 0.9, "%" ], "free": [ 7.92, "GiB" ], "swapped_out": [ 0.2, "GiB" ], "swapped_in": [ 0.11, "GiB" ], "total": [ 8.0, "GiB" ] }, "virtual": { "available": [ 5.69, "GiB" ], "total": [ 7.58, "GiB" ], "percent": [ 24.9, "%" ], "free": [ 3.28, "GiB" ], "used": [ 1.46, "GiB" ] } }, "services": [], "disk": { "mount": {}, "logical": { "|var": { "used_percent": [ 4.2, "%" ], "used": [ 0.83, "GiB" ], "max_file_length": 255, "inodes_used": [ 2145, "inodes" ], "free": [ 19.16, "GiB" ], "device_name": [ "/dev/mapper/rhel-var" ], "inodes_free": [ 10483615, "inodes" ], "opts": "rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota", "total": [ 19.99, "GiB" ], "fstype": "xfs", "max_path_length": 4096, "inodes": [ 10485760, "inodes" ], "inodes_used_percent": [ 1.0, "%" ] }, "|boot": { "used_percent": [ 30.2, "%" ], "used": [ 0.3, "GiB" ], "max_file_length": 255, "inodes_used": [ 317, "inodes" ], "free": [ 0.69, "GiB" ], "device_name": [ "/dev/sda1" ], "inodes_free": [ 523971, "inodes" ], "opts": "rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota", "total": [ 0.99, "GiB" ], "fstype": "xfs", "max_path_length": 4096, "inodes": [ 524288, "inodes" ], "inodes_used_percent": [ 1.0, "%" ] }, "|sys|fs|bpf": { "used_percent": [ 0.0, "%" ], "used": [ 0.0, "GiB" ], "max_file_length": 255, "inodes_used": [ 0, "inodes" ], "free": [ 0.0, "GiB" ], "device_name": [ "bpf" ], "inodes_free": [ 0, "inodes" ], "opts": "rw,nosuid,nodev,noexec,relatime,mode=700", "total": [ 0.0, "GiB" ], "fstype": "bpf", "max_path_length": 4096, "inodes": [ 0, "inodes" ], "inodes_used_percent": [ 0, "%" ] }, "|var|log": { "used_percent": [ 4.5, "%" ], "used": [ 0.68, "GiB" ], "max_file_length": 255, "inodes_used": [ 238, "inodes" ], "free": [ 14.31, "GiB" ], "device_name": [ "/dev/mapper/rhel-var_log" ], "inodes_free": [ 7864082, "inodes" ], "opts": "rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota", "total": [ 14.99, "GiB" ], "fstype": "xfs", "max_path_length": 4096, "inodes": [ 7864320, "inodes" ], "inodes_used_percent": [ 1.0, "%" ] }, "|tmp": { "used_percent": [ 1.0, "%" ], "used": [ 0.1, "GiB" ], "max_file_length": 255, "inodes_used": [ 7, "inodes" ], "free": [ 9.89, "GiB" ], "device_name": [ "/dev/mapper/rhel-tmp" ], "inodes_free": [ 5242873, "inodes" ], "opts": "rw,seclabel,nosuid,nodev,noexec,attr2,inode64,logbufs=8,logbsize=32k,noquota", "total": [ 9.99, "GiB" ], "fstype": "xfs", "max_path_length": 4096, "inodes": [ 5242880, "inodes" ], "inodes_used_percent": [ 1.0, "%" ] }, "|home": { "used_percent": [ 0.9, "%" ], "used": [ 0.17, "GiB" ], "max_file_length": 255, "inodes_used": [ 48, "inodes" ], "free": [ 19.82, "GiB" ], "device_name": [ "/dev/mapper/rhel-home" ], "inodes_free": [ 10485712, "inodes" ], "opts": "rw,seclabel,nodev,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota", "total": [ 19.99, "GiB" ], "fstype": "xfs", "max_path_length": 4096, "inodes": [ 10485760, "inodes" ], "inodes_used_percent": [ 1.0, "%" ] }, "|appdata": { "used_percent": [ 0.7, "%" ], "used": [ 0.73, "GiB" ], "max_file_length": 255, "inodes_used": [ 3, "inodes" ], "free": [ 99.22, "GiB" ], "device_name": [ "/dev/mapper/vg_app-lv_app" ], "inodes_free": [ 52426749, "inodes" ], "opts": "rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota", "total": [ 99.95, "GiB" ], "fstype": "xfs", "max_path_length": 4096, "inodes": [ 52426752, "inodes" ], "inodes_used_percent": [ 1.0, "%" ] }, "|var|log|audit": { "used_percent": [ 1.8, "%" ], "used": [ 0.11, "GiB" ], "max_file_length": 255, "inodes_used": [ 8, "inodes" ], "free": [ 5.88, "GiB" ], "device_name": [ "/dev/mapper/rhel-var_log_audit" ], "inodes_free": [ 3145720, "inodes" ], "opts": "rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota", "total": [ 5.99, "GiB" ], "fstype": "xfs", "max_path_length": 4096, "inodes": [ 3145728, "inodes" ], "inodes_used_percent": [ 1.0, "%" ] }, "|var|tmp": { "used_percent": [ 1.0, "%" ], "used": [ 0.1, "GiB" ], "max_file_length": 255, "inodes_used": [ 14, "inodes" ], "free": [ 9.89, "GiB" ], "device_name": [ "/dev/mapper/rhel-var_tmp" ], "inodes_free": [ 5242866, "inodes" ], "opts": "rw,seclabel,nosuid,nodev,noexec,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota", "total": [ 9.99, "GiB" ], "fstype": "xfs", "max_path_length": 4096, "inodes": [ 5242880, "inodes" ], "inodes_used_percent": [ 1.0, "%" ] }, "|": { "used_percent": [ 49.0, "%" ], "used": [ 24.51, "GiB" ], "max_file_length": 255, "inodes_used": [ 58755, "inodes" ], "free": [ 25.47, "GiB" ], "device_name": [ "/dev/mapper/rhel-root" ], "inodes_free": [ 26155645, "inodes" ], "opts": "rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota", "total": [ 49.98, "GiB" ], "fstype": "xfs", "max_path_length": 4096, "inodes": [ 26214400, "inodes" ], "inodes_used_percent": [ 1.0, "%" ] }, "|mnt": { "used_percent": [ 100.0, "%" ], "used": [ 10.68, "GiB" ], "max_file_length": 255, "inodes_used": [ 0, "inodes" ], "free": [ 0.0, "GiB" ], "device_name": [ "/dev/loop0" ], "inodes_free": [ 0, "inodes" ], "opts": "ro,relatime,nojoliet,check=s,map=n,blocksize=2048", "total": [ 10.68, "GiB" ], "fstype": "iso9660", "max_path_length": 4096, "inodes": [ 0, "inodes" ], "inodes_used_percent": [ 0, "%" ] } }, "physical": { "dm-5": { "read_count": [ 198, "c" ], "write_bytes": [ 0.88, "GiB" ], "write_count": [ 85649, "c" ], "read_time": [ 94, "ms" ], "read_bytes": [ 0.01, "GiB" ], "write_time": [ 103387, "ms" ] }, "dm-8": { "read_count": [ 380, "c" ], "write_bytes": [ 0.05, "GiB" ], "write_count": [ 239, "c" ], "read_time": [ 49, "ms" ], "read_bytes": [ 0.01, "GiB" ], "write_time": [ 1902, "ms" ] }, "sdb1": { "read_count": [ 4396, "c" ], "write_bytes": [ 0.05, "GiB" ], "write_count": [ 328, "c" ], "read_time": [ 1317, "ms" ], "read_bytes": [ 0.16, "GiB" ], "write_time": [ 2029, "ms" ] }, "sr0": { "read_count": [ 9, "c" ], "write_bytes": [ 0.0, "GiB" ], "write_count": [ 0, "c" ], "read_time": [ 0, "ms" ], "read_bytes": [ 0.0, "GiB" ], "write_time": [ 0, "ms" ] }, "sda2": { "read_count": [ 77814, "c" ], "write_bytes": [ 75.82, "GiB" ], "write_count": [ 5707164, "c" ], "read_time": [ 126407, "ms" ], "read_bytes": [ 10.96, "GiB" ], "write_time": [ 4648826, "ms" ] }, "sda": { "read_count": [ 81546, "c" ], "write_bytes": [ 75.95, "GiB" ], "write_count": [ 5708283, "c" ], "read_time": [ 128957, "ms" ], "read_bytes": [ 11.23, "GiB" ], "write_time": [ 4650955, "ms" ] }, "sdb": { "read_count": [ 5456, "c" ], "write_bytes": [ 0.05, "GiB" ], "write_count": [ 329, "c" ], "read_time": [ 1808, "ms" ], "read_bytes": [ 0.24, "GiB" ], "write_time": [ 2056, "ms" ] }, "sda1": { "read_count": [ 1091, "c" ], "write_bytes": [ 0.13, "GiB" ], "write_count": [ 1119, "c" ], "read_time": [ 1809, "ms" ], "read_bytes": [ 0.17, "GiB" ], "write_time": [ 2129, "ms" ] }, "dm-6": { "read_count": [ 1479, "c" ], "write_bytes": [ 3.27, "GiB" ], "write_count": [ 632447, "c" ], "read_time": [ 1684, "ms" ], "read_bytes": [ 0.02, "GiB" ], "write_time": [ 383986, "ms" ] }, "dm-7": { "read_count": [ 270, "c" ], "write_bytes": [ 0.83, "GiB" ], "write_count": [ 175099, "c" ], "read_time": [ 302, "ms" ], "read_bytes": [ 0.04, "GiB" ], "write_time": [ 118992, "ms" ] }, "dm-4": { "read_count": [ 29964, "c" ], "write_bytes": [ 27.93, "GiB" ], "write_count": [ 2575677, "c" ], "read_time": [ 22449, "ms" ], "read_bytes": [ 4.2, "GiB" ], "write_time": [ 1337759, "ms" ] }, "loop0": { "read_count": [ 1127, "c" ], "write_bytes": [ 0.0, "GiB" ], "write_count": [ 0, "c" ], "read_time": [ 241, "ms" ], "read_bytes": [ 0.03, "GiB" ], "write_time": [ 0, "ms" ] }, "dm-2": { "read_count": [ 330, "c" ], "write_bytes": [ 0.01, "GiB" ], "write_count": [ 1231, "c" ], "read_time": [ 112, "ms" ], "read_bytes": [ 0.0, "GiB" ], "write_time": [ 369, "ms" ] }, "dm-3": { "read_count": [ 3133, "c" ], "write_bytes": [ 8.06, "GiB" ], "write_count": [ 268208, "c" ], "read_time": [ 38627, "ms" ], "read_bytes": [ 5.68, "GiB" ], "write_time": [ 216819, "ms" ] }, "dm-0": { "read_count": [ 27825, "c" ], "write_bytes": [ 34.64, "GiB" ], "write_count": [ 3128682, "c" ], "read_time": [ 24454, "ms" ], "read_bytes": [ 0.8, "GiB" ], "write_time": [ 1906324, "ms" ] }, "dm-1": { "read_count": [ 27782, "c" ], "write_bytes": [ 0.2, "GiB" ], "write_count": [ 52748, "c" ], "read_time": [ 12926, "ms" ], "read_bytes": [ 0.11, "GiB" ], "write_time": [ 487575, "ms" ] } } }, "cpu": { "count": [ [ 8 ], "cores" ], "idle": [ [ 8531979.75, 8534103.26, 8527669.31, 8528600.28, 8533907.16, 8523204.05, 8533202.84, 8533550.08 ], "ms" ], "percent": [], "system": [ [ 3511.64, 3602.28, 5140.18, 4816.76, 3490.66, 3718.25, 3908.83, 3595.04 ], "ms" ], "user": [ [ 4717.0, 5011.77, 7229.7, 6861.55, 4599.94, 5203.06, 5070.68, 4761.84 ], "ms" ] } } }

ericloyd commented 1 year ago

But you're running it as root. Nagios runs commands as the nagios user. Try "su - nagios" first and then run it as that user.

ongsunglau commented 1 year ago

[nagios@nagiosserver~]$ /usr/local/nagios/libexec/check_ncpa.py -H serverIP -t nagiosadmin -P 5693 -M 'services' -q 'service=ncpa_listener,status=running,match=search' UNKNOWN: An error occured connecting to API. (HTTP error: '500 INTERNAL SERVER ERROR') [nagios@nagiosserver~]$ /usr/local/nagios/libexec/check_ncpa.py -H serverIP -t nagiosadmin -P 5693 -M 'disk/logical/|/used_percent' -w 70 -c 85 OK: Used_percent was 49.00 % | 'used_percent'=49.00%;70;85; [nagios@nagiosserver~]$

I hitting same error

ericloyd commented 1 year ago

So it works fine if you run it as root, and it fails if you run it as nagios, correct? So there's something different between your root user and your nagios user that's causing it to fail. The trick it to figure out what that is. I'm guessing it's a permissions issue on some file somewhere on your Nagios server that the nagios user cannot access properly.

Can you use the check_ncpa.py script to check anything else via NCPA as the nagios user?

ongsunglau commented 1 year ago

Reference i

Slightly correction, i getting same errror messages in root and nagios actually.

`[nagios@nagiosserver ~]$ /usr/local/nagios/libexec/check_ncpa.py -H serverIP -t nagiosadmin -P 5693 -M 'services' -q 'service=ncpa_listener,status=running,match=search' UNKNOWN: An error occured connecting to API. (HTTP error: '500 INTERNAL SERVER ERROR')

[root@nagiosserver ]# /usr/local/nagios/libexec/check_ncpa.py -H serverIP -t nagiosadmin -P 5693 -M 'services' -q 'service=ncpa_listener,status=running,match=search' UNKNOWN: An error occured connecting to API. (HTTP error: '500 INTERNAL SERVER ERROR')

`

ericloyd commented 1 year ago

You switched commands. Run the --list command as the nagios user and see what happens.

ongsunglau commented 1 year ago
[nagios@NagiosServer ~]$  /usr/local/nagios/libexec/check_ncpa.py -H ServerIP -t nagiosadmin -P 5693 --list
{
    "root": {
        "processes": [],
        "plugins": [
            "check_ncpa.py"
        ],
        "system": {
            "node": "TargetServer",
            "uptime": [
                8917562.0,
                "s"
            ],
            "system": "Linux",
            "machine": "x86_64",
            "version": "#1 SMP Tue Jun 28 03:02:21 EDT 2022",
            "time": 1692629713.482748,
            "release": "4.18.0-372.16.1.el8_6.x86_64",
            "timezone": [
                "+08",
                "+08"
            ],
            "processor": "x86_64",
            "agent_version": "2.4.1"
        },
        "interface": {
            "lo": {
                "dropin": [
                    0,
                    "packets"
                ],
                "bytes_recv": [
                    0.01,
                    "GiB"
                ],
                "packets_recv": [
                    105450,
                    "packets"
                ],
                "dropout": [
                    0,
                    "packets"
                ],
                "bytes_sent": [
                    0.01,
                    "GiB"
                ],
                "errout": [
                    0,
                    "errors"
                ],
                "errin": [
                    0,
                    "errors"
                ],
                "packets_sent": [
                    105450,
                    "packets"
                ]
            },
            "ens192": {
                "dropin": [
                    0,
                    "packets"
                ],
                "bytes_recv": [
                    185.38,
                    "GiB"
                ],
                "packets_recv": [
                    110694713,
                    "packets"
                ],
                "dropout": [
                    0,
                    "packets"
                ],
                "bytes_sent": [
                    167.32,
                    "GiB"
                ],
                "errout": [
                    0,
                    "errors"
                ],
                "errin": [
                    0,
                    "errors"
                ],
                "packets_sent": [
                    82906876,
                    "packets"
                ]
            }
        },
        "user": {
            "count": [
                1,
                "users"
            ],
            "list": [
                [
                    "root"
                ],
                "users"
            ]
        },
        "memory": {
            "swap": {
                "used": [
                    0.08,
                    "GiB"
                ],
                "percent": [
                    0.9,
                    "%"
                ],
                "free": [
                    7.92,
                    "GiB"
                ],
                "swapped_out": [
                    0.2,
                    "GiB"
                ],
                "swapped_in": [
                    0.11,
                    "GiB"
                ],
                "total": [
                    8.0,
                    "GiB"
                ]
            },
            "virtual": {
                "available": [
                    5.67,
                    "GiB"
                ],
                "total": [
                    7.58,
                    "GiB"
                ],
                "percent": [
                    25.2,
                    "%"
                ],
                "free": [
                    3.2,
                    "GiB"
                ],
                "used": [
                    1.48,
                    "GiB"
                ]
            }
        },
        "services": [],
        "disk": {
            "mount": {},
            "logical": {
                "|var": {
                    "used_percent": [
                        4.1,
                        "%"
                    ],
                    "used": [
                        0.83,
                        "GiB"
                    ],
                    "max_file_length": 255,
                    "inodes_used": [
                        2145,
                        "inodes"
                    ],
                    "free": [
                        19.16,
                        "GiB"
                    ],
                    "device_name": [
                        "/dev/mapper/rhel-var"
                    ],
                    "inodes_free": [
                        10483615,
                        "inodes"
                    ],
                    "opts": "rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota",
                    "total": [
                        19.99,
                        "GiB"
                    ],
                    "fstype": "xfs",
                    "max_path_length": 4096,
                    "inodes": [
                        10485760,
                        "inodes"
                    ],
                    "inodes_used_percent": [
                        1.0,
                        "%"
                    ]
                },
                "|boot": {
                    "used_percent": [
                        30.2,
                        "%"
                    ],
                    "used": [
                        0.3,
                        "GiB"
                    ],
                    "max_file_length": 255,
                    "inodes_used": [
                        317,
                        "inodes"
                    ],
                    "free": [
                        0.69,
                        "GiB"
                    ],
                    "device_name": [
                        "/dev/sda1"
                    ],
                    "inodes_free": [
                        523971,
                        "inodes"
                    ],
                    "opts": "rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota",
                    "total": [
                        0.99,
                        "GiB"
                    ],
                    "fstype": "xfs",
                    "max_path_length": 4096,
                    "inodes": [
                        524288,
                        "inodes"
                    ],
                    "inodes_used_percent": [
                        1.0,
                        "%"
                    ]
                },
                "|sys|fs|bpf": {
                    "used_percent": [
                        0.0,
                        "%"
                    ],
                    "used": [
                        0.0,
                        "GiB"
                    ],
                    "max_file_length": 255,
                    "inodes_used": [
                        0,
                        "inodes"
                    ],
                    "free": [
                        0.0,
                        "GiB"
                    ],
                    "device_name": [
                        "bpf"
                    ],
                    "inodes_free": [
                        0,
                        "inodes"
                    ],
                    "opts": "rw,nosuid,nodev,noexec,relatime,mode=700",
                    "total": [
                        0.0,
                        "GiB"
                    ],
                    "fstype": "bpf",
                    "max_path_length": 4096,
                    "inodes": [
                        0,
                        "inodes"
                    ],
                    "inodes_used_percent": [
                        0,
                        "%"
                    ]
                },
                "|var|log": {
                    "used_percent": [
                        4.7,
                        "%"
                    ],
                    "used": [
                        0.71,
                        "GiB"
                    ],
                    "max_file_length": 255,
                    "inodes_used": [
                        238,
                        "inodes"
                    ],
                    "free": [
                        14.29,
                        "GiB"
                    ],
                    "device_name": [
                        "/dev/mapper/rhel-var_log"
                    ],
                    "inodes_free": [
                        7864082,
                        "inodes"
                    ],
                    "opts": "rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota",
                    "total": [
                        14.99,
                        "GiB"
                    ],
                    "fstype": "xfs",
                    "max_path_length": 4096,
                    "inodes": [
                        7864320,
                        "inodes"
                    ],
                    "inodes_used_percent": [
                        1.0,
                        "%"
                    ]
                },
                "|tmp": {
                    "used_percent": [
                        1.0,
                        "%"
                    ],
                    "used": [
                        0.1,
                        "GiB"
                    ],
                    "max_file_length": 255,
                    "inodes_used": [
                        7,
                        "inodes"
                    ],
                    "free": [
                        9.89,
                        "GiB"
                    ],
                    "device_name": [
                        "/dev/mapper/rhel-tmp"
                    ],
                    "inodes_free": [
                        5242873,
                        "inodes"
                    ],
                    "opts": "rw,seclabel,nosuid,nodev,noexec,attr2,inode64,logbufs=8,logbsize=32k,noquota",
                    "total": [
                        9.99,
                        "GiB"
                    ],
                    "fstype": "xfs",
                    "max_path_length": 4096,
                    "inodes": [
                        5242880,
                        "inodes"
                    ],
                    "inodes_used_percent": [
                        1.0,
                        "%"
                    ]
                },
                "|home": {
                    "used_percent": [
                        0.9,
                        "%"
                    ],
                    "used": [
                        0.17,
                        "GiB"
                    ],
                    "max_file_length": 255,
                    "inodes_used": [
                        48,
                        "inodes"
                    ],
                    "free": [
                        19.82,
                        "GiB"
                    ],
                    "device_name": [
                        "/dev/mapper/rhel-home"
                    ],
                    "inodes_free": [
                        10485712,
                        "inodes"
                    ],
                    "opts": "rw,seclabel,nodev,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota",
                    "total": [
                        19.99,
                        "GiB"
                    ],
                    "fstype": "xfs",
                    "max_path_length": 4096,
                    "inodes": [
                        10485760,
                        "inodes"
                    ],
                    "inodes_used_percent": [
                        1.0,
                        "%"
                    ]
                },
                "|appdata": {
                    "used_percent": [
                        0.7,
                        "%"
                    ],
                    "used": [
                        0.73,
                        "GiB"
                    ],
                    "max_file_length": 255,
                    "inodes_used": [
                        3,
                        "inodes"
                    ],
                    "free": [
                        99.22,
                        "GiB"
                    ],
                    "device_name": [
                        "/dev/mapper/vg_app-lv_app"
                    ],
                    "inodes_free": [
                        52426749,
                        "inodes"
                    ],
                    "opts": "rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota",
                    "total": [
                        99.95,
                        "GiB"
                    ],
                    "fstype": "xfs",
                    "max_path_length": 4096,
                    "inodes": [
                        52426752,
                        "inodes"
                    ],
                    "inodes_used_percent": [
                        1.0,
                        "%"
                    ]
                },
                "|var|log|audit": {
                    "used_percent": [
                        1.8,
                        "%"
                    ],
                    "used": [
                        0.11,
                        "GiB"
                    ],
                    "max_file_length": 255,
                    "inodes_used": [
                        8,
                        "inodes"
                    ],
                    "free": [
                        5.88,
                        "GiB"
                    ],
                    "device_name": [
                        "/dev/mapper/rhel-var_log_audit"
                    ],
                    "inodes_free": [
                        3145720,
                        "inodes"
                    ],
                    "opts": "rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota",
                    "total": [
                        5.99,
                        "GiB"
                    ],
                    "fstype": "xfs",
                    "max_path_length": 4096,
                    "inodes": [
                        3145728,
                        "inodes"
                    ],
                    "inodes_used_percent": [
                        1.0,
                        "%"
                    ]
                },
                "|var|tmp": {
                    "used_percent": [
                        1.3,
                        "%"
                    ],
                    "used": [
                        0.13,
                        "GiB"
                    ],
                    "max_file_length": 255,
                    "inodes_used": [
                        15,
                        "inodes"
                    ],
                    "free": [
                        9.86,
                        "GiB"
                    ],
                    "device_name": [
                        "/dev/mapper/rhel-var_tmp"
                    ],
                    "inodes_free": [
                        5242865,
                        "inodes"
                    ],
                    "opts": "rw,seclabel,nosuid,nodev,noexec,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota",
                    "total": [
                        9.99,
                        "GiB"
                    ],
                    "fstype": "xfs",
                    "max_path_length": 4096,
                    "inodes": [
                        5242880,
                        "inodes"
                    ],
                    "inodes_used_percent": [
                        1.0,
                        "%"
                    ]
                },
                "|": {
                    "used_percent": [
                        49.0,
                        "%"
                    ],
                    "used": [
                        24.51,
                        "GiB"
                    ],
                    "max_file_length": 255,
                    "inodes_used": [
                        58768,
                        "inodes"
                    ],
                    "free": [
                        25.47,
                        "GiB"
                    ],
                    "device_name": [
                        "/dev/mapper/rhel-root"
                    ],
                    "inodes_free": [
                        26155632,
                        "inodes"
                    ],
                    "opts": "rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota",
                    "total": [
                        49.98,
                        "GiB"
                    ],
                    "fstype": "xfs",
                    "max_path_length": 4096,
                    "inodes": [
                        26214400,
                        "inodes"
                    ],
                    "inodes_used_percent": [
                        1.0,
                        "%"
                    ]
                },
                "|mnt": {
                    "used_percent": [
                        100.0,
                        "%"
                    ],
                    "used": [
                        10.68,
                        "GiB"
                    ],
                    "max_file_length": 255,
                    "inodes_used": [
                        0,
                        "inodes"
                    ],
                    "free": [
                        0.0,
                        "GiB"
                    ],
                    "device_name": [
                        "/dev/loop0"
                    ],
                    "inodes_free": [
                        0,
                        "inodes"
                    ],
                    "opts": "ro,relatime,nojoliet,check=s,map=n,blocksize=2048",
                    "total": [
                        10.68,
                        "GiB"
                    ],
                    "fstype": "iso9660",
                    "max_path_length": 4096,
                    "inodes": [
                        0,
                        "inodes"
                    ],
                    "inodes_used_percent": [
                        0,
                        "%"
                    ]
                }
            },
            "physical": {
                "dm-5": {
                    "read_count": [
                        198,
                        "c"
                    ],
                    "write_bytes": [
                        0.95,
                        "GiB"
                    ],
                    "write_count": [
                        91087,
                        "c"
                    ],
                    "read_time": [
                        94,
                        "ms"
                    ],
                    "read_bytes": [
                        0.01,
                        "GiB"
                    ],
                    "write_time": [
                        109940,
                        "ms"
                    ]
                },
                "dm-8": {
                    "read_count": [
                        380,
                        "c"
                    ],
                    "write_bytes": [
                        0.05,
                        "GiB"
                    ],
                    "write_count": [
                        249,
                        "c"
                    ],
                    "read_time": [
                        49,
                        "ms"
                    ],
                    "read_bytes": [
                        0.01,
                        "GiB"
                    ],
                    "write_time": [
                        2015,
                        "ms"
                    ]
                },
                "sdb1": {
                    "read_count": [
                        4646,
                        "c"
                    ],
                    "write_bytes": [
                        0.05,
                        "GiB"
                    ],
                    "write_count": [
                        338,
                        "c"
                    ],
                    "read_time": [
                        1407,
                        "ms"
                    ],
                    "read_bytes": [
                        0.17,
                        "GiB"
                    ],
                    "write_time": [
                        2144,
                        "ms"
                    ]
                },
                "sr0": {
                    "read_count": [
                        9,
                        "c"
                    ],
                    "write_bytes": [
                        0.0,
                        "GiB"
                    ],
                    "write_count": [
                        0,
                        "c"
                    ],
                    "read_time": [
                        0,
                        "ms"
                    ],
                    "read_bytes": [
                        0.0,
                        "GiB"
                    ],
                    "write_time": [
                        0,
                        "ms"
                    ]
                },
                "sda2": {
                    "read_count": [
                        77943,
                        "c"
                    ],
                    "write_bytes": [
                        77.91,
                        "GiB"
                    ],
                    "write_count": [
                        5963605,
                        "c"
                    ],
                    "read_time": [
                        126480,
                        "ms"
                    ],
                    "read_bytes": [
                        10.96,
                        "GiB"
                    ],
                    "write_time": [
                        4849602,
                        "ms"
                    ]
                },
                "sda": {
                    "read_count": [
                        81889,
                        "c"
                    ],
                    "write_bytes": [
                        78.04,
                        "GiB"
                    ],
                    "write_count": [
                        5964786,
                        "c"
                    ],
                    "read_time": [
                        129168,
                        "ms"
                    ],
                    "read_bytes": [
                        11.24,
                        "GiB"
                    ],
                    "write_time": [
                        4851763,
                        "ms"
                    ]
                },
                "sdb": {
                    "read_count": [
                        5758,
                        "c"
                    ],
                    "write_bytes": [
                        0.05,
                        "GiB"
                    ],
                    "write_count": [
                        339,
                        "c"
                    ],
                    "read_time": [
                        1932,
                        "ms"
                    ],
                    "read_bytes": [
                        0.25,
                        "GiB"
                    ],
                    "write_time": [
                        2171,
                        "ms"
                    ]
                },
                "sda1": {
                    "read_count": [
                        1141,
                        "c"
                    ],
                    "write_bytes": [
                        0.13,
                        "GiB"
                    ],
                    "write_count": [
                        1181,
                        "c"
                    ],
                    "read_time": [
                        1899,
                        "ms"
                    ],
                    "read_bytes": [
                        0.18,
                        "GiB"
                    ],
                    "write_time": [
                        2161,
                        "ms"
                    ]
                },
                "dm-6": {
                    "read_count": [
                        1479,
                        "c"
                    ],
                    "write_bytes": [
                        3.52,
                        "GiB"
                    ],
                    "write_count": [
                        678731,
                        "c"
                    ],
                    "read_time": [
                        1684,
                        "ms"
                    ],
                    "read_bytes": [
                        0.02,
                        "GiB"
                    ],
                    "write_time": [
                        402128,
                        "ms"
                    ]
                },
                "dm-7": {
                    "read_count": [
                        270,
                        "c"
                    ],
                    "write_bytes": [
                        0.85,
                        "GiB"
                    ],
                    "write_count": [
                        180111,
                        "c"
                    ],
                    "read_time": [
                        302,
                        "ms"
                    ],
                    "read_bytes": [
                        0.04,
                        "GiB"
                    ],
                    "write_time": [
                        123717,
                        "ms"
                    ]
                },
                "dm-4": {
                    "read_count": [
                        29964,
                        "c"
                    ],
                    "write_bytes": [
                        28.46,
                        "GiB"
                    ],
                    "write_count": [
                        2649470,
                        "c"
                    ],
                    "read_time": [
                        22449,
                        "ms"
                    ],
                    "read_bytes": [
                        4.2,
                        "GiB"
                    ],
                    "write_time": [
                        1381246,
                        "ms"
                    ]
                },
                "loop0": {
                    "read_count": [
                        1131,
                        "c"
                    ],
                    "write_bytes": [
                        0.0,
                        "GiB"
                    ],
                    "write_count": [
                        0,
                        "c"
                    ],
                    "read_time": [
                        243,
                        "ms"
                    ],
                    "read_bytes": [
                        0.03,
                        "GiB"
                    ],
                    "write_time": [
                        0,
                        "ms"
                    ]
                },
                "dm-2": {
                    "read_count": [
                        330,
                        "c"
                    ],
                    "write_bytes": [
                        0.01,
                        "GiB"
                    ],
                    "write_count": [
                        1248,
                        "c"
                    ],
                    "read_time": [
                        112,
                        "ms"
                    ],
                    "read_bytes": [
                        0.0,
                        "GiB"
                    ],
                    "write_time": [
                        377,
                        "ms"
                    ]
                },
                "dm-3": {
                    "read_count": [
                        3133,
                        "c"
                    ],
                    "write_bytes": [
                        8.12,
                        "GiB"
                    ],
                    "write_count": [
                        283709,
                        "c"
                    ],
                    "read_time": [
                        38627,
                        "ms"
                    ],
                    "read_bytes": [
                        5.68,
                        "GiB"
                    ],
                    "write_time": [
                        223397,
                        "ms"
                    ]
                },
                "dm-0": {
                    "read_count": [
                        27825,
                        "c"
                    ],
                    "write_bytes": [
                        35.8,
                        "GiB"
                    ],
                    "write_count": [
                        3307358,
                        "c"
                    ],
                    "read_time": [
                        24454,
                        "ms"
                    ],
                    "read_bytes": [
                        0.8,
                        "GiB"
                    ],
                    "write_time": [
                        2001195,
                        "ms"
                    ]
                },
                "dm-1": {
                    "read_count": [
                        27853,
                        "c"
                    ],
                    "write_bytes": [
                        0.2,
                        "GiB"
                    ],
                    "write_count": [
                        52748,
                        "c"
                    ],
                    "read_time": [
                        12964,
                        "ms"
                    ],
                    "read_bytes": [
                        0.11,
                        "GiB"
                    ],
                    "write_time": [
                        487575,
                        "ms"
                    ]
                }
            }
        },
        "cpu": {
            "count": [
                [
                    8
                ],
                "cores"
            ],
            "idle": [
                [
                    8901005.16,
                    8903235.96,
                    8896524.73,
                    8897486.71,
                    8903051.24,
                    8891884.05,
                    8902299.63,
                    8902665.57
                ],
                "ms"
            ],
            "percent": [],
            "system": [
                [
                    3729.15,
                    3820.94,
                    5424.09,
                    5086.67,
                    3712.75,
                    3948.38,
                    4140.26,
                    3812.65
                ],
                "ms"
            ],
            "user": [
                [
                    4999.65,
                    5323.68,
                    7629.72,
                    7235.82,
                    4892.45,
                    5531.15,
                    5383.93,
                    5056.68
                ],
                "ms"
            ]
        }
    }
}
[nagios@NagiosServer ~]$
ericloyd commented 1 year ago

Great, so you have determined that the underlying check_ncpa.py plugin is actually working on the Nagios server, and the NCPA listener on the client is running, responding, and returning results (we call this "the three R's"). So now the problem is to figure out why you get a failure when trying to run a specific check.

Do you get the 500 error with every NCPA check? Like, can you do this:

/usr/local/nagios/libexec/check_ncpa.py -H HOSTNAME -t TOKEN -M "cpu/percent"
ongsunglau commented 1 year ago

It's return the results.

[nagios@NagiosServer~]$  /usr/local/nagios/libexec/check_ncpa.py -H ServerIP -t nagiosadmin -P 5693 -M "cpu/percent"
OK: Percent was 0.00 %, 0.00 %, 0.00 %, 0.00 %, 0.00 %, 0.00 %, 0.00 %, 0.00 % | 'percent_0'=0.00%;;; 'percent_1'=0.00%;;; 'percent_2'=0.00%;;; 'percent_3'=0.00%;;; 'percent_4'=0.00%;;; 'percent_5'=0.00%;;; 'percent_6'=0.00%;;; 'percent_7'=0.00%;;;
ericloyd commented 1 year ago

Please try with just

-q "service=ncpa_listener"

with none of the other options for -q

ongsunglau commented 1 year ago

Hi,

Do you mean like this?

/usr/local/nagios/libexec/check_ncpa.py -H ServerIP -t nagiosadmin -P 5693 -M -q "service=ncpa_listener"

ericloyd commented 1 year ago

yes.

ongsunglau commented 1 year ago

No luck for me

[nagios@NagiosServer~]$ /usr/local/nagios/libexec/check_ncpa.py -H ServerIP -t nagiosadmin -P 5693 -M -q "service=ncpa_listener" UNKNOWN: The node (-q) requested does not exist.

ericloyd commented 1 year ago

No, leave the other stuff alone so it looks like this:

/usr/local/nagios/libexec/check_ncpa.py -H ServerIP -t nagiosadmin -P 5693 -M "services" -q "service=ncpa_listener"
SNapier commented 1 year ago

./check_ncpa.py -H -t '' -P 5693 -M 'services' -q 'service=ncpa_listener'

Try this command, you're missing the Module definition in the line above

On Mon, Aug 21, 2023 at 11:15 AM eddieontheroad @.***> wrote:

No luck for me

@.***~]$ /usr/local/nagios/libexec/check_ncpa.py -H ServerIP -t nagiosadmin -P 5693 -M -q "service=ncpa_listener" UNKNOWN: The node (-q) requested does not exist.

— Reply to this email directly, view it on GitHub https://github.com/NagiosEnterprises/ncpa/issues/969#issuecomment-1686625890, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMCUBVWCKWWEELOWPKDTCIDXWOCSLANCNFSM6AAAAAA3TKFTU4 . You are receiving this because you are subscribed to this thread.Message ID: @.***>

ongsunglau commented 1 year ago

Thanks for correction

[nagios@NagiosServer~]$ /usr/local/nagios/libexec/check_ncpa.py -H ServerIP -t nagiosadmin -P 5693 -M "services" -q "service=ncpa_listener" UNKNOWN: An error occured connecting to API. (HTTP error: '500 INTERNAL SERVER ERROR')

SNapier commented 1 year ago
  1. Login to the WebUI for the agent on the target server.

  2. Use the API navigator to create the service check you are wanting to run. [image: image.png] (This is 3-Beta so the NCPA service is now singular but, you get the point.)

  3. Execute this as a nagios check and see what you get for output.

It's possible your install is either broken or corrupt, just saying... --SN.

On Mon, Aug 21, 2023 at 11:25 AM eddieontheroad @.***> wrote:

Thanks for correction

@.***~]$ /usr/local/nagios/libexec/check_ncpa.py -H ServerIP -t nagiosadmin -P 5693 -M "services" -q "service=ncpa_listener" UNKNOWN: An error occured connecting to API. (HTTP error: '500 INTERNAL SERVER ERROR')

— Reply to this email directly, view it on GitHub https://github.com/NagiosEnterprises/ncpa/issues/969#issuecomment-1686642998, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMCUBVUEM5BBS5Z3UL2A763XWODV3ANCNFSM6AAAAAA3TKFTU4 . You are receiving this because you commented.Message ID: @.***>

ericloyd commented 1 year ago

I was typing something longer when this reply came in. I agree that the client install is broken. Can you reinstall?

ongsunglau commented 1 year ago
  1. Login to the WebUI for the agent on the target server. 2. Use the API navigator to create the service check you are wanting to run. [image: image.png] (This is 3-Beta so the NCPA service is now singular but, you get the point.) 3. Execute this as a nagios check and see what you get for output. It's possible your install is either broken or corrupt, just saying... --SN. On Mon, Aug 21, 2023 at 11:25 AM eddieontheroad @.> wrote: Thanks for correction @.~]$ /usr/local/nagios/libexec/check_ncpa.py -H ServerIP -t nagiosadmin -P 5693 -M "services" -q "service=ncpa_listener" UNKNOWN: An error occured connecting to API. (HTTP error: '500 INTERNAL SERVER ERROR') — Reply to this email directly, view it on GitHub <#969 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMCUBVUEM5BBS5Z3UL2A763XWODV3ANCNFSM6AAAAAA3TKFTU4 . You are receiving this because you commented.Message ID: @.***>

I trying to look for image.png, but can't see it.

SNapier commented 1 year ago

This section of the getting started docs talks about the WebUI and how do what the image showed.

https://www.nagios.org/ncpa/getting-started.php#using-the-gui

--SN

On Mon, Aug 21, 2023 at 11:36 AM eddieontheroad @.***> wrote:

  1. Login to the WebUI for the agent on the target server. 2. Use the API navigator to create the service check you are wanting to run. [image: image.png] (This is 3-Beta so the NCPA service is now singular but, you get the point.) 3. Execute this as a nagios check and see what you get for output. It's possible your install is either broken or corrupt, just saying... --SN. … <#m8668581500687826720> On Mon, Aug 21, 2023 at 11:25 AM eddieontheroad @.> wrote: Thanks for correction @.~]$ /usr/local/nagios/libexec/check_ncpa.py -H ServerIP -t nagiosadmin -P 5693 -M "services" -q "service=ncpa_listener" UNKNOWN: An error occured connecting to API. (HTTP error: '500 INTERNAL SERVER ERROR') — Reply to this email directly, view it on GitHub <#969 (comment) https://github.com/NagiosEnterprises/ncpa/issues/969#issuecomment-1686642998>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMCUBVUEM5BBS5Z3UL2A763XWODV3ANCNFSM6AAAAAA3TKFTU4 . You are receiving this because you commented.Message ID: @.***>

I trying to look for image.png, but can't see it.

— Reply to this email directly, view it on GitHub https://github.com/NagiosEnterprises/ncpa/issues/969#issuecomment-1686660150, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMCUBVUWHHT34XOYBOHDCADXWOFBNANCNFSM6AAAAAA3TKFTU4 . You are receiving this because you commented.Message ID: @.***>

ericloyd commented 1 year ago

Go to the API, go to services, select ncpa_listener (I'm looking at my local Windows desktop so it's ncpalistener) and see it as a Nagios check_ncpa.py check format as this: image

ongsunglau commented 1 year ago

NCPA listener is running but when switch to tab API the services API Endpoint keep loading, tried others server which is immediately will get the results, will try reinstall while waiting reply.

loading ncpa
ongsunglau commented 1 year ago

I believe I was downloaded latest el8 rpm but the ncpa packages is build for el7

[root@Server~]# rpm -qi ncpa-2.4.1.el8.x86_64.rpm Name : ncpa Version : 2.4.1 Release : 1.el7 Architecture: x86_64 Install Date: (not installed) Group : Network/Monitoring Size : 30947026 License : Nagios Community Software License Version 1.3 Signature : (none) Source RPM : ncpa-2.4.1-1.el7.src.rpm Build Date : Sat 18 Feb 2023 05:15:08 AM +08 Build Host : cent7 Relocations : /usr/local Vendor : Nagios Enterprises, LLC URL : https://www.nagios.org/ncpa/help.php Summary : A cross-platform active and passive monitoring agent Description : The Nagios Cross-Platform Agent is used with Nagios XI and Nagios Core to run active and/or passive checks on any operating system. Installs with zero requirements using a bundled version of Python.

I have done reinstall using ncpa-2.4.0-1.el8.x86_64, but still hitting same error.

ericloyd commented 1 year ago

At this point, I'm blaming selinux. What is the output of:

sestatus
ongsunglau commented 1 year ago

At this point, I'm blaming selinux. What is the output of:

sestatus

sestatus is running actually.

ericloyd commented 1 year ago

Yes, that may be the problem. Try disabling it:

sudo setenforce 0

Then see how things go

ongsunglau commented 1 year ago

Not working,

I do setenforce 0 and retry again on Nagios server still hitting same error

ericloyd commented 1 year ago

Do you have another machine that you can test it on that is similar to the one that isn't working?

ongsunglau commented 1 year ago

Do you have another machine that you can test it on that is similar to the one that isn't working?

We only have 1 Linux and 1 Unix machine having same issue and others not getting same error.

MrPippin66 commented 1 year ago

On the server with the agent, can you "su" to 'nagios' id and verify you can write files to /tmp?

"su -s /bin/bash - nagios"

Also verify you can run systemd commands like "systemctl list-unit-files"

phreditorNG commented 1 year ago

Hey @ongsunglau: Does the log (probably at /usr/local/ncpa/var/log/ncpa_listener.log) which might give insight into where things are going wrong?

ongsunglau commented 1 year ago

On the server with the agent, can you "su" to 'nagios' id and verify you can write files to /tmp?

"su -s /bin/bash - nagios"

Also verify you can run systemd commands like "systemctl list-unit-files"

[nagios@TargetServer ~]$ cd /tmp/
[nagios@TargetServer tmp]$ ls
systemd-private-9428b9285cbb46d68380e39a838b9f68-chronyd.service-Ej1AdW
systemd-private-9428b9285cbb46d68380e39a838b9f68-httpd.service-kwTUX5
[nagios@TargetServer tmp]$ touch 123
touch: cannot touch '123': Permission denied

[nagios@TargetServer tmp]$ systemctl list-unit-files | grep ncpa
ncpa_listener.service                      generated
ncpa_passive.service                       generated
ongsunglau commented 1 year ago

I managed to resolve it after changes from

[root@TargetServer ~]# ls -ld /tmp drwxr-xr-x. 4 root root 162 Sep 1 10:23 /tmp

image

to

[root@TargetServer ~]# ls -ld /tmp drwxrwxrwt. 4 root root 162 Sep 1 10:23 /tmp

This is the root cause been found.

image

MrPippin66 commented 1 year ago

Please close out this issue, if you feel your problem has been resolved on your side.

ericloyd commented 1 year ago

Nice catch, @ongsunglau!

On the server with the agent, can you "su" to 'nagios' id and verify you can write files to /tmp?