qk4l / zabbix-cachet

Python script which sync Zabbix IT Services with Cachet
MIT License
80 stars 32 forks source link

Error with embedding 'soft' dependencies #54

Open DonRichards opened 4 years ago

DonRichards commented 4 years ago

EDIT: Just noticed this causes an error. So I'm switching this from a "how to" to a "how to replicate error exactly"

$ systemctl status zabbix-cachethq.service
....
● zabbix-cachethq.service - Zabbix to Cachet Sync Service
   Loaded: loaded (/usr/lib/systemd/system/zabbix-cachethq.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2020-01-28 07:16:16 EST; 7h ago
 Main PID: 103293 (python3)
   CGroup: /system.slice/zabbix-cachethq.service
           └─103293 /PATH/.local/share/virtualenvs/zabbix-cachet-d3Qr61UG/bin/python3 /PATH/zabbix-cachet/zabbix-cachet.py

Jan 28 14:30:19 peacock zabbix-cachethq[103293]: 2020-01-28 14:30:19 EST ERROR: (Trigger Watcher) ClientHttpError[https://status.utk.edu/api/v1/incidents, 500: {"errors":[{"id":"f8b10c85-9031-48fd-b107-b9bb4598888d","status":500,"title":"Internal Server Error","detail":"An error has occurred and this resource cannot be displayed."}]}]
Jan 28 14:30:19 peacock zabbix-cachethq[103293]: 2020-01-28 14:30:19 EST ERROR: (Trigger Watcher) triggers_watcher() raised an Exception. Something gone wrong
Jan 28 14:30:19 peacock zabbix-cachethq[103293]: 2020-01-28 14:30:19 EST ERROR: (Trigger Watcher) 'NoneType' object is not subscriptable
Jan 28 14:30:19 peacock zabbix-cachethq[103293]: Traceback (most recent call last):
Jan 28 14:30:19 peacock zabbix-cachethq[103293]: File "/PATH/zabbix-cachet/zabbix-cachet.py", line 611, in triggers_watcher_worker
Jan 28 14:30:19 peacock zabbix-cachethq[103293]: triggers_watcher(service_map)
Jan 28 14:30:19 peacock zabbix-cachethq[103293]: File "/PATH/zabbix-cachet/zabbix-cachet.py", line 580, in triggers_watcher
Jan 28 14:30:19 peacock zabbix-cachethq[103293]: component_id=i['component_id'], component_status=comp_status)
Jan 28 14:30:19 peacock zabbix-cachethq[103293]: File "/PATH/zabbix-cachet/zabbix-cachet.py", line 444, in new_incidents
Jan 28 14:30:19 peacock zabbix-cachethq[103293]: incident_id=data['data']['id'],
Jan 28 14:30:19 peacock zabbix-cachethq[103293]: TypeError: 'NoneType' object is not subscriptable

I want to only show Parents + Children but not grandchildren (individual triggers/services) on the frontend page.

Turns out to be a little complicated but I hope this might help others attempting to not disclose too much information about their site on the status page.

Create the grouping you need in Zabbix Services first

Create groups of services with their triggers.

├──  ServerA HTTP/HTTPS Services
│   ├── ServerA: HTTP service is down
│   └── ServerA: HTTPs service is down
└──  ServerA Support Services
    ├── ServerA: I/O overloaded
    ├── ServerA:  Lack of available memory
    └── ServerA: MySQL is down

Set them to soft

Service Dependencies soft trigger image

Create a alternative to root for zabbix-cachet to look at

From within Zabbix Create a new service with a name and NO triggers. The example in this image I used 'cachethq'. Name it anything you want. Create new root

Modify the config.yml file

Change it from a blank string to the name you created in the previous step.

settings:
  root_service: 'cachethq'

Remove all of the components from cachet

Before you start I would stop the zabbix-cachet service first. Navigate to your Cachethq site's dashboard/components and remove everything (all those items that are normally imported. Otherwise the low level services that you want to hide will continue to appear. Remove the component groups as well and restart the zabbix-cachet service.

It should not do anything yet but if it creates a component on the main status page of the blank component then you might need to change the root_service: ''cachethq" string to the number value of the zabbix service root_service: ''42".

Create the Children you want to show on the site.

Create a child with no triggers and the dependencies should only be the parents (not the child triggers) from root with soft checked for the services. Nothing should show up as a trigger on this page. Properly set children

Once complete all that should show under cachethq is parents and children but not triggers.

Screen Shot 2020-01-24 at 11 25 50 AM

This should omit the children but still report the status. I've modified my component.blade.php file so the final info looks a bit different but I'm hoping it's obvious how this limits the information being disclosed. Frontend View

This is a lot of screenshots. Apologies. I'd like to see something like this in the documentation or a simplified process for this.

DonRichards commented 4 years ago

@qk4l Just to recap/replicate. If you use Zabbix's the build in functionality of soft dependencies, zabbix-cachet breaks when "Trigger Watcher" is fired.

(Trigger Watcher) start trigger watcher
(Trigger Watcher) start trigger watcher
(Trigger Watcher) ClientHttpError[https://status.utk.edu/api/v1/incidents, 500: {"errors":[{"id":"1111-1111-1111-1111-1111111111","status":500,"title":"Internal Server Error","detail":"An error has occurred and this resource cannot be displayed."}]}]
# I replaced the ID with ones (1111) 

2020-03-16 10:02:08 EDT ERROR: (Trigger Watcher) triggers_watcher() raised an Exception. Something gone wrong
2020-03-16 10:02:08 EDT ERROR: (Trigger Watcher) 'NoneType' object is not subscriptable
Traceback (most recent call last):
File "/zabbix-cachet/zabbix-cachet.py", line 611, in triggers_watcher_worker
triggers_watcher(service_map)
File "/zabbix-cachet/zabbix-cachet.py", line 580, in triggers_watcher
component_id=i['component_id'], component_status=comp_status)
File "/zabbix-cachet/zabbix-cachet.py", line 444, in new_incidents
incident_id=data['data']['id'],
TypeError: 'NoneType' object is not subscriptable
DonRichards commented 4 years ago

https://status.utk.edu/api/v1/incidents comes up with no issues in the browser. It returns a standard json document.

{
  "meta": {
    "pagination": {
      "total": 209,
      "count": 20,
      "per_page": 20,
      "current_page": 1,
      "total_pages": 11,
      "links": {
        "next_page": "https://status.utk.edu/api/v1/incidents?page=2",
        "previous_page": null
      }
    }
  },
  "data": [
    {
      "id": 4,
      "component_id": 71,
      "name": "StagingServer | Agent is unreachable for 10 minutes",
      "status": 1,
      "visible": 1,
      "message": "StagingServer |  Agent check **failed** - Jul 20, 15:42\r\n\r\n```System's Status is unreachable for 10 minutes```",
      "scheduled_at": "2020-03-16 10:43:39",
      "created_at": "2018-07-20 15:47:00",
      "updated_at": "2018-07-20 15:47:00",
      "deleted_at": null,
      "human_status": "Investigating"
    },
    {
      "id": 5,
      "component_id": 72,
      "name": "StagingServer | Agent is unreachable for 10 minutes",
      "status": 4,
      "visible": 1,
      "message": "__Resolved__ - Jul 20, 15:50\r\n\r\n______\r\nStagingServer |  Agent check **failed** - Jul 20, 15:42\r\n\r\n```System's Status is unreachable for 10 minutes```",
      "scheduled_at": "2020-03-16 10:43:39",
      "created_at": "2018-07-20 15:48:00",
      "updated_at": "2018-07-20 15:48:00",
      "deleted_at": null,
      "human_status": "Fixed"
    },
    {
      "id": 6,
      "component_id": 53,
      "name": "Forms.Site | Zabbix agent on esb is unreachable for 10 minutes",
      "status": 4,
      "visible": 1,
      "message": "__Resolved__ - Jul 30, 02:24\n\n______\nForms.Site | 2nd Check check **failed** - Jul 30, 02:05\n\n```Zabbix agent on esb is unreachable for 10 minutes```",
      "scheduled_at": "2020-03-16 10:43:39",
      "created_at": "2018-07-30 02:22:25",
      "updated_at": "2018-07-30 02:24:32",
      "deleted_at": null,
      "human_status": "Fixed"
    },
    {
      "id": 7,
      "component_id": 64,
      "name": "Main site (www) | www: apache down",
      "status": 4,
      "visible": 1,
      "message": "__Resolved__ - Aug 06, 07:59\n\n______\nMain site (www) | www apache check **failed** - Aug 06, 07:56\n\n```www: apache down```",
      "scheduled_at": "2020-03-16 10:43:39",
      "created_at": "2018-08-06 07:57:27",
      "updated_at": "2018-08-06 07:59:31",
      "deleted_at": null,
      "human_status": "Fixed"
    },
    {
      "id": 8,
      "component_id": 65,
      "name": "Main site (www) | HTTPS service is down on www",
      "status": 4,
      "visible": 1,
      "message": "__Resolved__ - Aug 06, 07:59\n\n______\nMain site (www) | www https check **failed** - Aug 06, 07:56\n\n```HTTPS service is down on www```",
      "scheduled_at": "2020-03-16 10:43:39",
      "created_at": "2018-08-06 07:57:27",
      "updated_at": "2018-08-06 07:59:32",
      "deleted_at": null,
      "human_status": "Fixed"
    },
    {
      "id": 9,
      "component_id": 66,
      "name": "Main site (www) | HTTP service is down on www",
      "status": 4,
      "visible": 1,
      "message": "__Resolved__ - Aug 06, 07:59\n\n______\nMain site (www) | www http check **failed** - Aug 06, 07:56\n\n```HTTP service is down on www```",
      "scheduled_at": "2020-03-16 10:43:39",
      "created_at": "2018-08-06 07:57:28",
      "updated_at": "2018-08-06 07:59:32",
      "deleted_at": null,
      "human_status": "Fixed"
    },
    {
      "id": 12,
      "component_id": 66,
      "name": "Main site (www) | HTTP service is down on www",
      "status": 4,
      "visible": 1,
      "message": "__Resolved__ - Aug 07, 07:29\n\n______\nMain site (www) | www http check **failed** - Aug 07, 07:25\n\n```HTTP service is down on www```",
      "scheduled_at": "2020-03-16 10:43:39",
      "created_at": "2018-08-07 07:25:11",
      "updated_at": "2018-08-07 07:29:19",
      "deleted_at": null,
      "human_status": "Fixed"
    },
    {
      "id": 22,
      "component_id": 85,
      "name": "Streaming Service | Zabbix agent on streamer is unreachable for 10 minutes",
      "status": 4,
      "visible": 1,
      "message": "__Resolved__ - Oct 05, 12:31\n\n______\nStreaming Service | Streaming Server services have been unreachable for 10 minutes check **failed** - Oct 05, 12:15\n\n```Zabbix agent on streamer is unreachable for 10 minutes```",
      "scheduled_at": "2020-03-16 10:43:39",
      "created_at": "2018-10-05 12:31:27",
      "updated_at": "2018-10-05 12:31:27",
      "deleted_at": null,
      "human_status": "Fixed"
    },
    {
      "id": 23,
      "component_id": 85,
      "name": "Streaming Service | Zabbix agent on streamer is unreachable for 10 minutes",
      "status": 4,
      "visible": 1,
      "message": "__Resolved__ - Oct 05, 12:33\n\n______\nStreaming Service | Streaming Server services have been unreachable for 10 minutes check **failed** - Oct 05, 12:15\n\n```Zabbix agent on streamer is unreachable for 10 minutes```",
      "scheduled_at": "2020-03-16 10:43:39",
      "created_at": "2018-10-05 12:33:31",
      "updated_at": "2018-10-05 12:33:32",
      "deleted_at": null,
      "human_status": "Fixed"
    },
    {
      "id": 24,
      "component_id": 85,
      "name": "Streaming Service | Zabbix agent on streamer is unreachable for 10 minutes",
      "status": 4,
      "visible": 1,
      "message": "__Resolved__ - Oct 05, 12:35\n\n______\nStreaming Service | Streaming Server services have been unreachable for 10 minutes check **failed** - Oct 05, 12:15\n\n```Zabbix agent on streamer is unreachable for 10 minutes```",
      "scheduled_at": "2020-03-16 10:43:39",
      "created_at": "2018-10-05 12:35:36",
      "updated_at": "2018-10-05 12:35:37",
      "deleted_at": null,
      "human_status": "Fixed"
    },
    {
      "id": 25,
      "component_id": 85,
      "name": "Streaming Service | Zabbix agent on streamer is unreachable for 10 minutes",
      "status": 4,
      "visible": 1,
      "message": "__Resolved__ - Oct 05, 12:37\n\n______\nStreaming Service | Streaming Server services have been unreachable for 10 minutes check **failed** - Oct 05, 12:15\n\n```Zabbix agent on streamer is unreachable for 10 minutes```",
      "scheduled_at": "2020-03-16 10:43:39",
      "created_at": "2018-10-05 12:37:41",
      "updated_at": "2018-10-05 12:37:42",
      "deleted_at": null,
      "human_status": "Fixed"
    },
    {
      "id": 26,
      "component_id": 85,
      "name": "Streaming Service | Zabbix agent on streamer is unreachable for 10 minutes",
      "status": 4,
      "visible": 1,
      "message": "__Resolved__ - Oct 05, 12:39\n\n______\nStreaming Service | Streaming Server services have been unreachable for 10 minutes check **failed** - Oct 05, 12:15\n\n```Zabbix agent on streamer is unreachable for 10 minutes```",
      "scheduled_at": "2020-03-16 10:43:39",
      "created_at": "2018-10-05 12:39:46",
      "updated_at": "2018-10-05 12:39:47",
      "deleted_at": null,
      "human_status": "Fixed"
    },
    {
      "id": 27,
      "component_id": 85,
      "name": "Streaming Service | Zabbix agent on streamer is unreachable for 10 minutes",
      "status": 4,
      "visible": 1,
      "message": "__Resolved__ - Oct 05, 12:41\n\n______\nStreaming Service | Streaming Server services have been unreachable for 10 minutes check **failed** - Oct 05, 12:15\n\n```Zabbix agent on streamer is unreachable for 10 minutes```",
      "scheduled_at": "2020-03-16 10:43:39",
      "created_at": "2018-10-05 12:41:51",
      "updated_at": "2018-10-05 12:41:52",
      "deleted_at": null,
      "human_status": "Fixed"
    },
    {
      "id": 28,
      "component_id": 85,
      "name": "Streaming Service | Zabbix agent on streamer is unreachable for 10 minutes",
      "status": 4,
      "visible": 1,
      "message": "__Resolved__ - Oct 05, 12:43\n\n______\nStreaming Service | Streaming Server services have been unreachable for 10 minutes check **failed** - Oct 05, 12:15\n\n```Zabbix agent on streamer is unreachable for 10 minutes```",
      "scheduled_at": "2020-03-16 10:43:39",
      "created_at": "2018-10-05 12:43:56",
      "updated_at": "2018-10-05 12:43:57",
      "deleted_at": null,
      "human_status": "Fixed"
    },
    {
      "id": 29,
      "component_id": 85,
      "name": "Streaming Service | Zabbix agent on streamer is unreachable for 10 minutes",
      "status": 4,
      "visible": 1,
      "message": "__Resolved__ - Oct 05, 12:46\n\n______\nStreaming Service | Streaming Server services have been unreachable for 10 minutes check **failed** - Oct 05, 12:15\n\n```Zabbix agent on streamer is unreachable for 10 minutes```",
      "scheduled_at": "2020-03-16 10:43:39",
      "created_at": "2018-10-05 12:46:01",
      "updated_at": "2018-10-05 12:46:02",
      "deleted_at": null,
      "human_status": "Fixed"
    },
    {
      "id": 30,
      "component_id": 85,
      "name": "Streaming Service | Zabbix agent on streamer is unreachable for 10 minutes",
      "status": 4,
      "visible": 1,
      "message": "__Resolved__ - Oct 05, 12:48\n\n______\nStreaming Service | Streaming Server services have been unreachable for 10 minutes check **failed** - Oct 05, 12:15\n\n```Zabbix agent on streamer is unreachable for 10 minutes```",
      "scheduled_at": "2020-03-16 10:43:39",
      "created_at": "2018-10-05 12:48:06",
      "updated_at": "2018-10-05 12:48:07",
      "deleted_at": null,
      "human_status": "Fixed"
    },
    {
      "id": 31,
      "component_id": 85,
      "name": "Streaming Service | Zabbix agent on streamer is unreachable for 10 minutes",
      "status": 4,
      "visible": 1,
      "message": "__Resolved__ - Oct 05, 12:50\n\n______\nStreaming Service | Streaming Server services have been unreachable for 10 minutes check **failed** - Oct 05, 12:15\n\n```Zabbix agent on streamer is unreachable for 10 minutes```",
      "scheduled_at": "2020-03-16 10:43:39",
      "created_at": "2018-10-05 12:50:10",
      "updated_at": "2018-10-05 12:50:12",
      "deleted_at": null,
      "human_status": "Fixed"
    },
    {
      "id": 32,
      "component_id": 85,
      "name": "Streaming Service | Zabbix agent on streamer is unreachable for 10 minutes",
      "status": 4,
      "visible": 1,
      "message": "__Resolved__ - Oct 05, 12:52\n\n______\nStreaming Service | Streaming Server services have been unreachable for 10 minutes check **failed** - Oct 05, 12:15\n\n```Zabbix agent on streamer is unreachable for 10 minutes```",
      "scheduled_at": "2020-03-16 10:43:39",
      "created_at": "2018-10-05 12:52:15",
      "updated_at": "2018-10-05 12:52:16",
      "deleted_at": null,
      "human_status": "Fixed"
    },
    {
      "id": 33,
      "component_id": 85,
      "name": "Streaming Service | Zabbix agent on streamer is unreachable for 10 minutes",
      "status": 4,
      "visible": 1,
      "message": "__Resolved__ - Oct 05, 12:54\n\n______\nStreaming Service | Streaming Server services have been unreachable for 10 minutes check **failed** - Oct 05, 12:15\n\n```Zabbix agent on streamer is unreachable for 10 minutes```",
      "scheduled_at": "2020-03-16 10:43:39",
      "created_at": "2018-10-05 12:54:20",
      "updated_at": "2018-10-05 12:54:21",
      "deleted_at": null,
      "human_status": "Fixed"
    },
    {
      "id": 34,
      "component_id": 85,
      "name": "Streaming Service | Zabbix agent on streamer is unreachable for 10 minutes",
      "status": 4,
      "visible": 1,
      "message": "__Resolved__ - Oct 05, 12:56\n\n______\nStreaming Service | Streaming Server services have been unreachable for 10 minutes check **failed** - Oct 05, 12:15\n\n```Zabbix agent on streamer is unreachable for 10 minutes```",
      "scheduled_at": "2020-03-16 10:43:39",
      "created_at": "2018-10-05 12:56:25",
      "updated_at": "2018-10-05 12:56:26",
      "deleted_at": null,
      "human_status": "Fixed"
    }
  ]
}