Catch permission errors with statuspollers

jriberg commented 6 years ago

Crashes after installing from icinga repo, configuring it and running "systemctl start icingabeat".

sudo /usr/bin/icingabeat
1m0s[][]panic: interface conversion: interface {} is float64, not []interface {}

goroutine 26 [running]:
github.com/icinga/icingabeat/beater.BuildStatusEvents(0xc420018600, 0x2c, 0x600, 0x2c, 0x600, 0x0)
        /go/src/github.com/icinga/icingabeat/beater/statuspoller.go:46 +0xa0d
github.com/icinga/icingabeat/beater.(*Statuspoller).Run(0xc420106820, 0x0, 0x0)
        /go/src/github.com/icinga/icingabeat/beater/statuspoller.go:123 +0x4b7
created by github.com/icinga/icingabeat/beater.(*Icingabeat).Run
        /go/src/github.com/icinga/icingabeat/beater/icingabeat.go:54 +0x284

icingabeat:
  host: "watcher01.prd"
  port: 5665
  user: "icingabeat"
  password: "HIDDENSUPERPASSWORD"
  skip_ssl_verify: false
  eventstream:
    types:
      - CheckResult
      - StateChange
    filter: ""
    retry_interval: 10s
  statuspoller:
    interval: 60s
fields_under_root: true
fields:
  env: prd
  servicename: watcher01.prd
  subservice: icinga
output.logstash:
  enabled: true
  hosts: [ "log01.fqdn" ]
  worker: 4
  compression_level: 3
  loadbalance: true
  index: 'icingabeat'
logging.to_files: true
logging.files:
  path: /var/log/icingabeat
  name: icingabeat
  rotateeverybytes: 10485760 # = 10MB
  permissions: 0600

Expected Behavior

I expect it icingabeat to stay running and collect my data

Current Behavior

Currently it just dies after connecting to icinga2.

Possible Solution

Steps to Reproduce (for bugs)

Install icingabeat on Ubuntu16 from "officiall" repo.
Configure an API-user as described in https://devhub.io/repos/Icinga-icingabeat
Configure icingabeat.yml

Context

Your Environment

Beat version (icingabeat -version): admin@watcher01:/etc/icingabeat$ icingabeat --version icingabeat version 6.1.1 (amd64), libbeat 6.1.1
Icinga 2 version (icinga2 --version): admin@watcher01:/etc/icingabeat$ icinga2 --version icinga2 - The Icinga 2 network monitoring daemon (version: r2.8.2-1)

Copyright (c) 2012-2017 Icinga Development Team (https://www.icinga.com/) License GPLv2+: GNU GPL version 2 or later http://gnu.org/licenses/gpl2.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.

Application information: Installation root: /usr Sysconf directory: /etc Run directory: /run Local state directory: /var Package data directory: /usr/share/icinga2 State path: /var/lib/icinga2/icinga2.state Modified attributes path: /var/lib/icinga2/modified-attributes.conf Objects path: /var/cache/icinga2/icinga2.debug Vars path: /var/cache/icinga2/icinga2.vars PID path: /run/icinga2/icinga2.pid

System information: Platform: Ubuntu Platform version: 16.04.4 LTS (Xenial Xerus) Kernel: Linux Kernel version: 4.4.0-116-generic Architecture: x86_64

Build information: Compiler: GNU 5.3.1 Build host: 86927c12b6d8

Elasticsearch version (curl -XGET 'localhost:9200'): 6.2.2
Logstash version, if used (bin/logstash -V): 6.2.2
Kibana version, if used (curl -XGET http://localhost:5601/status -I): 6.2.2
Operating System and version: Ubuntu 16, Icinga-API-user have permissions = [ "events/CheckResult" ] Network interfacename: ens160

admin@watcher01:~# apparmor_status apparmor module is loaded. 5 profiles are loaded. 5 profiles are in enforce mode. /sbin/dhclient /usr/lib/NetworkManager/nm-dhcp-client.action /usr/lib/NetworkManager/nm-dhcp-helper /usr/lib/connman/scripts/dhclient-script /usr/sbin/mysqld 0 profiles are in complain mode. 1 processes have profiles defined. 1 processes are in enforce mode. /usr/sbin/mysqld (1074) 0 processes are in complain mode. 0 processes are unconfined but have a profile defined.

dnsmichi commented 6 years ago

Please also fill in the list underneath your environment.

bobapple commented 6 years ago

@joshiegy additionally to the environment information, can you please paste the output of your Icinga status information: curl -k -u user:password https://icinga2-master:5665/v1/status

bobapple commented 6 years ago

After some testing I'm not able to reproduce this. Still need some more information (see previous comment)

jriberg commented 6 years ago

curl -k -u user:passwd https://localhost:5665/v1/status {"error":404.0,"status":"No objects found."}

I tested with the icingabeat user whom have permissions = [ "events/CheckResult" ]

dnsmichi commented 6 years ago

Permissions are too strict, one needs more for the /v1/status endpoint: status/query.

https://www.icinga.com/docs/icinga2/latest/doc/12-icinga2-api/#permissions

The error then seems to originate from the error message returned by Icinga 2 and handled by Icingabeat.

jriberg commented 6 years ago

@dnsmichi - thank you for your quick response! We solved it by adding more permissions to icingabeat:

object ApiUser "icingabeat" { password = "HIDDENPASSWORD" permissions = [ "events/CheckResult", "status/query", "events/statechange", "events/notification", "events/AcknowledgementSet", "events/AcknowledgementCleared", "events/CommentAdded", "events/CommentRemoved", "events/DowntimeAdded", "events/DowntimeRemoved", "events/DowntimeStarted", "events/DowntimeTriggered" ] }

bobapple commented 6 years ago

We need better handling for permission errors in the status poller. I changed the subject.

Icinga / icingabeat