search

mocdaniel / dashing-icinga2

Dashing dashboard for Icinga 2 using the REST API
MIT License
204 stars 47 forks source link

[Bug] Receiving 401 errors after Installation #119

Closed stevethepirate88 closed 3 years ago

stevethepirate88 commented 3 years ago

After installing and starting the service on a Debian 10 VPS, I am seeing blank fields in the dashboard. Upon investigation of /usr/share/dashing-icinga2/log/thin.log it appears the culprit is 401 errors.

scheduler caught exception:
401 Unauthorized
/var/lib/gems/2.5.0/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:249:in `exception_with_response'
/var/lib/gems/2.5.0/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:129:in `return!'
/var/lib/gems/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:836:in `process_result'
/var/lib/gems/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:743:in `block in transmit'
/usr/lib/ruby/2.5.0/net/http.rb:910:in `start'
/var/lib/gems/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:727:in `transmit'
/var/lib/gems/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:163:in `execute'
/var/lib/gems/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:63:in `execute'
/var/lib/gems/2.5.0/gems/rest-client-2.1.0/lib/restclient/resource.rb:51:in `get'
/usr/share/dashing-icinga2/lib/icinga2.rb:258:in `getApiData'
/usr/share/dashing-icinga2/lib/icinga2.rb:282:in `getIcingaApplicationData'
/usr/share/dashing-icinga2/lib/icinga2.rb:774:in `run'
/usr/share/dashing-icinga2/jobs/icinga2.rb:27:in `block in <top (required)>'
/var/lib/gems/2.5.0/gems/rufus-scheduler-2.0.24/lib/rufus/sc/jobs.rb:230:in `trigger_block'
/var/lib/gems/2.5.0/gems/rufus-scheduler-2.0.24/lib/rufus/sc/jobs.rb:204:in `block in trigger'
/var/lib/gems/2.5.0/gems/rufus-scheduler-2.0.24/lib/rufus/sc/scheduler.rb:430:in `block in trigger_job'

Expected Behavior

The fields should populate

Current Behavior

The fields are not populating

Context

I'm wondering if this is an issue with how I've installed ruby? I performed everything in the installation instructions as a limited user, only using sudo when prompted, but I think I might have installed ruby as root? What permissions should be set for the /var/lib/gems/ folder?

Your Environment

mocdaniel commented 3 years ago

401 Unauthorized points at wrong API credentials. Could you please double-check that the credentials you configured for dashing-icinga2 are authorized to connect to your Icinga2 master?

stevethepirate88 commented 3 years ago

So this is what I'm seeing:

sudo cat /etc/icinga2/conf.d/api-users.conf
 [...]
object ApiUser "dashing" {
  password = "ScraneDevIcinga21DashingPleaseWork"
  permissions = [ "status/query", "objects/query/*" ]
}

And then

 sudo cat config/icinga2.local.json
{
  "icinga2": {
    "api": {
      "host": "localhost",
      "port": 5665,
      "user": "dashing",
      "password": "ScraneDevIcinga21DashingPleaseWork"
    }
  },
  "icingaweb2": {
    "url": "http://localhost/icingaweb2"
  },
  "dashboard": {
    "show_only_hard_state_problems": false,
    "timezone": "UTC"
  }
}

Is there something else I am missing with this to properly add the user?

mocdaniel commented 3 years ago

Looks fine to me... Did you enable the api module of your Icinga2 master and restart the icinga2.service after adding the APIUser object?

stevethepirate88 commented 3 years ago

Yeah, it's all enabled and I restarted things.

sudo icinga2 feature enable api
warning/cli: Feature 'api' already enabled.
sudo systemctl restart icinga2
curl -k -s -u dashing:icinga 'https://localhost:5665/v1'
<h1>Unauthorized. Please check your user credentials.</h1

I tried disabling API, restarting Icinga, enabling the API, restarting Icinga, same deal.

mocdaniel commented 3 years ago

The latest output definitely points at a problem with your Icinga2 setup rather than your dashing-icinga2 setup. I can assure you that the way you installed Ruby etc. was correct since you could start the dashing-icinga2 server. Problems arise from your API connectivity. Please head over to Icinga2's community forum and try solving your API problems with their help. They know their way around Icinga2 much better than I do :)

elguenny commented 3 years ago

I am facing the same error with the same setup, but my api is definitely working:

curl -k -s -u dashing:xyz 'https://localhost:5665/v1'
<html><head><title>Icinga 2</title></head><h1>Hello from Icinga 2 (Version: r2.12.4-1)!</h1><p>You are authenticated as <b>dashing</b>. Your user has the following permissions:</p> <ul><li>status/query</li><li>objects/query/*</li></ul><p>More information about API requests is available in the <a href="https://icinga.com/docs/icinga2/latest/" target="_blank">documentation</a>.</p></html>: /usr/share/dashing-icinga2/ $

If I start the service foreground it works for me, thats really strange ICINGA2_API_HOST=localhost ICINGA2_API_PORT=5665 ICINGA2_API_USERNAME=dashing ICINGA2_API_PASSWORD=xyz dashing start -p 8005


Doing `require 'backports'` is deprecated and will not load any backport in the next major release.
Require just the needed backports instead, or 'backports/latest'.
/var/lib/gems/2.5.0/gems/rufus-scheduler-2.0.24/lib/rufus/sc/jobs.rb:368: warning: constant ::Fixnum is deprecated
First trying to read environment variables
Using environment variable configuration on 'localhost:5665'.
Thin web server (v1.6.4 codename Gob Bluth)
Maximum connections set to 1024
Listening on 0.0.0.0:8005, CTRL+C to stop
First trying to read environment variables
Using environment variable configuration on 'localhost:5665'.
First trying to read environment variables
Using environment variable configuration on 'localhost:5665'.
First trying to read environment variables
Using environment variable configuration on 'localhost:5665'.
First trying to read environment variables
Using environment variable configuration on 'localhost:5665'.
10.50.24.11 - - [23/Jun/2021 11:44:17] "GET /icinga2 HTTP/1.1" 200 7193 0.0041
10.50.24.11 - - [23/Jun/2021 11:44:19] "GET /views/clock.html HTTP/1.1" 200 82 0.0007
10.50.24.11 - - [23/Jun/2021 11:44:19] "GET /views/list.html HTTP/1.1" 200 613 0.0005
10.50.24.11 - - [23/Jun/2021 11:44:19] "GET /views/meter.html HTTP/1.1" 200 375 0.0007```
mocdaniel commented 3 years ago

Thank you for confirming this issue. I will look into it.

mocdaniel commented 3 years ago

I can't wrap my head around this issue right now... I quickly set up a Debian VM with the following versions:

, following the installation and configuration instructions from the Readme.md and everything worked just fine - dashing from foreground, dashing detached as well as dashing via systemctl.

@stevethepirate88 can you confirm that your configured api-user is being registered with Icinga2 by executing icinga2 object list --type apiuser, please, since not even curl is able to connect to your Icinga API? You might have disabled inclusion of icinga2/conf.d/api-users.conf accidentally.

@elguenny since your API seems to behave differently for curl and dashing-icinga (and even dashing being detached or not), may I ask you to capture traffic via Wireshark or a similar tool? Insight into the GET-requests sent from detached dashing vs. foreground dashing might be helpful.

stevethepirate88 commented 3 years ago

Sure thing! Here's the output for me:

sudo icinga2 object list --type apiuser
Object 'root' of type 'ApiUser':
  % declared in '/etc/icinga2/conf.d/api-users.conf', lines 4:1-4:21
  * __name = "root"
  * client_cn = ""
  * name = "root"
  * package = "_etc"
  * password_hash = ""
  * permissions = [ "*" ]
    % = modified in '/etc/icinga2/conf.d/api-users.conf', lines 8:3-8:23
  * source_location
    * first_column = 1
    * first_line = 4
    * last_column = 21
    * last_line = 4
    * path = "/etc/icinga2/conf.d/api-users.conf"
  * templates = [ "root" ]
    % = modified in '/etc/icinga2/conf.d/api-users.conf', lines 4:1-4:21
  * type = "ApiUser"
  * zone = ""

Object 'icingaweb2' of type 'ApiUser':
  % declared in '/etc/icinga2/conf.d/api-users.conf', lines 10:1-10:27
  * __name = "icingaweb2"
  * client_cn = ""
  * name = "icingaweb2"
  * package = "_etc"
  * password_hash = ""
  * permissions = [ "status/query", "actions/*", "objects/modify/*", "objects/query/*" ]
    % = modified in '/etc/icinga2/conf.d/api-users.conf', lines 12:2-12:85
  * source_location
    * first_column = 1
    * first_line = 10
    * last_column = 27
    * last_line = 10
    * path = "/etc/icinga2/conf.d/api-users.conf"
  * templates = [ "icingaweb2" ]
    % = modified in '/etc/icinga2/conf.d/api-users.conf', lines 10:1-10:27
  * type = "ApiUser"
  * zone = ""

Object 'dashing' of type 'ApiUser':
  % declared in '/etc/icinga2/conf.d/api-users.conf', lines 14:1-14:24
  * __name = "dashing"
  * client_cn = ""
  * name = "dashing"
  * package = "_etc"
  * password_hash = ""
  * permissions = [ "status/query", "objects/query/*" ]
    % = modified in '/etc/icinga2/conf.d/api-users.conf', lines 16:3-16:53
  * source_location
    * first_column = 1
    * first_line = 14
    * last_column = 24
    * last_line = 14
    * path = "/etc/icinga2/conf.d/api-users.conf"
  * templates = [ "dashing" ]
    % = modified in '/etc/icinga2/conf.d/api-users.conf', lines 14:1-14:24
  * type = "ApiUser"
  * zone = ""
elguenny commented 3 years ago

I can't wrap my head around this issue right now... I quickly set up a Debian VM with the following versions:

* **Debian**: 10 (Buster)

* **Ruby**: 2.5.5p157 (2019-03-15 revision 67260) [x86_64-linux-gnu]

* **Icinga**: r2.12.4-1

, following the installation and configuration instructions from the Readme.md and everything worked just fine - dashing from foreground, dashing detached as well as dashing via systemctl.

@elguenny since your API seems to behave differently for curl and dashing-icinga (and even dashing being detached or not), may I ask you to capture traffic via Wireshark or a similar tool? Insight into the GET-requests sent from detached dashing vs. foreground dashing might be helpful.

My Testsystem is working now with systemctl and foreground command, but the error log remains. Maybe I forgot to restart/reboot and icinga api settings were not set, I really can't find my fault I also rerun bundle and bundler and updated node.js to 14.x The Dashboard looks not the same as the screenshot above, but there are some metrics shown.

127.0.0.1 - - [24/Jun/2021 14:24:23] "GET /icinga2 HTTP/1.0" 200 7236 0.0063
127.0.0.1 - - [24/Jun/2021 14:24:23] "GET /views/clock.html HTTP/1.0" 200 82 0.0007
127.0.0.1 - - [24/Jun/2021 14:24:23] "GET /views/list.html HTTP/1.0" 200 613 0.0004
127.0.0.1 - - [24/Jun/2021 14:24:23] "GET /views/meter.html HTTP/1.0" 200 375 0.0004
================================================================================
scheduler caught exception:
401 Unauthorized
/var/lib/gems/2.5.0/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:249:in `exception_with_response'
/var/lib/gems/2.5.0/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:129:in `return!'
/var/lib/gems/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:836:in `process_result'
/var/lib/gems/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:743:in `block in transmit'
/usr/lib/ruby/2.5.0/net/http.rb:910:in `start'
/var/lib/gems/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:727:in `transmit'
/var/lib/gems/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:163:in `execute'
/var/lib/gems/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:63:in `execute'
/var/lib/gems/2.5.0/gems/rest-client-2.1.0/lib/restclient/resource.rb:51:in `get'
/usr/share/dashing-icinga2/lib/icinga2.rb:258:in `getApiData'
/usr/share/dashing-icinga2/lib/icinga2.rb:282:in `getIcingaApplicationData'
/usr/share/dashing-icinga2/lib/icinga2.rb:789:in `run'
/usr/share/dashing-icinga2/jobs/icinga2.rb:27:in `block in <top (required)>'
/var/lib/gems/2.5.0/gems/rufus-scheduler-2.0.24/lib/rufus/sc/jobs.rb:230:in `trigger_block'
/var/lib/gems/2.5.0/gems/rufus-scheduler-2.0.24/lib/rufus/sc/jobs.rb:204:in `block in trigger'
/var/lib/gems/2.5.0/gems/rufus-scheduler-2.0.24/lib/rufus/sc/scheduler.rb:430:in `block in trigger_job'
================================================================================
mocdaniel commented 3 years ago

Just clarifying: You're seeing correct, real-time metrics now when running dashing via systemctl or in foreground, but you still observe 401 Unauthorized errors in your thin.log? The dashboard not looking the same as in the screenshot is intended, the default dashboard currently shipped is much simpler than it used to be.

elguenny commented 3 years ago

You're right, I double checked the metrics in the dashboard and they are frozen.

mocdaniel commented 3 years ago

I really can't pinpoint this. Your API seems to be set up correctly, either Icinga2 or curl would've complained otherwise. We're all using the same setup, also regarding involved gems such as rest-client v2.1.0, yet we observe 3 distinctive behaviours...

Thing is, even if values in your dashboard seem to be frozen again, their existence means that at some point the API queries were getting through to Icinga. Still, some packet sniffing might bring up something useful, but since I can't reproduce the 401 errors you would have to provide those.

elguenny commented 3 years ago

After further investigations, I found a typo in the api auth, the 401er is gone now. Sorry for wasting your time

Log form the Master

[2021-06-24 17:38:32 +0200] information/HttpServerConnection: HTTP client disconnected (from [127.0.0.1]:37612)
[2021-06-24 17:38:32 +0200] information/ApiListener: New client connection from [127.0.0.1]:37614 (no client certificate)
[2021-06-24 17:38:32 +0200] information/HttpServerConnection: Request: GET /v1/status/CIB (from [127.0.0.1]:37614), user: dashing, agent: rest-client/2.1.0 (linux-gnu x86_64) ruby/2.5.5p157, status: OK).
[2021-06-24 17:38:32 +0200] information/HttpServerConnection: HTTP client disconnected (from [127.0.0.1]:37614)

The dashboard still doesn't show any "down" Client or Service , but the log shows another error.

scheduler caught exception:
no implicit conversion of nil into String
/usr/share/dashing-icinga2/lib/icinga2.rb:647:in `+'
/usr/share/dashing-icinga2/lib/icinga2.rb:647:in `block in getProblemServices'
/usr/share/dashing-icinga2/lib/icinga2.rb:633:in `each'
/usr/share/dashing-icinga2/lib/icinga2.rb:633:in `getProblemServices'
/usr/share/dashing-icinga2/lib/icinga2.rb:854:in `run'
/usr/share/dashing-icinga2/jobs/icinga2.rb:27:in `block in <top (required)>'
/var/lib/gems/2.5.0/gems/rufus-scheduler-2.0.24/lib/rufus/sc/jobs.rb:230:in `trigger_block'
/var/lib/gems/2.5.0/gems/rufus-scheduler-2.0.24/lib/rufus/sc/jobs.rb:204:in `block in trigger'
/var/lib/gems/2.5.0/gems/rufus-scheduler-2.0.24/lib/rufus/sc/scheduler.rb:430:in `block in trigger_job'
mocdaniel commented 3 years ago

So the general connection to the API is working now that the typo is resolved? The other error seems to point at data expected from the API which is not provided, I will look into it as well.

elguenny commented 3 years ago

Yes the general connection is working now. But down hosts are not shown in the frontend down-problem

mocdaniel commented 3 years ago

I was able to replicate this behaviour and opened a new bug. I will close this issue for now since I was not able to replicate described behaviour for @stevethepirate88 and @elguenny's problem turned out to be a typo.