search

Icinga / icinga2

The core of our monitoring platform with a powerful configuration language and REST API.
https://icinga.com/docs/icinga2/latest
GNU General Public License v2.0
2.03k stars 578 forks source link

v2.11.0-rc1 Windows Agent does not create debug.log #7388

Closed R-Sommer closed 5 years ago

R-Sommer commented 5 years ago

Describe the bug

Enabling the feature debuglog does not create debug.log file.

To Reproduce

  1. C:\Program Files\ICINGA2\sbin\icinga2.exe feature enable debuglog
  2. net stop icinga2
  3. net start icinga2
  4. C:\Program Files\ICINGA2\sbin\icinga2.exe feature list Disabled features: api elasticsearch gelf graphite influxdb notification opentsdb perfdata Enabled features: checker debuglog mainlog

Expected behavior

C:\ProgramData\icinga2\var\log\icinga2\debug.log shall be created

Your Environment

Include as many relevant details about the environment you experienced the problem in

dnsmichi commented 5 years ago

I've just tested this with the rc1 package on Windows.

C:\Program Files\ICINGA2\sbin> .\icinga2.exe --version
icinga2.exe - The Icinga 2 network monitoring daemon (version: v2.11.0-rc1-1-g2b891fd1e)

It took ~1 minute up until the debug log cache was flushed. Then it is written and visible on disk.

Screen Shot 2019-08-02 at 12 49 27

Can you verify that in your environment?

R-Sommer commented 5 years ago

Restart was at 10:40 and still no debug.log. Did another service restart just now, still no file after 1 minute.

R-Sommer commented 5 years ago

Even a reboot does not help:

[2019-08-02 13:33:32 +0200] information/Application: Received request to shut down.
[2019-08-02 13:33:32 +0200] information/Application: Shutting down...
[2019-08-02 13:33:32 +0200] information/ApiListener: 'api' stopped.
[2019-08-02 13:34:06 +0200] information/FileLogger: 'main-log' started.
[2019-08-02 13:34:06 +0200] information/ApiListener: 'api' started.
[2019-08-02 13:34:06 +0200] information/ApiListener: Started new listener on '[::]:5665'
[2019-08-02 13:34:06 +0200] information/ConfigItem: Activated all objects.
[2019-08-02 13:34:06 +0200] information/ApiListener: Reconnecting to endpoint 'main.em.lan' via host '192.168.53.210' and port '5665'

BTW: feature api disabled but started. Shall I create another issue here?

dnsmichi commented 5 years ago

Is the api feature disabled? To me this sounds as if features-enabled still contains the api.conf file. Or the service wasn't really stopped/started.

Are you running the icinga2 service with an account with limited permissions or restricted by group policies? That would maybe explain why the debug log is not created, likewise feature enable/disable doesn't work.

R-Sommer commented 5 years ago

The api feature is disabled (as noted in my inital post), but let's have a look:

C:\ProgramData\icinga2\etc\icinga2\features-enabled>dir
 Volume in drive C has no label.
 Volume Serial Number is 9412-DFB8

 Directory of C:\ProgramData\icinga2\etc\icinga2\features-enabled

08/02/2019  10:37 AM    <DIR>          .
08/02/2019  10:37 AM    <DIR>          ..
07/25/2019  10:49 AM                46 checker.conf
08/02/2019  10:37 AM                47 debuglog.conf
07/25/2019  10:49 AM                46 mainlog.conf
               3 File(s)            139 bytes
               2 Dir(s)  26,116,567,040 bytes free

Port is unfortunately always open (recognized last year the first time, Lennart is aware of this):

C:\Users\Administrator>netstat -ant | findstr 5665
  TCP    0.0.0.0:5665           0.0.0.0:0              LISTENING       InHost
  TCP    192.168.53.254:49184   192.168.53.210:5665    ESTABLISHED     InHost
  TCP    [::]:5665              [::]:0                 LISTENING       InHost

The icinga2 service runs as LocalSystem.

dnsmichi commented 5 years ago

Ok, in my Windows 10 VM, disabling the API feature doesn't do any listening. Your system is Windows Server 2016, I guess?

Another guess, maybe you still have an old x86 icinga2 instance installed on the system, which actually is started/stopped with the net command.

I'm a newbie to Powershell, but this command unveils the exe path in PS:

C:\Program Files\ICINGA2\sbin> Get-WmiObject win32_service | ?{$_.Name -like '*icinga*'} | select Name, DisplayName, State, PathName

Name    DisplayName State   PathName
----    ----------- -----   --------
icinga2 Icinga 2    Running "C:\Program Files\ICINGA2\sbin\icinga2.exe" --scm "daemon"
R-Sommer commented 5 years ago 
PS C:\Users\Administrator> Get-WmiObject win32_service | ?{$_.Name -like '*icinga*'} | select Name, DisplayName, State, PathName

Name                     DisplayName              State                    PathName
----                     -----------              -----                    --------
icinga2                  Icinga 2                 Running                  "C:\Program Files\ICINGA2\sbin\icinga2.exe" --...

Icinga Files Icinga Version Winodows Version

dnsmichi commented 5 years ago

Ok thanks, then I need a 2k12 VM.

R-Sommer commented 5 years ago

Some more test shows a changed behaviour. Up to 2.10.5 disabling api closes 5665/tcp, v2.11.0-rc1-1-g2b891fd1e does not.

C:\Program Files\ICINGA2\sbin>icinga2.exe feature list
Disabled features: api elasticsearch gelf graphite influxdb notification opentsdb perfdata
Enabled features: checker debuglog mainlog

C:\Program Files\ICINGA2\sbin>netstat -ant | findstr 5665
  TCP    0.0.0.0:5665           0.0.0.0:0              LISTENING       InHost
  TCP    192.168.53.254:49998   192.168.53.210:5665    ESTABLISHED     InHost
  TCP    [::]:5665              [::]:0                 LISTENING       InHost
dnsmichi commented 5 years ago

Hm. And in terms of the debuglog, did that change? Anything in the Windows event console about this?

dnsmichi commented 5 years ago

I now have an OpenStack VM with Windows Server 2012 R2. A fresh installation of Icinga 2 2.11 RC1 works with two things:

Screen Shot 2019-08-05 at 16 35 23 Screen Shot 2019-08-05 at 16 36 28

Not sure where else to look.

dnsmichi commented 5 years ago

Also, tried some disable/enable dances with simulating a connecting client. In the first shot, I was able to connect with the api enabled, then disabled it, no listener, no connect.

Screen Shot 2019-08-05 at 18 09 14 Screen Shot 2019-08-05 at 18 09 35

To me, this really sounds like a permission issue on the Windows box of yours. Or missing/hanging service restarts. Whenever there's a connection established, I'm not sure how intelligent the connection is cleaned up by the Windows kernel.

dnsmichi commented 5 years ago

Anyhow - since we endorse the Windows agent, having the API enabled is a key feature here. The API listening for incoming connections doesn't hurt, Windows has a firewall which hit me during testing as well.

Coming back from offtopic, is your debug.log file written in %ProgramData%\var\log\icinga2 now?

R-Sommer commented 5 years ago

Ok, but then it's time to remove the TCP Listener setting from setup wizard (as it hasn't been making any difference since I've been using it, means v2.8.0).

debuglog haven't been created.

  1. Next try was install on identical other VM, debuglog was created.
  2. Uninstall and rerun the powershell modul on the first machine, debuglog was created.
  3. Uninstall and delete the host object in the director and rerun the powershell modul, debuglog haven't been created until now
  4. cross check: rerun step 2, same result

To me it looks like it's not an issue of the agent but the powershell modul. Therefore, I looked into icinga.conf (generated by the powershell modul) and found this section:

/* Define our block required to enable or disable Icinga 2 debug log
 * Enable or disable it by using the PowerShell Module with
 * argument -IcingaEnableDebugLog or by switching
 * PowerShellIcinga2EnableDebug to true or false manually.
 * true: Debug log is active
 * false: Debug log is deactivated
 * IMPORTANT: ";" after true or false has to remain to allow the
 *            PowerShell Module to switch this feature on or off.
 */
const PowerShellIcinga2EnableDebug = false;
const PowerShellIcinga2EnableLog = true;

if (PowerShellIcinga2EnableDebug) {
  object FileLogger "debug-file" {
    severity = "debug"
    path = LocalStateDir + "/log/icinga2/debug.log"
  }
}

BTW: With run 2 api is disabled and port 5665/tcp is closed. With run 3 api is also disabled but port 5665/tcp is open.

dnsmichi commented 5 years ago

Orrrrrrrrrrrrrrr.

The Powershell module doesn't support feature enable/disable, it does that differently. Same goes for enabling or disabling the API.

@lippserd - that PS module needs a rework, definitely.

Conclusion - when using the Powershell script, you'll need to use different options to enable/disable the api and likewise, the debuglog feature.

In terms of the Windows wizard GUI - the "TCP Listener" setting populates the api.conf file with bind_host etc. That file is not included when using the Powershell script.

dnsmichi commented 5 years ago

I think we can close this issue as being related to the Powershell module. The native Icinga agent on Windows works and also the setup GUI plus CLI commands in the background. I'll take a note for our product strategy meeting to refine the components working together.

dnsmichi commented 5 years ago

@R-Sommer thanks for keeping up here, those tests a really worthy with helping shape the RC 👍 Unfortunately or good this time, it wasn't related to Icinga Core itself.

R-Sommer commented 5 years ago

Totally agree with closing this issue here.

Thanks for clarification about bind_host, however, I'd expect the port being closed when "Do not listen..." is selected. This is no a big thing, but still confusing/bugging me.