search

Icinga / icinga-powershell-framework

This PowerShell module will allow to fetch data from Windows hosts and use them for inventory and monitoring solutions. Together with the Icinga Web 2 module, a detailed overview of your Windows infrastructure will be drawn.
MIT License
78 stars 33 forks source link

API Check Forwarder - Unable to connect to the remote server #597

Closed fluxX04 closed 7 months ago

fluxX04 commented 1 year ago

Hi,

this is mostly a follow-up issue to #483.

We want to migrate to the API Check Forwarder to get rid of the errors/warnings referenced at IWKB000008

Expected Behavior

That the internal api checks are working.

Current Behavior

Tried it on two servers but running everytime into the same error Unable to connect to the remote server.

Stacktrace:

Exact Position:
At C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-framework\cache\framework_cache.psm1:3016 char:22
+ ... ApiResult = Invoke-WebRequest -Method POST -UseBasicParsing -Uri ([st ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

StackTrace:
   at System.Net.HttpWebRequest.GetRequestStream(TransportContext& context)
   at System.Net.HttpWebRequest.GetRequestStream()
   at Microsoft.PowerShell.Commands.WebRequestPSCmdlet.SetRequestContent(WebRequest request, Byte[] content)
   at Microsoft.PowerShell.Commands.WebRequestPSCmdlet.ProcessRecord()
   at System.Management.Automation.CommandProcessor.ProcessRecord()
Full log ``` Failed to query Icinga check over internal REST-Api check handler A service check could not be executed by using the internal REST-Api check handler. The check either ran into a timeout or could not be processed. Maybe the check was not registered to be allowed for being executed. Further details can be found below. Icinga for Windows exception report: Exception Message: Unable to connect to the remote server Invocation Name: Invoke-WebRequest Command Origin: Internal Script Line Number: 3016 Exact Position: At C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-framework\cache\framework_cache.psm1:3016 char:22 + ... ApiResult = Invoke-WebRequest -Method POST -UseBasicParsing -Uri ([st ... + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ StackTrace: at System.Net.HttpWebRequest.GetRequestStream(TransportContext& context) at System.Net.HttpWebRequest.GetRequestStream() at Microsoft.PowerShell.Commands.WebRequestPSCmdlet.SetRequestContent(WebRequest request, Byte[] content) at Microsoft.PowerShell.Commands.WebRequestPSCmdlet.ProcessRecord() at System.Management.Automation.CommandProcessor.ProcessRecord() Call Stack: Command Arguments ------- --------- Get-IcingaExceptionString {ExceptionObject=Unable to connect to the remote server} Write-IcingaEventMessage {Namespace=Framework, EventId=1553, ExceptionObject=Unable to connect to the remote... Invoke-IcingaInternalServiceCall {Command=Invoke-IcingaCheckUptime, Arguments=System.Collections.Hashtable} Exit-IcingaExecutePlugin {Command=Invoke-IcingaCheckUptime, -Verbosity, 2} {} Object details: Invoke-IcingaCheckUptime Name Value ---- ----- Verbosity 2 ```

If i restart the Icinga Powershell Service through powershell the log says that the service was correctly started and the service is also running:

Starting Icinga for Windows service with arguments '-NoProfile -NoLogo -Command Invoke-Command { Import-Module 'C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-framework\icinga-powershell-framework.psd1'; Use-Icinga -Daemon; if (Test-IcingaFunction -Name 'Start-IcingaForWindowsDaemon') { Start-IcingaForWindowsDaemon -RunAsService | Out-Null; } else { Start-IcingaPowerShellDaemon -RunAsService | Out-Null; } }' 
PS C:\Windows\system32> Get-Service icingapowershell

Status   Name               DisplayName
------   ----               -----------
Running  icingapowershell   Icinga PowerShell Service

Possible Solution

It's not a solution, just a workaround that i found out.

If i run the command in a powershell session Start-IcingaForWindowsDaemon connections are working perfectly and no error is getting logged. netstat also list port 5668 in the active connections and i can query the api through the browser.

https://localhost:5668/v1/checker?command=cpu

{"Invoke-IcingaCheckCPU":{"exitcode":0,"checkresult":"[OK] CPU Load","perfdata":["\u0027total::ifw_cpu::load\u0027=10.44916%;;;0;100","\u00270::ifw_cpu::load\u0027=10.6251%;;;0;100","\u00271::ifw_cpu::load\u0027=1.37997%;;;0;100","\u00272::ifw_cpu::load\u0027=15.77641%;;;0;100","\u00273::ifw_cpu::load\u0027=14.38354%;;;0;100"]}}

If i close the powershell window the connection fails again. Errors are getting logged Unable to connect to the remote server and the browser displays an ERR_CONNECTION_REFUSED.

Your Environment

Windows datacenter server 2019 and 2022.
Powershell on version 5.
Icinga Powershell Framework, Service and RESTApi on the latest available version.

Greetz

LordHepipud commented 1 year ago

Thank you for the report. Can you please share the content of Show-Icinga and remove private/non-public data there? I just want need some additional information provided there.

fluxX04 commented 1 year ago

Hi,

sure:

Windows Datacenter Server 2022 ``` PS C:\Windows\system32> Show-Icinga Icinga for Windows environment: ----------- Environment configuration: PowerShell Root => C:\Program Files\WindowsPowerShell\Modules\ Icinga for Windows Service Path => C:\Program Files\icinga-framework-service\ Icinga for Windows Service User => NT Authority\NetworkService Icinga for Windows Service Pid => 7272 Icinga for Windows JEA Pid => Icinga Agent Path => c:\Program Files\ICINGA2\ Icinga Agent User => NT AUTHORITY\NetworkService Defined Default User => NT Authority\NetworkService Icinga Managed User => False PowerShell Version => 5.1.20348.859 Operating System => Microsoft Windows Server 2022 Datacenter Operating System Version => 10.0.20348 JEA Context => JEA Session File => Api Check Forwarder => True Debug Mode => False Icinga for Windows Certificate: Not installed List of configured background daemons on this system: Start-IcingaServiceCheckDaemon ----------- No arguments defined Start-IcingaWindowsRESTApi ----------- -CertFile => C:\ProgramData\icinga2\var\lib\icinga2\certs\myserver.mydomain.com.crt List of configured background service checks on this system: => https://icinga.com/docs/icinga-for-windows/latest/doc/110-Installation/06-Collect-Metrics-over-Time/ No background service checks configured List of configured repositories on this system. The list order matches the apply order: Icinga Stable ----------- CloneSource => Enabled => True LocalPath => Order => 0 RemotePath => https://packages.icinga.com/IcingaForWindows/stable/ifw.repo.json UseSCP => False Installed components on this system: Component Version Available --- --- --- agent 2.13.4 2.13.6 framework 1.10.0 1.10.0 plugins 1.10.0 1.10.0 restapi 1.2.0 1.2.0 service 1.2.0 1.2.0 Available versions flagged with "*" mean that this component is locked to this version ```
Windows Datacenter Server 2019 ``` PS C:\Windows\system32> Show-Icinga Icinga for Windows environment: ----------- Environment configuration: PowerShell Root => C:\Program Files\WindowsPowerShell\Modules\ Icinga for Windows Service Path => C:\Program Files\icinga-framework-service\ Icinga for Windows Service User => NT Authority\NetworkService Icinga for Windows Service Pid => 4572 Icinga for Windows JEA Pid => Icinga Agent Path => c:\Program Files\ICINGA2\ Icinga Agent User => NT AUTHORITY\NetworkService Defined Default User => NT Authority\NetworkService Icinga Managed User => False PowerShell Version => 5.1.17763.3770 Operating System => Microsoft Windows Server 2019 Datacenter Operating System Version => 10.0.17763 JEA Context => JEA Session File => Api Check Forwarder => False Debug Mode => False Icinga for Windows Certificate: Not installed List of configured background daemons on this system: Start-IcingaServiceCheckDaemon ----------- No arguments defined Start-IcingaWindowsRESTApi ----------- -CertFile => C:\ProgramData\icinga2\var\lib\icinga2\certs\myserver.mydomain.com.crt List of configured background service checks on this system: => https://icinga.com/docs/icinga-for-windows/latest/doc/110-Installation/06-Collect-Metrics-over-Time/ No background service checks configured List of configured repositories on this system. The list order matches the apply order: Icinga Stable ----------- CloneSource => Enabled => True LocalPath => Order => 0 RemotePath => https://packages.icinga.com/IcingaForWindows/stable/ifw.repo.json UseSCP => False Installed components on this system: Component Version Available --- --- --- agent 2.13.4 2.13.6 framework 1.10.0 1.10.0 plugins 1.10.0 1.10.0 restapi 1.2.0 1.2.0 service 1.2.0 1.2.0 Available versions flagged with "*" mean that this component is locked to this version ```

Greetz

fluxX04 commented 7 months ago

After some time later :D works all as expected. Docs can be improved tho, as the restapi is deprecated.

Greetz