Closed markdavies71 closed 2 years ago
Hello! The integration currently retrieves services data leveraging the windows prometheus exporter.
Can you provide the /metrics
output of the exporter?
You can find the port and the binded address in the config:`
# exporter_bind_address: 127.0.0.1
# exporter_bind_port: 9182
Then in the output we can check if those services are present.
Afternoon
The following is the config.yml that I am trying to use, but I not see any metrics. Please advise where I am going wrong.
collectors:
enabled: service
log:
level: info
scrape:
timeout-margin: 0.5
telemetry:
addr: "127.0.0.1:9182"
path: /metrics
max-requests: 5
Output in PowerShell
time="2022-07-01T11:07:35Z" level=info msg="Loading configuration file: c:\\temp\\config.yml" source="config.go:37"
time="2022-07-01T11:07:35Z" level=warning msg="No where-clause specified for service collector. This will generate a very large number of metrics!" source="service.go:47"
time="2022-07-01T11:07:35Z" level=info msg="Enabled collectors: service" source="exporter.go:347"
time="2022-07-01T11:07:35Z" level=info msg="Starting windows_exporter (version=0.16.0-61-g3e37b7b-dirty, branch=HEAD, revision=3e37b7b6f0422f980da644717419201727c95000)" source="exporter.go:399"
time="2022-07-01T11:07:35Z" level=info msg="Build context (go=go1.16.8, user=fv-az8-106\\runneradmin@fv-az8-106, date=20210928-14:19:49)" source="exporter.go:400"
time="2022-07-01T11:07:35Z" level=info msg="Starting server on 127.0.0.1:9182" source="exporter.go:403"
time="2022-07-01T11:07:35Z" level=info msg="TLS is disabled." source="gokit_adapter.go:38"
envagent.txt
[
{
"Name": "ALLUSERSPROFILE",
"Value": "C:\\ProgramData"
},
{
"Name": "APPDATA",
"Value": "C:\\windows\\system32\\config\\systemprofile\\AppData\\Roaming"
},
{
"Name": "CommonProgramFiles",
"Value": "C:\\Program Files\\Common Files"
},
{
"Name": "CommonProgramFiles(x86)",
"Value": "C:\\Program Files (x86)\\Common Files"
},
{
"Name": "CommonProgramW6432",
"Value": "C:\\Program Files\\Common Files"
},
{
"Name": "COMPUTERNAME",
"Value": "azvmtstappz0001"
},
{
"Name": "ComSpec",
"Value": "C:\\windows\\system32\\cmd.exe"
},
{
"Name": "CONFIG_PATH",
"Value": "C:\\windows\\TEMP\\discovered3646436570"
},
{
"Name": "LOCALAPPDATA",
"Value": "C:\\windows\\system32\\config\\systemprofile\\AppData\\Local"
},
{
"Name": "NEWRELIC_INSTALL_PATH",
"Value": "C:\\Program Files\\New Relic\\.NET Agent\\"
},
{
"Name": "NRI_CONFIG_INTERVAL",
"Value": "30s"
},
{
"Name": "NRI_HOST_ID",
"Value": "166605c2-8759-4c8d-a51e-f56c8a58090d"
},
{
"Name": "Path",
"Value": "C:\\windows\\system32;C:\\windows;C:\\windows\\System32\\Wbem;C:\\windows\\System32\\WindowsPowerShell\\v1.0\\;C:\\windows\\System32\\OpenSSH\\;C:\\ProgramData\\chocolatey\\bin;C:\\Program Files\\Microsoft SQL Server\\Client SDK\\ODBC\\170\\Tools\\Binn\\;C:\\Program Files (x86)\\Microsoft SQL Server\\150\\Tools\\Binn\\;C:\\Program Files\\Microsoft SQL Server\\150\\Tools\\Binn\\;C:\\Program Files\\Microsoft SQL Server\\150\\DTS\\Binn\\;C:\\Program Files (x86)\\Microsoft SQL Server\\150\\DTS\\Binn\\;C:\\Program Files\\Azure Data Studio\\bin;C:\\Program Files\\Microsoft\\Web Platform Installer\\;C:\\Program Files\\PowerShell\\7\\;C:\\windows\\system32\\config\\systemprofile\\AppData\\Local\\Microsoft\\WindowsApps"
},
{
"Name": "PATHEXT",
"Value": ".COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.CPL"
},
{
"Name": "ProgramData",
"Value": "C:\\ProgramData"
},
{
"Name": "ProgramFiles",
"Value": "C:\\Program Files"
},
{
"Name": "ProgramFiles(x86)",
"Value": "C:\\Program Files (x86)"
},
{
"Name": "ProgramW6432",
"Value": "C:\\Program Files"
},
{
"Name": "PSModulePath",
"Value": "C:\\Program Files\\WindowsPowerShell\\Modules;C:\\windows\\system32\\WindowsPowerShell\\v1.0\\Modules;C:\\Program Files (x86)\\Microsoft SQL Server\\150\\Tools\\PowerShell\\Modules\\"
},
{
"Name": "SystemDrive",
"Value": "C:"
},
{
"Name": "SystemRoot",
"Value": "C:\\windows"
},
{
"Name": "TEMP",
"Value": "C:\\windows\\TEMP"
},
{
"Name": "TMP",
"Value": "C:\\windows\\TEMP"
},
{
"Name": "VERBOSE",
"Value": "1"
},
{
"Name": "windir",
"Value": "C:\\windows"
}
]
Hi @markdavies71
As @paologallinaharbur mentioned, we would need to check the output of the metrics endpoint from the exporter.
You can retrieve it either through a browser using the url http://127.0.0.1:9182/metrics
or from a powershell running the command:
Invoke-Webrequest -Uri http://127.0.0.1:9182/metrics | Select-Object -ExpandProperty Content
This will dump all the output to the screen. Alternatively you can redirect the output to a file:
Invoke-Webrequest -Uri http://127.0.0.1:9182/metrics | Select-Object -ExpandProperty Content > services.txt
Then grab that services.txt
and attach it here
If you want to have a look at the output, each service will produce the following lines of info:
windows_service_info{display_name="New Relic Infrastructure Agent",name="newrelic-infra",process_id="3644",run_as="LocalSystem"} 1
...
windows_service_start_mode{name="newrelic-infra",start_mode="auto"} 1
windows_service_start_mode{name="newrelic-infra",start_mode="boot"} 0
windows_service_start_mode{name="newrelic-infra",start_mode="disabled"} 0
windows_service_start_mode{name="newrelic-infra",start_mode="manual"} 0
windows_service_start_mode{name="newrelic-infra",start_mode="system"} 0
...
windows_service_state{name="newrelic-infra",state="continue pending"} 0
windows_service_state{name="newrelic-infra",state="pause pending"} 0
windows_service_state{name="newrelic-infra",state="paused"} 0
windows_service_state{name="newrelic-infra",state="running"} 1
windows_service_state{name="newrelic-infra",state="start pending"} 0
windows_service_state{name="newrelic-infra",state="stop pending"} 0
windows_service_state{name="newrelic-infra",state="stopped"} 0
windows_service_state{name="newrelic-infra",state="unknown"} 0
...
The above example is for the newrelic-infra
service but they need to exist for all the 16 services that you are trying to monitor.
Thanks @carlossscastro
Here are the details of the output through the browser when I specify just windefend
in the config.yml
and run the command from a powershell prompt
.\windows_exporter.exe --config.file=c:\temp\config.yml
windows_service_info{display_name="Windows Defender Antivirus Service",name="windefend",process_id="3200",run_as="LocalSystem"} 1
windows_service_start_mode{name="windefend",start_mode="auto"} 1
windows_service_start_mode{name="windefend",start_mode="boot"} 0
windows_service_start_mode{name="windefend",start_mode="disabled"} 0
windows_service_start_mode{name="windefend",start_mode="manual"} 0
windows_service_start_mode{name="windefend",start_mode="system"} 0
windows_service_state{name="windefend",state="continue pending"} 0
windows_service_state{name="windefend",state="pause pending"} 0
windows_service_state{name="windefend",state="paused"} 0
windows_service_state{name="windefend",state="running"} 1
windows_service_state{name="windefend",state="start pending"} 0
windows_service_state{name="windefend",state="stop pending"} 0
windows_service_state{name="windefend",state="stopped"} 0
windows_service_state{name="windefend",state="unknown"} 0
windows_service_status{name="windefend",status="degraded"} 0
windows_service_status{name="windefend",status="error"} 0
windows_service_status{name="windefend",status="lost comm"} 0
windows_service_status{name="windefend",status="no contact"} 0
windows_service_status{name="windefend",status="nonrecover"} 0
windows_service_status{name="windefend",status="ok"} 1
windows_service_status{name="windefend",status="pred fail"} 0
windows_service_status{name="windefend",status="service"} 0
windows_service_status{name="windefend",status="starting"} 0
windows_service_status{name="windefend",status="stopping"} 0
windows_service_status{name="windefend",status="stressed"} 0
windows_service_status{name="windefend",status="unknown"} 0
However when I look at the output being generated from nri-winservices
I see no reference to windefend
windows_service_info{display_name="Windows Connection Manager",name="wcmsvc",process_id="1888",run_as="NT Authority\\LocalService"} 1
windows_service_info{display_name="Windows Encryption Provider Host Service",name="wephostsvc",process_id="0",run_as="NT AUTHORITY\\LocalService"} 1
Hello @markdavies71.
We spotted the issue, since the exporter we are using calls the windows Api and tries to collect the services with the rights "SC_MANAGER_ALL_ACCESS", some restricted services can't be opened.
We have opened the following PR to the windows exporter making the services to be opened with "GENERIC_READ", fixing the issue.
https://github.com/prometheus-community/windows_exporter/pull/1036
Once this PR is merged, we will bump the dependency in our integration and generate a new release.
We will let you know as soon as the new release is done.
Hello, we have just created the prerelease of the nri-winservices beta that is adding the exporter solving the issue. It will still not be added to the infrastructure-agent until next week, we will communicate it here once we do the infrastructure-agent release.
The last released version of the agent https://github.com/newrelic/infrastructure-agent/releases/tag/1.29.0 is including the release of nri-winservices beta that solves this issue.
Description
Metrics from certain services are not being collected and sent to New Relic
Expected Behavior
The status and startup metrics for all services configured within
winservices-config.yml
are captured and sent to New RelicSteps to Reproduce
The following are the services that I am trying to collect metrics from in the
winservices-config.yml
These are being parsed and recorded in the newrelic-infra.log
However only 13 services are returned, and not the 16 as listed in the yml file.
Your Environment
Windows Server 2019 New Relic Infrastructure Agent: v1.26.0 winservices.exe: v0.5.0-beta
tested config:
Additional context
The metric data is being capture and sent to New Relic for all the other 13 services listed in the yml file.