search

newrelic / nri-winservices

Windows services Integration for New Relic Infrastructure
Apache License 2.0
8 stars 8 forks source link

Certain Service status and state not collected #128

Closed markdavies71 closed 2 years ago

markdavies71 commented 2 years ago

Description

Metrics from certain services are not being collected and sent to New Relic

Expected Behavior

The status and startup metrics for all services configured within winservices-config.yml are captured and sent to New Relic

Steps to Reproduce

The following are the services that I am trying to collect metrics from in the winservices-config.yml

  include_matching_entities:
    windowsService.name:
      # - regex ".*"
      - "newrelic-infra"
      - "Winmgmt"
      - "Netman"
      - "RpcSs"
      - "W32Time"
      - "WinDefend"
      - "W3SVC"
      - "WinRM"
      - "Dhcp"
      - "lmhosts"
      - "IISADMIN"
      - "LanmanWorkstation"
      - "EventLog"
      - "LanmanServer"
      - "PlugPlay"
      - "mpssvc"

These are being parsed and recorded in the newrelic-infra.log

time="2022-06-23T07:28:47Z" level=info msg="Integration stderr (not parsed)." component=integrations.runner.Runner integration_name=nri-winservices line="[DEBUG] integration version: v0.5.0-beta commit: 56672a58e2324a17f41662b0dd382c8965913bd3"
time="2022-06-23T07:28:47Z" level=info msg="Integration stderr (not parsed)." component=integrations.runner.Runner integration_name=nri-winservices line="[DEBUG] pattern added regex: ^newrelic-infra$ "
time="2022-06-23T07:28:47Z" level=info msg="Integration stderr (not parsed)." component=integrations.runner.Runner integration_name=nri-winservices line="[DEBUG] pattern added regex: ^Winmgmt$ "
time="2022-06-23T07:28:47Z" level=info msg="Integration stderr (not parsed)." component=integrations.runner.Runner integration_name=nri-winservices line="[DEBUG] pattern added regex: ^Netman$ "
time="2022-06-23T07:28:47Z" level=info msg="Integration stderr (not parsed)." component=integrations.runner.Runner integration_name=nri-winservices line="[DEBUG] pattern added regex: ^RpcSs$ "
time="2022-06-23T07:28:47Z" level=info msg="Integration stderr (not parsed)." component=integrations.runner.Runner integration_name=nri-winservices line="[DEBUG] pattern added regex: ^W32Time$ "
time="2022-06-23T07:28:47Z" level=info msg="Integration stderr (not parsed)." component=integrations.runner.Runner integration_name=nri-winservices line="[DEBUG] pattern added regex: ^WinDefend$ "
time="2022-06-23T07:28:47Z" level=info msg="Integration stderr (not parsed)." component=integrations.runner.Runner integration_name=nri-winservices line="[DEBUG] pattern added regex: ^W3SVC$ "
time="2022-06-23T07:28:47Z" level=info msg="Integration stderr (not parsed)." component=integrations.runner.Runner integration_name=nri-winservices line="[DEBUG] pattern added regex: ^WinRM$ "
time="2022-06-23T07:28:47Z" level=info msg="Integration stderr (not parsed)." component=integrations.runner.Runner integration_name=nri-winservices line="[DEBUG] pattern added regex: ^Dhcp$ "
time="2022-06-23T07:28:47Z" level=info msg="Integration stderr (not parsed)." component=integrations.runner.Runner integration_name=nri-winservices line="[DEBUG] pattern added regex: ^lmhosts$ "
time="2022-06-23T07:28:47Z" level=info msg="Integration stderr (not parsed)." component=integrations.runner.Runner integration_name=nri-winservices line="[DEBUG] pattern added regex: ^IISADMIN$ "
time="2022-06-23T07:28:47Z" level=info msg="Integration stderr (not parsed)." component=integrations.runner.Runner integration_name=nri-winservices line="[DEBUG] pattern added regex: ^LanmanWorkstation$ "
time="2022-06-23T07:28:47Z" level=info msg="Integration stderr (not parsed)." component=integrations.runner.Runner integration_name=nri-winservices line="[DEBUG] pattern added regex: ^EventLog$ "
time="2022-06-23T07:28:47Z" level=info msg="Integration stderr (not parsed)." component=integrations.runner.Runner integration_name=nri-winservices line="[DEBUG] pattern added regex: ^LanmanServer$ "
time="2022-06-23T07:28:47Z" level=info msg="Integration stderr (not parsed)." component=integrations.runner.Runner integration_name=nri-winservices line="[DEBUG] pattern added regex: ^PlugPlay$ "
time="2022-06-23T07:28:47Z" level=info msg="Integration stderr (not parsed)." component=integrations.runner.Runner integration_name=nri-winservices line="[DEBUG] pattern added regex: ^mpssvc$ "
time="2022-06-23T07:28:47Z" level=info msg="Integration stderr (not parsed)." component=integrations.runner.Runner integration_name=nri-winservices line="[DEBUG] running with scrape interval: 30s"
time="2022-06-23T07:28:47Z" level=info msg="Integration stderr (not parsed)." component=integrations.runner.Runner integration_name=nri-winservices line="[DEBUG] Running exporter"
time="2022-06-23T07:28:47Z" level=info msg="Integration stderr (not parsed)." component=integrations.runner.Runner integration_name=nri-winservices line="[DEBUG] Running Integration"
time="2022-06-23T07:28:47Z" level=debug msg="Sending events to metrics-ingest." component=MetricsIngestSender key=4057350291622284934 numEvents=12 postCount=0 timestamps="[2022-06-23 07:28:46 +0000 GMT 2022-06-23 07:28:47 +0000 GMT]"

However only 13 services are returned, and not the 16 as listed in the yml file.

time="2022-06-23T07:29:22Z" level=info msg="Integration stderr (not parsed)." component=integrations.runner.Runner integration_name=nri-winservices line="[DEBUG] exporter msg=collector service succeeded after 0.104446s. source=exporter.go:209"
time="2022-06-23T07:29:22Z" level=info msg="Integration stderr (not parsed)." component=integrations.runner.Runner integration_name=nri-winservices line="[DEBUG] HTTP request performed - Status: 200 OK, total time taken to perform request: 4.5190576s"
time="2022-06-23T07:29:22Z" level=info msg="Integration stderr (not parsed)." component=integrations.runner.Runner integration_name=nri-winservices line="[DEBUG] Parsing body of the exporter answer"
time="2022-06-23T07:29:22Z" level=info msg="Integration stderr (not parsed)." component=integrations.runner.Runner integration_name=nri-winservices line="[DEBUG] Body of the exporter answer parsed"
time="2022-06-23T07:29:22Z" level=info msg="Integration stderr (not parsed)." component=integrations.runner.Runner integration_name=nri-winservices line="[DEBUG] Metrics scraped, MetricsByFamily found: 42, time elapsed: 4.5286218s"
time="2022-06-23T07:29:22Z" level=debug msg="Received payload." component=integrations.emitter.Emitter env="map[NRI_CONFIG_INTERVAL:30s]" integration_name=nri-winservices payload="{\"protocol_version\":\"4\",\"integration\":{\"name\":\"com.newrelic.winservices\",\"version\":\"v0.5.0-beta\"},\"data\":[{\"common\":{},\"entity\":{\"name\":\"WIN_SERVICE:localhost:dhcp\",\"displayName\":\"DHCP Client\",\"type\":\"WIN_SERVICE\",\"metadata\":{\"display_name\":\"DHCP Client\",\"hostname\":\"azvmtstappz0001\",\"process_id\":\"1624\",\"run_as\":\"NT Authority\\\\LocalService\",\"service_name\":\"dhcp\",\"start_mode\":\"auto\"}},\"metrics\":[{\"timestamp\":1655969362,\"name\":\"windows_service_start_mode\",\"type\":\"gauge\",\"attributes\":{},\"value\":1},{\"timestamp\":1655969362,\"name\":\"windows_service_state\",\"type\":\"gauge\",\"attributes\":{\"state\":\"running\"},\"value\":1}],\"inventory\":{},\"events\":[]},{\"common\":{},\"entity\":{\"name\":\"WIN_SERVICE:localhost:iisadmin\",\"displayName\":\"IIS Admin Service\",\"type\":\"WIN_SERVICE\",\"metadata\":{\"display_name\":\"IIS Admin Service\",\"hostname\":\"azvmtstappz0001\",\"process_id\":\"2864\",\"run_as\":\"localSystem\",\"service_name\":\"iisadmin\",\"start_mode\":\"auto\"}},\"metrics\":[{\"timestamp\":1655969362,\"name\":\"windows_service_start_mode\",\"type\":\"gauge\",\"attributes\":{},\"value\":1},{\"timestamp\":1655969362,\"name\":\"windows_service_state\",\"type\":\"gauge\",\"attributes\":{\"state\":\"running\"},\"value\":1}],\"inventory\":{},\"events\":[]},{\"common\":{},\"entity\":{\"name\":\"WIN_SERVICE:localhost:netman\",\"displayName\":\"Network Connections\",\"type\":\"WIN_SERVICE\",\"metadata\":{\"display_name\":\"Network Connections\",\"hostname\":\"azvmtstappz0001\",\"process_id\":\"0\",\"run_as\":\"LocalSystem\",\"service_name\":\"netman\",\"start_mode\":\"manual\"}},\"metrics\":[{\"timestamp\":1655969362,\"name\":\"windows_service_start_mode\",\"type\":\"gauge\",\"attributes\":{},\"value\":1},{\"timestamp\":1655969362,\"name\":\"windows_service_state\",\"type\":\"gauge\",\"attributes\":{\"state\":\"stopped\"},\"value\":1}],\"inventory\":{},\"events\":[]},{\"common\":{},\"entity\":{\"name\":\"WIN_SERVICE:localhost:newrelic-infra\",\"displayName\":\"New Relic Infrastructure Agent\",\"type\":\"WIN_SERVICE\",\"metadata\":{\"display_name\":\"New Relic Infrastructure Agent\",\"hostname\":\"azvmtstappz0001\",\"process_id\":\"2012\",\"run_as\":\"LocalSystem\",\"service_name\":\"newrelic-infra\",\"start_mode\":\"auto\"}},\"metrics\":[{\"timestamp\":1655969362,\"name\":\"windows_service_start_mode\",\"type\":\"gauge\",\"attributes\":{},\"value\":1},{\"timestamp\":1655969362,\"name\":\"windows_service_state\",\"type\":\"gauge\",\"attributes\":{\"state\":\"running\"},\"value\":1}],\"inventory\":{},\"events\":[]},{\"common\":{},\"entity\":{\"name\":\"WIN_SERVICE:localhost:plugplay\",\"displayName\":\"Plug and Play\",\"type\":\"WIN_SERVICE\",\"metadata\":{\"display_name\":\"Plug and Play\",\"hostname\":\"azvmtstappz0001\",\"process_id\":\"836\",\"run_as\":\"LocalSystem\",\"service_name\":\"plugplay\",\"start_mode\":\"manual\"}},\"metrics\":[{\"timestamp\":1655969362,\"name\":\"windows_service_start_mode\",\"type\":\"gauge\",\"attributes\":{},\"value\":1},{\"timestamp\":1655969362,\"name\":\"windows_service_state\",\"type\":\"gauge\",\"attributes\":{\"state\":\"running\"},\"value\":1}],\"inventory\":{},\"events\":[]},{\"common\":{},\"entity\":{\"name\":\"WIN_SERVICE:localhost:lanmanserver\",\"displayName\":\"Server\",\"type\":\"WIN_SERVICE\",\"metadata\":{\"display_name\":\"Server\",\"hostname\":\"azvmtstappz0001\",\"process_id\":\"3312\",\"run_as\":\"LocalSystem\",\"service_name\":\"lanmanserver\",\"start_mode\":\"auto\"}},\"metrics\":[{\"timestamp\":1655969362,\"name\":\"windows_service_start_mode\",\"type\":\"gauge\",\"attributes\":{},\"value\":1},{\"timestamp\":1655969362,\"name\":\"windows_service_state\",\"type\":\"gauge\",\"attributes\":{\"state\":\"running\"},\"value\":1}],\"inventory\":{},\"events\":[]},{\"common\":{},\"entity\":{\"name\":\"WIN_SERVICE:localhost:lmhosts\",\"displayName\":\"TCP/IP NetBIOS Helper\",\"type\":\"WIN_SERVICE\",\"metadata\":{\"display_name\":\"TCP/IP NetBIOS Helper\",\"hostname\":\"azvmtstappz0001\",\"process_id\":\"1052\",\"run_as\":\"NT AUTHORITY\\\\LocalService\",\"service_name\":\"lmhosts\",\"start_mode\":\"manual\"}},\"metrics\":[{\"timestamp\":1655969362,\"name\":\"windows_service_start_mode\",\"type\":\"gauge\",\"attributes\":{},\"value\":1},{\"timestamp\":1655969362,\"name\":\"windows_service_state\",\"type\":\"gauge\",\"attributes\":{\"state\":\"running\"},\"value\":1}],\"inventory\":{},\"events\":[]},{\"common\":{},\"entity\":{\"name\":\"WIN_SERVICE:localhost:eventlog\",\"displayName\":\"Windows Event Log\",\"type\":\"WIN_SERVICE\",\"metadata\":{\"display_name\":\"Windows Event Log\",\"hostname\":\"azvmtstappz0001\",\"process_id\":\"1348\",\"run_as\":\"NT AUTHORITY\\\\LocalService\",\"service_name\":\"eventlog\",\"start_mode\":\"auto\"}},\"metrics\":[{\"timestamp\":1655969362,\"name\":\"windows_service_start_mode\",\"type\":\"gauge\",\"attributes\":{},\"value\":1},{\"timestamp\":1655969362,\"name\":\"windows_service_state\",\"type\":\"gauge\",\"attributes\":{\"state\":\"running\"},\"value\":1}],\"inventory\":{},\"events\":[]},{\"common\":{},\"entity\":{\"name\":\"WIN_SERVICE:localhost:winmgmt\",\"displayName\":\"Windows Management Instrumentation\",\"type\":\"WIN_SERVICE\",\"metadata\":{\"display_name\":\"Windows Management Instrumentation\",\"hostname\":\"azvmtstappz0001\",\"process_id\":\"2692\",\"run_as\":\"localSystem\",\"service_name\":\"winmgmt\",\"start_mode\":\"auto\"}},\"metrics\":[{\"timestamp\":1655969362,\"name\":\"windows_service_start_mode\",\"type\":\"gauge\",\"attributes\":{},\"value\":1},{\"timestamp\":1655969362,\"name\":\"windows_service_state\",\"type\":\"gauge\",\"attributes\":{\"state\":\"running\"},\"value\":1}],\"inventory\":{},\"events\":[]},{\"common\":{},\"entity\":{\"name\":\"WIN_SERVICE:localhost:winrm\",\"displayName\":\"Windows Remote Management (WS-Management)\",\"type\":\"WIN_SERVICE\",\"metadata\":{\"display_name\":\"Windows Remote Management (WS-Management)\",\"hostname\":\"azvmtstappz0001\",\"process_id\":\"2928\",\"run_as\":\"NT AUTHORITY\\\\NetworkService\",\"service_name\":\"winrm\",\"start_mode\":\"auto\"}},\"metrics\":[{\"timestamp\":1655969362,\"name\":\"windows_service_start_mode\",\"type\":\"gauge\",\"attributes\":{},\"value\":1},{\"timestamp\":1655969362,\"name\":\"windows_service_state\",\"type\":\"gauge\",\"attributes\":{\"state\":\"running\"},\"value\":1}],\"inventory\":{},\"events\":[]},{\"common\":{},\"entity\":{\"name\":\"WIN_SERVICE:localhost:w32time\",\"displayName\":\"Windows Time\",\"type\":\"WIN_SERVICE\",\"metadata\":{\"display_name\":\"Windows Time\",\"hostname\":\"azvmtstappz0001\",\"process_id\":\"2880\",\"run_as\":\"NT AUTHORITY\\\\LocalService\",\"service_name\":\"w32time\",\"start_mode\":\"auto\"}},\"metrics\":[{\"timestamp\":1655969362,\"name\":\"windows_service_start_mode\",\"type\":\"gauge\",\"attributes\":{},\"value\":1},{\"timestamp\":1655969362,\"name\":\"windows_service_state\",\"type\":\"gauge\",\"attributes\":{\"state\":\"running\"},\"value\":1}],\"inventory\":{},\"events\":[]},{\"common\":{},\"entity\":{\"name\":\"WIN_SERVICE:localhost:lanmanworkstation\",\"displayName\":\"Workstation\",\"type\":\"WIN_SERVICE\",\"metadata\":{\"display_name\":\"Workstation\",\"hostname\":\"azvmtstappz0001\",\"process_id\":\"2092\",\"run_as\":\"NT AUTHORITY\\\\NetworkService\",\"service_name\":\"lanmanworkstation\",\"start_mode\":\"auto\"}},\"metrics\":[{\"timestamp\":1655969362,\"name\":\"windows_service_start_mode\",\"type\":\"gauge\",\"attributes\":{},\"value\":1},{\"timestamp\":1655969362,\"name\":\"windows_service_state\",\"type\":\"gauge\",\"attributes\":{\"state\":\"running\"},\"value\":1}],\"inventory\":{},\"events\":[]},{\"common\":{},\"entity\":{\"name\":\"WIN_SERVICE:localhost:w3svc\",\"displayName\":\"World Wide Web Publishing Service\",\"type\":\"WIN_SERVICE\",\"metadata\":{\"display_name\":\"World Wide Web Publishing Service\",\"hostname\":\"azvmtstappz0001\",\"process_id\":\"2872\",\"run_as\":\"localSystem\",\"service_name\":\"w3svc\",\"start_mode\":\"auto\"}},\"metrics\":[{\"timestamp\":1655969362,\"name\":\"windows_service_start_mode\",\"type\":\"gauge\",\"attributes\":{},\"value\":1},{\"timestamp\":1655969362,\"name\":\"windows_service_state\",\"type\":\"gauge\",\"attributes\":{\"state\":\"running\"},\"value\":1}],\"inventory\":{},\"events\":[]}]}"
time="2022-06-23T07:29:22Z" level=info msg="Integration health check finished with success" component=integrations.runner.Runner integration_name=nri-winservices
time="2022-06-23T07:29:22Z" level=info msg="Integration stderr (not parsed)." component=integrations.runner.Runner integration_name=nri-winservices line="[DEBUG] Metrics processed, entities found: 13, time elapsed: 4.5297045s"
time="2022-06-23T07:29:22Z" level=info msg="Integration stderr (not parsed)." component=integrations.runner.Runner integration_name=nri-winservices line="[DEBUG] Metrics published"
time="2022-06-23T07:29:22Z" level=info msg="Integration stderr (not parsed)." component=integrations.runner.Runner integration_name=nri-winservices line="[DEBUG] Sending heartBeat"
time="2022-06-23T07:29:22Z" level=debug msg="Received heartbeat." component=integrations.runner.Runner integration_name=nri-winservices payload="{}"
time="2022-06-23T07:29:22Z" level=debug msg="Still waiting on plugins." component=Agent pluginIds="[integration/nri-perfmon]"
time="2022-06-23T07:29:23Z" level=debug msg="Sending events to metrics-ingest." component=MetricsIngestSender key=4057350291622284934 numEvents=1 postCount=13 timestamps="[2022-06-23 07:29:22 +0000 GMT]"
time="2022-06-23T07:29:23Z" level=debug msg="Preparing metrics post." component=MetricsIngestSender postCount=13
time="2022-06-23T07:29:23Z" level=debug msg="Metrics post succeeded." component=MetricsIngestSender postCount=13

Your Environment

Windows Server 2019 New Relic Infrastructure Agent: v1.26.0 winservices.exe: v0.5.0-beta

tested config:

    config:
      exporter_bind_address: 127.0.0.1
      exporter_bind_port: 9182
      include_matching_entities:
        windowsService.name:
          # - regex ".*"
          - "newrelic-infra"
          - "Winmgmt"
          - "Netman"
          - "RpcSs"
          - "W32Time"
          - "WinDefend"
          - "W3SVC"
          - "WinRM"
          - "Dhcp"
          - "lmhosts"
          - "IISADMIN"
          - "LanmanWorkstation"
          - "EventLog"
          - "LanmanServer"
          - "PlugPlay"
          - "mpssvc"
      scrape_interval: 30s
    timeout: 60s`

Additional context

The metric data is being capture and sent to New Relic for all the other 13 services listed in the yml file.

paologallinaharbur commented 2 years ago

Hello! The integration currently retrieves services data leveraging the windows prometheus exporter.

Can you provide the /metrics output of the exporter? You can find the port and the binded address in the config:`

      # exporter_bind_address: 127.0.0.1
      # exporter_bind_port: 9182

Then in the output we can check if those services are present.

markdavies71 commented 2 years ago

Afternoon

The following is the config.yml that I am trying to use, but I not see any metrics. Please advise where I am going wrong. 

collectors:
  enabled: service

log:
  level: info
scrape:
  timeout-margin: 0.5
telemetry:
  addr: "127.0.0.1:9182"
  path: /metrics
  max-requests: 5

Output in PowerShell

time="2022-07-01T11:07:35Z" level=info msg="Loading configuration file: c:\\temp\\config.yml" source="config.go:37"
time="2022-07-01T11:07:35Z" level=warning msg="No where-clause specified for service collector. This will generate a very large number of metrics!" source="service.go:47"
time="2022-07-01T11:07:35Z" level=info msg="Enabled collectors: service" source="exporter.go:347"
time="2022-07-01T11:07:35Z" level=info msg="Starting windows_exporter (version=0.16.0-61-g3e37b7b-dirty, branch=HEAD, revision=3e37b7b6f0422f980da644717419201727c95000)" source="exporter.go:399"
time="2022-07-01T11:07:35Z" level=info msg="Build context (go=go1.16.8, user=fv-az8-106\\runneradmin@fv-az8-106, date=20210928-14:19:49)" source="exporter.go:400"
time="2022-07-01T11:07:35Z" level=info msg="Starting server on 127.0.0.1:9182" source="exporter.go:403"
time="2022-07-01T11:07:35Z" level=info msg="TLS is disabled." source="gokit_adapter.go:38"

envagent.txt

[
    {
        "Name":  "ALLUSERSPROFILE",
        "Value":  "C:\\ProgramData"
    },
    {
        "Name":  "APPDATA",
        "Value":  "C:\\windows\\system32\\config\\systemprofile\\AppData\\Roaming"
    },
    {
        "Name":  "CommonProgramFiles",
        "Value":  "C:\\Program Files\\Common Files"
    },
    {
        "Name":  "CommonProgramFiles(x86)",
        "Value":  "C:\\Program Files (x86)\\Common Files"
    },
    {
        "Name":  "CommonProgramW6432",
        "Value":  "C:\\Program Files\\Common Files"
    },
    {
        "Name":  "COMPUTERNAME",
        "Value":  "azvmtstappz0001"
    },
    {
        "Name":  "ComSpec",
        "Value":  "C:\\windows\\system32\\cmd.exe"
    },
    {
        "Name":  "CONFIG_PATH",
        "Value":  "C:\\windows\\TEMP\\discovered3646436570"
    },
    {
        "Name":  "LOCALAPPDATA",
        "Value":  "C:\\windows\\system32\\config\\systemprofile\\AppData\\Local"
    },
    {
        "Name":  "NEWRELIC_INSTALL_PATH",
        "Value":  "C:\\Program Files\\New Relic\\.NET Agent\\"
    },
    {
        "Name":  "NRI_CONFIG_INTERVAL",
        "Value":  "30s"
    },
    {
        "Name":  "NRI_HOST_ID",
        "Value":  "166605c2-8759-4c8d-a51e-f56c8a58090d"
    },
    {
        "Name":  "Path",
        "Value":  "C:\\windows\\system32;C:\\windows;C:\\windows\\System32\\Wbem;C:\\windows\\System32\\WindowsPowerShell\\v1.0\\;C:\\windows\\System32\\OpenSSH\\;C:\\ProgramData\\chocolatey\\bin;C:\\Program Files\\Microsoft SQL Server\\Client SDK\\ODBC\\170\\Tools\\Binn\\;C:\\Program Files (x86)\\Microsoft SQL Server\\150\\Tools\\Binn\\;C:\\Program Files\\Microsoft SQL Server\\150\\Tools\\Binn\\;C:\\Program Files\\Microsoft SQL Server\\150\\DTS\\Binn\\;C:\\Program Files (x86)\\Microsoft SQL Server\\150\\DTS\\Binn\\;C:\\Program Files\\Azure Data Studio\\bin;C:\\Program Files\\Microsoft\\Web Platform Installer\\;C:\\Program Files\\PowerShell\\7\\;C:\\windows\\system32\\config\\systemprofile\\AppData\\Local\\Microsoft\\WindowsApps"
    },
    {
        "Name":  "PATHEXT",
        "Value":  ".COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.CPL"
    },
    {
        "Name":  "ProgramData",
        "Value":  "C:\\ProgramData"
    },
    {
        "Name":  "ProgramFiles",
        "Value":  "C:\\Program Files"
    },
    {
        "Name":  "ProgramFiles(x86)",
        "Value":  "C:\\Program Files (x86)"
    },
    {
        "Name":  "ProgramW6432",
        "Value":  "C:\\Program Files"
    },
    {
        "Name":  "PSModulePath",
        "Value":  "C:\\Program Files\\WindowsPowerShell\\Modules;C:\\windows\\system32\\WindowsPowerShell\\v1.0\\Modules;C:\\Program Files (x86)\\Microsoft SQL Server\\150\\Tools\\PowerShell\\Modules\\"
    },
    {
        "Name":  "SystemDrive",
        "Value":  "C:"
    },
    {
        "Name":  "SystemRoot",
        "Value":  "C:\\windows"
    },
    {
        "Name":  "TEMP",
        "Value":  "C:\\windows\\TEMP"
    },
    {
        "Name":  "TMP",
        "Value":  "C:\\windows\\TEMP"
    },
    {
        "Name":  "VERBOSE",
        "Value":  "1"
    },
    {
        "Name":  "windir",
        "Value":  "C:\\windows"
    }
]
carlossscastro commented 2 years ago

Hi @markdavies71

As @paologallinaharbur mentioned, we would need to check the output of the metrics endpoint from the exporter. You can retrieve it either through a browser using the url http://127.0.0.1:9182/metrics or from a powershell running the command:

Invoke-Webrequest -Uri http://127.0.0.1:9182/metrics | Select-Object -ExpandProperty Content

This will dump all the output to the screen. Alternatively you can redirect the output to a file:

Invoke-Webrequest -Uri http://127.0.0.1:9182/metrics | Select-Object -ExpandProperty Content > services.txt

Then grab that services.txt and attach it here

If you want to have a look at the output, each service will produce the following lines of info:

windows_service_info{display_name="New Relic Infrastructure Agent",name="newrelic-infra",process_id="3644",run_as="LocalSystem"} 1
...
windows_service_start_mode{name="newrelic-infra",start_mode="auto"} 1
windows_service_start_mode{name="newrelic-infra",start_mode="boot"} 0
windows_service_start_mode{name="newrelic-infra",start_mode="disabled"} 0
windows_service_start_mode{name="newrelic-infra",start_mode="manual"} 0
windows_service_start_mode{name="newrelic-infra",start_mode="system"} 0
...
windows_service_state{name="newrelic-infra",state="continue pending"} 0
windows_service_state{name="newrelic-infra",state="pause pending"} 0
windows_service_state{name="newrelic-infra",state="paused"} 0
windows_service_state{name="newrelic-infra",state="running"} 1
windows_service_state{name="newrelic-infra",state="start pending"} 0
windows_service_state{name="newrelic-infra",state="stop pending"} 0
windows_service_state{name="newrelic-infra",state="stopped"} 0
windows_service_state{name="newrelic-infra",state="unknown"} 0
...

The above example is for the newrelic-infra service but they need to exist for all the 16 services that you are trying to monitor.

markdavies71 commented 2 years ago

Thanks @carlossscastro

Here are the details of the output through the browser when I specify just windefend in the config.yml and run the command from a powershell prompt

.\windows_exporter.exe --config.file=c:\temp\config.yml

windows_service_info{display_name="Windows Defender Antivirus Service",name="windefend",process_id="3200",run_as="LocalSystem"} 1

windows_service_start_mode{name="windefend",start_mode="auto"} 1
windows_service_start_mode{name="windefend",start_mode="boot"} 0
windows_service_start_mode{name="windefend",start_mode="disabled"} 0
windows_service_start_mode{name="windefend",start_mode="manual"} 0
windows_service_start_mode{name="windefend",start_mode="system"} 0

windows_service_state{name="windefend",state="continue pending"} 0
windows_service_state{name="windefend",state="pause pending"} 0
windows_service_state{name="windefend",state="paused"} 0
windows_service_state{name="windefend",state="running"} 1
windows_service_state{name="windefend",state="start pending"} 0
windows_service_state{name="windefend",state="stop pending"} 0
windows_service_state{name="windefend",state="stopped"} 0
windows_service_state{name="windefend",state="unknown"} 0

windows_service_status{name="windefend",status="degraded"} 0
windows_service_status{name="windefend",status="error"} 0
windows_service_status{name="windefend",status="lost comm"} 0
windows_service_status{name="windefend",status="no contact"} 0
windows_service_status{name="windefend",status="nonrecover"} 0
windows_service_status{name="windefend",status="ok"} 1
windows_service_status{name="windefend",status="pred fail"} 0
windows_service_status{name="windefend",status="service"} 0
windows_service_status{name="windefend",status="starting"} 0
windows_service_status{name="windefend",status="stopping"} 0
windows_service_status{name="windefend",status="stressed"} 0
windows_service_status{name="windefend",status="unknown"} 0

However when I look at the output being generated from nri-winservices I see no reference to windefend

windows_service_info{display_name="Windows Connection Manager",name="wcmsvc",process_id="1888",run_as="NT Authority\\LocalService"} 1
windows_service_info{display_name="Windows Encryption Provider Host Service",name="wephostsvc",process_id="0",run_as="NT AUTHORITY\\LocalService"} 1
alvarocabanas commented 2 years ago

Hello @markdavies71.

We spotted the issue, since the exporter we are using calls the windows Api and tries to collect the services with the rights "SC_MANAGER_ALL_ACCESS", some restricted services can't be opened.

We have opened the following PR to the windows exporter making the services to be opened with "GENERIC_READ", fixing the issue.

https://github.com/prometheus-community/windows_exporter/pull/1036

Once this PR is merged, we will bump the dependency in our integration and generate a new release.

We will let you know as soon as the new release is done.

alvarocabanas commented 2 years ago

Hello, we have just created the prerelease of the nri-winservices beta that is adding the exporter solving the issue. It will still not be added to the infrastructure-agent until next week, we will communicate it here once we do the infrastructure-agent release.

alvarocabanas commented 2 years ago

The last released version of the agent https://github.com/newrelic/infrastructure-agent/releases/tag/1.29.0 is including the release of nri-winservices beta that solves this issue.