search

DataDog / integrations-extras

Community developed integrations and plugins for the Datadog Agent.
BSD 3-Clause "New" or "Revised" License
254 stars 750 forks source link

gnatsd_streaming bug: class GnatsdStreamingConfig's self.url is looking for the non-existing "/streaming" endpoint #2527

Open donghoang89 opened 3 weeks ago

donghoang89 commented 3 weeks ago

See https://github.com/DataDog/integrations-extras/blob/master/gnatsd_streaming/datadog_checks/gnatsd_streaming/gnatsd_streaming.py#L17, self.url is looking for the "/streaming" endpoint.

However, this endpoint does not exist on nats monitor. When I hit my nats monitor port at 8222, I am seeing the following endpoints

         </div>
    <br/>
    <a href=./varz>General</a>
    <a href=./jsz>JetStream</a>
    <a href=./connz>Connections</a>
    <a href=./accountz>Accounts</a>
    <a href=./accstatz>Account Stats</a>
    <a href=./subsz>Subscriptions</a>
    <a href=./routez>Routes</a>
    <a href=./leafz>LeafNodes</a>
    <a href=./gatewayz>Gateways</a>
    <a href=./healthz class=last>Health Probe</a>

Nats monitoring document at https://github.com/nats-io/nats.docs/blob/master/running-a-nats-service/nats_admin/monitoring/readme.md#jetstream-information also corroborate this finding. It only lists the 10 above-mentioned endpoints as the available endpoints. Endpoint "/streaming" is nowhere to be seen.

I am trying to install and configure gnatsd_streaming integration on my datadog cluster to monitor my nats cluster stream metrics. Both clusters are running on AWS EKS. However, I keep running into the following error:

    gnatsd_streaming (1.0.0)
    ------------------------
      Instance ID: gnatsd_streaming:1f2b9dc634362286 [ERROR]
      Configuration Source: container:containerd://0c81d50fbd27638df5e37df69b2d8d6961de216e8d11466b774a4f20fb72a802
      Total Runs: 1,012
      Metric Samples: Last Run: 0, Total: 0
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 1, Total: 1,012
      Average Execution Time : 3ms
      Last Execution Date : 2024-10-30 02:21:53 UTC (1730254913000)
      Last Successful Execution Date : Never
      Error: Non 200 response from NATS monitor port
      Traceback (most recent call last):
        File "/opt/datadog-agent/embedded/lib/python3.12/site-packages/datadog_checks/base/checks/base.py", line 1298, in run
          self.check(instance)
        File "/opt/datadog-agent/embedded/lib/python3.12/site-packages/datadog_checks/gnatsd_streaming/gnatsd_streaming.py", line 160, in check
          GnatsdStreamingCheckInvocation(instance, self).check()
        File "/opt/datadog-agent/embedded/lib/python3.12/site-packages/datadog_checks/gnatsd_streaming/gnatsd_streaming.py", line 57, in check
          self._status_check()
        File "/opt/datadog-agent/embedded/lib/python3.12/site-packages/datadog_checks/gnatsd_streaming/gnatsd_streaming.py", line 77, in _status_check
          raise e
        File "/opt/datadog-agent/embedded/lib/python3.12/site-packages/datadog_checks/gnatsd_streaming/gnatsd_streaming.py", line 71, in _status_check
          raise ValueError('Non 200 response from NATS monitor port')
      ValueError: Non 200 response from NATS monitor port

The error message didn't give me much info as to what is wrong/missing with my gnatsd_streaming integration configuration. So, I went through the gnatsd_streaming.py file hoping to find some clue. That's when I noticed that gnatsd_streaming check is looking for the "/streaming" endpoint (instead of the "/jsz" endpoint in this case?)

SPEC: AWS EKS version: 1.29 Datadog agent verison: Agent 7.58.1 - Commit: 6f52a65 - Serialization version: v5.0.130 - Go version: go1.22.7 Nats-server version: v2.10.4 Gnatsd_streaming integration version: 1.0.0

donghoang89 commented 3 weeks ago

@stephenprater @jaredhoyt I can confirm that I have jetstream enabled. However the "/streaming" endpoint does not exist.