DataDog / integrations-extras

Community developed integrations and plugins for the Datadog Agent.
BSD 3-Clause "New" or "Revised" License
252 stars 734 forks source link

redis_sentinel fails to complete successfully due to AUTH error #2395

Open melissa-hale opened 4 months ago

melissa-hale commented 4 months ago

Output of the info page

When running the redis_sentinel integration to monitor a group of Redis Sentinel servers, the check fails with the following error:

2024-05-24 22:57:33 UTC | CORE | WARN | (pkg/collector/python/datadog_agent.go:131 in LogMessage) | redis_sentinel:e8118be2e61e206d | (redis_sentinel.py:42) | Error collecting metrics for master myprimary: AUTH <password> called without any password configured for the default user. Are you sure your configuration is correct?

This is my conf.yaml:

init_config:

instances:
  - sentinel_host: ${{SENTINEL1_PRIVATE_DOMAIN}}
    sentinel_port: 26379
    sentinel_password: ${{REDIS_PRIMARY_PASSWORD}}
    masters:
      - ${{REDIS_PRIMARY_NAME}}
    tags:
      - instance:redis_sentinel_1
  - sentinel_host: ${{SENTINEL2_PRIVATE_DOMAIN}}
    sentinel_port: 26379
    sentinel_password: ${{REDIS_PRIMARY_PASSWORD}}
    masters:
      - ${{REDIS_PRIMARY_NAME}}
    tags:
      - instance:redis_sentinel_2
  - sentinel_host: ${{SENTINEL3_PRIVATE_DOMAIN}}
    sentinel_port: 26379
    sentinel_password: ${{REDIS_PRIMARY_PASSWORD}}
    masters:
      - ${{REDIS_PRIMARY_NAME}}
    tags:
      - instance:redis_sentinel_3

Additional environment details (Operating System, Cloud provider, etc):

Steps to reproduce the issue:

  1. Deploy a Redis replica set with Sentinel (followed this guide)
  2. Deploy a DataDog agent
  3. Install integtation: datadog-agent integration install -t datadog-redis_sentinel==1.1.0 (or 1.1.1) in agent container
  4. Configure the redis_sentinel check similar to above and restart agent
  5. Observe error upon check execution

Describe the results you received: Received an AUTH error during check execution

Describe the results you expected: For the check execution to complete successfully

Additional information you deem important (e.g. issue happens only occasionally): I was able to fix this in the redis_sentinel.py by passing a username of "default" when initializing the redis connection with redis.StrictRedis, e.g. -

redis_conn = redis.StrictRedis(host=host, port=port, username="default", password=password, db=0)

So I believe this is related to the implementation of ACLs after Redis version 5. According to this doc, you can configure Redis to function the same as it did prior to the change, but I haven't been successful in getting any configuration changes on Redis side to work.

Happy to accept this is an issue I need to resolve in my Redis configuration, but in case others run into this, I figured it would be helpful to have a record either way.

Thanks!