Upgrade from 8.7.1 -> 8.8.2 No longer getting RDS instances.

[m-standfuss]

Community forum post https://discuss.elastic.co/t/lost-aws-module-rds-metricset-metrics-after-upgrade-from-7-17-8-8-2/343388/4

8.8.2 Debug logs 8_8_2_logs.csv

8.7.1 Debug logs

Configuration

• Running in Elasticsearch Cloud
• AWS RDS (postgres if that matters) instances
• metricbeats is running in EKS cluster

metricbeat.yml

    metricbeat.config.modules:
      path: ${path.config}/modules.d/*.yml
      reload.enabled: true
    processors:
      - add_cloud_metadata:
      - add_kubernetes_metadata:
      - add_fields:
          target: orchestrator.cluster
          fields:
            name: ${CLUSTER}
    cloud.id: ${ELASTIC_CLOUD_ID}
    cloud.auth: ${ELASTIC_CLOUD_AUTH}
    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}
    setup.dashboards.enabled: false

aws_rds.yml

    module: aws
      period: 60s
      metricsets:
        - rds

Using a k8s service account to auth back into AWS

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: "arn:aws:iam::729154041854:role/terraform-20220317204940606500000001"
  name: metricbeat
  namespace: kube-system
  labels:
    k8s-app: metricbeat

IAM Permissions tied to that role match the permissions needed in the documentation for the aws module.

We have more configs in our production yaml but I confirmed its an issue with this basic one and just flipping the image version on the deployment back and forth

For confirmed bugs, please report:

• Version: 8.8.2 (confirmed working in 8.7.1)
• Operating System: EKS using this image docker.elastic.co/beats/metricbeat:8.8.2
• Discuss Forum URL: https://discuss.elastic.co/t/lost-aws-module-rds-metricset-metrics-after-upgrade-from-7-17-8-8-2/343388/4
• Steps to Reproduce: Have a RDS instance in us-east-1 enable the aws module and rds metricset, no metrics are collected.

[botelastic[bot]]

This issue doesn't have a Team:<team> label.

[botelastic[bot]]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!