saarw-opti
commented
5 months ago Same here; I don't know why it's not using the service account, and it prefers to use the Node role.
Open b-lancaster opened 6 months ago
saarw-opti
commented
5 months ago Same here; I don't know why it's not using the service account, and it prefers to use the Node role.
Ca-moes
commented
5 months ago Same here, on a fresh deployment using https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus-cloudwatch-exporter
Restarting the Deployment does not fix it. It uses the Karpenter role
Context information
Exporter configuration
```yaml apiVersion: v1 kind: ConfigMap metadata: name: cloudwatch-monitoring-general namespace: monitoring data: config.yml: | --- region: us-east-1 delay_seconds: 0 set_timestamp: false use_get_metric_data: true metrics: - aws_namespace: AWS/Lambda aws_metric_name: Errors aws_dimensions: [FunctionName] aws_statistics: [Sum] - aws_namespace: AWS/Lambda aws_metric_name: Invocations aws_dimensions: [FunctionName] aws_statistics: [Sum] - aws_namespace: AWS/Lambda aws_metric_name: Duration aws_dimensions: [FunctionName] aws_statistics: [Average] - aws_namespace: AWS/Lambda aws_metric_name: Throttles aws_dimensions: [FunctionName] aws_statistics: [Sum] - aws_namespace: AWS/Lambda aws_metric_name: OffsetLag aws_dimensions: [FunctionName] aws_statistics: [Maximum] - aws_namespace: AWS/ES aws_metric_name: ThreadpoolIndexQueue aws_dimensions: [ClientId, DomainName] aws_extended_statistics: [p95] - aws_namespace: AWS/ES aws_metric_name: ThreadpoolWriteQueue aws_dimensions: [ClientId, DomainName] aws_extended_statistics: [p95] - aws_namespace: AWS/ES aws_metric_name: ThreadpoolSearchQueue aws_dimensions: [ClientId, DomainName] aws_extended_statistics: [p95] - aws_namespace: AWS/ES aws_metric_name: ThreadpoolIndexQueue aws_dimensions: [ClientId, DomainName] aws_statistics: [Average] - aws_namespace: AWS/ES aws_metric_name: ThreadpoolWriteQueue aws_dimensions: [ClientId, DomainName] aws_statistics: [Average] - aws_namespace: AWS/ES aws_metric_name: ThreadpoolSearchQueue aws_dimensions: [ClientId, DomainName] aws_statistics: [Average] - aws_namespace: AWS/ES aws_metric_name: WriteLatency aws_dimensions: [ClientId, DomainName] aws_extended_statistics: [p95] - aws_namespace: AWS/ES aws_metric_name: WriteLatency aws_dimensions: [ClientId, DomainName] aws_statistics: [Average] - aws_namespace: AWS/ES aws_metric_name: ReadLatency aws_dimensions: [ClientId, DomainName] aws_extended_statistics: [p95] - aws_namespace: AWS/ES aws_metric_name: SearchLatency aws_dimensions: [ClientId, DomainName] aws_extended_statistics: [p95] - aws_namespace: AWS/ES aws_metric_name: SearchLatency aws_dimensions: [ClientId, DomainName] aws_statistics: [Average] - aws_namespace: AWS/ES aws_metric_name: IndexingLatency aws_dimensions: [ClientId, DomainName] aws_extended_statistics: [p95] - aws_namespace: AWS/ES aws_metric_name: IndexingLatency aws_dimensions: [ClientId, DomainName] aws_statistics: [Average] - aws_namespace: AWS/ES aws_metric_name: IndexingRate aws_dimensions: [ClientId, DomainName] aws_statistics: [Average] - aws_namespace: AWS/ES aws_metric_name: IndexingRate aws_dimensions: [ClientId, DomainName] aws_extended_statistics: [p95] - aws_namespace: AWS/ES aws_metric_name: SearchRate aws_dimensions: [ClientId, DomainName] aws_statistics: [Average] - aws_namespace: AWS/ES aws_metric_name: SearchRate aws_dimensions: [ClientId, DomainName] aws_extended_statistics: [p95] - aws_namespace: AWS/ES aws_metric_name: 5xx aws_dimensions: [ClientId, DomainName] aws_statistics: [Sum] - aws_namespace: AWS/ES aws_metric_name: 2xx aws_dimensions: [ClientId, DomainName] aws_statistics: [Sum] - aws_namespace: AWS/ES aws_metric_name: 3xx aws_dimensions: [ClientId, DomainName] aws_statistics: [Sum] - aws_namespace: AWS/ES aws_metric_name: 4xx aws_dimensions: [ClientId, DomainName] aws_statistics: [Sum] - aws_namespace: AWS/ES aws_metric_name: ClusterStatus.red aws_dimensions: [ClientId, DomainName] aws_statistics: [Maximum] - aws_namespace: AWS/ES aws_metric_name: ClusterStatus.yellow aws_dimensions: [ClientId, DomainName] aws_statistics: [Maximum] - aws_namespace: AWS/ES aws_metric_name: ClusterIndexWritesBlocked aws_dimensions: [ClientId, DomainName] aws_statistics: [Average] - aws_namespace: AWS/ES aws_metric_name: Nodes aws_dimensions: [ClientId, DomainName] aws_statistics: [Minimum] - aws_namespace: AWS/ES aws_metric_name: AutomatedSnapshotFailure aws_dimensions: [ClientId, DomainName] aws_statistics: [Maximum] - aws_namespace: AWS/ES aws_metric_name: KibanaHealthyNodes aws_dimensions: [ClientId, DomainName] aws_statistics: [Minimum] - aws_namespace: AWS/ES aws_metric_name: CPUUtilization aws_dimensions: [ClientId, DomainName] aws_statistics: [Average] - aws_namespace: AWS/ES aws_metric_name: FreeStorageSpace aws_dimensions: [ClientId, DomainName] aws_statistics: [Minimum] - aws_namespace: AWS/ES aws_metric_name: JVMMemoryPressure aws_dimensions: [ClientId, DomainName] aws_statistics: [Maximum] ```Service Account
```yaml apiVersion: v1 kind: ServiceAccount metadata: labels: app.kubernetes.io/name: cloudwatch-exporter name: cloudwatch-exporter namespace: monitoring annotations: eks.amazonaws.com/role-arn: arn:aws:iam::IAM Role
```yaml # Role for cloudwatch metrics exporter rCloudWatchMetricsReadOnlyRole: Type: 'AWS::IAM::Role' Properties: AssumeRolePolicyDocument: !Sub - | { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "${IamOidcProviderArn}" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "${OidcProviderEndpoint}:sub": "system:serviceaccount:monitoring:cloudwatch-exporter" } } } ] } - { "IamOidcProviderArn": !Ref pOidcProviderArn, "OidcProviderEndpoint": !Ref pIssuerHostPath } Path: / ManagedPolicyArns: - arn:aws:iam::aws:policy/CloudWatchReadOnlyAccess ```Deployment
```yaml apiVersion: apps/v1 kind: Deployment metadata: name: cloudwatch-metrics-exporter-general labels: app.kubernetes.io/name: cloudwatch-metrics-exporter app.kubernetes.io/instance: cloudwatch-metrics-exporter-general namespace: monitoring annotations: reloader.stakater.com/auto: "true" spec: replicas: 1 selector: matchLabels: app.kubernetes.io/name: cloudwatch-metrics-exporter app.kubernetes.io/instance: cloudwatch-metrics-exporter-general template: metadata: labels: app.kubernetes.io/name: cloudwatch-metrics-exporter app.kubernetes.io/instance: cloudwatch-metrics-exporter-general spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: k8s.swacorp.com/instancegroup operator: In values: - operations-job-nodes - arm-operations-job-nodes - key: kubernetes.io/arch operator: In values: - arm64 - amd64 preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 preference: matchExpressions: - key: kubernetes.io/arch operator: In values: - arm64 tolerations: - effect: NoExecute key: k8s.swacorp.com/dedicated operator: Equal value: operations-server - effect: NoSchedule key: kubernetes.io/arch operator: Equal value: arm64 serviceAccountName: cloudwatch-exporter containers: - name: cloudwatch-metrics-exporter image: quay.io/prometheus/cloudwatch-exporter:v0.15.5 ports: - containerPort: 9106 resources: requests: cpu: 100m memory: 600Mi volumeMounts: - mountPath: /config name: cloudwatch-metric-general volumes: - configMap: name: cloudwatch-monitoring-general name: cloudwatch-metric-general ```Exporter logs
```log Mar 15, 2024 4:15:46 PM io.prometheus.cloudwatch.CloudWatchCollector collect WARNING: CloudWatch scrape failed software.amazon.awssdk.services.cloudwatch.model.CloudWatchException: User: arn:aws:sts::What do you expect to happen?
I expected the cloudwatch-exporter to use the attached service-account with the permissions necessary to retrieve metric data.
What happened instead?
What actually happened was the cloudwatch-exporter stopped using the service-account and tried to use the k8s nodes IAM role. Nothing changed, we just stopped recieving the metrics in prometheus and then found the logs.
Restarting the deployment fixed the problem and it started using the service-account again, but the problem is if this would have happened in a production environment, the prometheus alerts we've setup to monitor these metrics wouldnt have met the threshold needed to fire.
Also, without looking at the logs, the pod appeared to be running as normal.