cloudwatch-agent pods has prometheus annotations but connection is refused

[henkka]

Issue Description:

The cloudwatch-agent pods launched by the DaemonSet/cloudwatch-agent created by this operator contain the following annotations:

• prometheus.io/scrape: true
• prometheus.io/port: 8888
• prometheus.io/path: /metrics

However, when attempting to connect to :8888/metrics, a "connection refused" error occurs.

Request for Enhancement:

1. Expose Prometheus Metrics: Is it possible to configure cloudwatch-agent to expose Prometheus metrics?
2. Configurable Annotations: If the cloudwatch-agent can be configured to expose Prometheus metrics, please make this behavior configurable. The pod annotations should be added only if this feature is enabled. Otherwise, these annotations should not be added to avoid confusion and potential connection errors.

https://github.com/aws/amazon-cloudwatch-agent-operator/blame/main/internal/manifests/collector/annotations.go#L18-L21

[lisguo]

Hello, we are aware of a feature gap with prometheus. There is a related github issue in the helm chart: https://github.com/aws-observability/helm-charts/issues/55

[krisko]

Hello, we are aware of a feature gap with prometheus. There is a related github issue in the helm chart: aws-observability/helm-charts#55

Hello @lisguo, I think that this issue is not about being able to scrape prometheus metrics with the agent. It's about that the agent has prometheus annotations and should expose metrics via metrics endpoint. But that endpoint is not available. Because of that we are getting alerts about unreachable metrics endpoint on the agent.