Using IAM user authentication instead of assume role arn and default ~/.aws/credentials

[bryanfang]

Is your feature request related to a problem? Please describe

prometheus-rds-exporter is nice, and I created service accounts for several aws accounts because have other use cases, for other cases, I can use IAM user (service account) credentials to invoke boto3 APIs, so I want to reuse the credentials for prometheus-rds-exporter, however the doc is not clear enough on configuring aws credentials especially on configuring multiple aws accounts.

Describe the solution you'd like

I'd like a solution similar to ec2_sd_scrapes, I just need provide region and different endpoint and aws credentials via configuration

  - job_name: 'tagged-ec2-nodes'
    scrape_interval: 30s
    ec2_sd_configs:
    - region: us-east-1
    - region: eu-central-1
    - region: ap-northeast-1
    - region: cn-north-1
      endpoint: https://ec2.cn-north-1.amazonaws.com.cn
      access_key: $AWS_CN_ACCESS_KEY_ID
      secret_key: $AWS_CN_SECRET_ACCESS_KEY
      port: 9100

and I see rds_exporter has such feature for multiple instances:

---
instances:
  - region: us-east-1
    instance: rds-aurora1

  - region: us-east-1
    instance: rds-mysql57
    aws_access_key: AKIAIOSFODNN7EXAMPLE
    aws_secret_key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
    aws_role_arn: arn:aws:iam::76784568345:role/my-role
    disable_basic_metrics: true
    disable_enhanced_metrics: false
    labels:
      foo: bar
      baz: qux

so I am seeking a doc to provide the same configuration for prometheus-rds-exporter

Describe alternatives you've considered

if it's not yet implemented, would you please share more details on enhancement point if I want to implement such feature?

Additional context

NA

[vmercierfr]

I understand that you would like to have a way to collect metrics of several AWS accounts/regions within in the same Prometheus RDS exporter deployment. As you mentionned, this is not yet possible, but that's something we would like to support in the future. The Percona's RDS exporter is probably a good alternative for this usecase for now.

Implementation will require bunch of changes because we need to pass AWS credentials and exporter settings per "Prometheus scrape".

I guess the best approach will be to support parameters in Prometheus RDS exporter URLs and leverage on Prometheus params parameter of scrape_config to pass parameters during scraping.

This is much more flexible than static configuration file and more relevant in cloud/container era. So we may image something like:

scrape_configs:
  - job_name: prometheus-rds-exporter-cn
    metrics_path: /metrics
    params:
      aws_role_arn: arn:aws:iam::76784568345:role/my-role
      aws_region: cn-north-1
      collect-quotas: false
      ...
    ...
  - job_name: prometheus-rds-exporter-us
    metrics_path: /metrics
    params:
      aws_role_arn: arn:aws:iam::12345677890:role/my-role
      aws_region: us-east-1
      collect-quotas: true
      ...
    ...

Macro steps:

• Support parameters in Prometheus RDS exporter URL
• Change AWS session handling to assume role/credentials specified in URL

We can't implement it right know ourself, but we are open to contribution (See CONTRIBUTING.md). If you are interested to implement it, let me know, we can schedule an "onboarding session" on the code base.

[bryanfang]

@vmercierfr thanks for the tips, I changed the code in my local, after making the change, the configuration part would be

#
# AWS credentials
#
accounts:
- aws_access_key_id: <Hashed>
  aws_secret_access_key: <Hashed>
  regions: 
  - cn-north-1
- aws_access_key_id: <Hashed>
  aws_secret_access_key:<Hashed>
  regions: 
  - ap-northeast-1
  - eu-central-1
  - us-east-1

it works fine for all regions from AWS Global, however it doesn't work for AWS China, I suspect the endpoint needs to be changed accordingly, since I am newbie to Golang, need some time to fix it. BTW, it's backward compatible!

[bryanfang]

@vmercierfr could you please review this PR https://github.com/qonto/prometheus-rds-exporter/pull/140, I tested the change in my local, it's possible to retrieve metrics across account and region, not sure if there is potential issue, since I am new to Golang, thanks!

[xucq07]

Is there any progress on this issue?

[qfritz]

Vincent's answer is the most up to date:

Implementation will require bunch of changes because we need to pass AWS credentials and exporter settings per "Prometheus scrape". We can't implement it right know ourself, but we are open to contribution.