Support for short lived dynamic credentials

I have a use case where I would like to be able to use short lived dynamic credentials with postgres_exporter.

More specifically I would like to able to authenticate to RDS using IAM tokens. This pretty much what is done in #453 and also reported in #326.

I am not sure developing something specifically for this use case would be the most beneficial to the community. I think it would be nice to have a generic solution to load credentials generated externally (which is already possible by loading username and password from files) but also to reload them as they get refreshed (this is the part I am currently missing).

This would allow supporting many use case like:

reloading credentials in kubernetes when the secret is updated without restarting the pod
using a separate app (or sidecar) to generate and refresh credentials from many sources (hashicorp vault, IAM, AWS secret manager...)

I think it could be very quickly and naively implemented by re-reading the username and password files each time the metrics are scraped (it should be fairly fast and that should not happen very often). It could be implemented a bit smarter by watching those files and only refreshing credentials as needed.

Is there any interest for such a feature? If there is I could find some time to work on it as it looks quite straightforward.

I managed to resolve this issue with a different approach.

Postgres_exporter uses lib/pq which supports postgresql passfile. Using the passfile allows to use an empty password in the postgres_exporter configuration and have an external file for the credentials (see https://www.postgresql.org/docs/9.1/libpq-pgpass.html) but unlike DATA_SOURCE_PASS_FILE, the postgresql passfile is reprocessed each time a new connection is established to the database.

My solution is quite simple: I configure an empty password in postgres_exporter and I use a sidecar container to periodically write the passfile (I run the exporter in EKS but if running outside of Kubernetes, a simple crontab would do the job). Keep in mind the passfile should only be readable by the user (not group or world readable) to a file in a shared volume and in the postgres_exporter container I set the PGPASSFILE variable to point to the generated passfile. Also, since : is used as a field separator in the file, any : in the generated credentials must be escaped with \.

Here is what my deployment looks like:

kind: Deployment
metadata:
  name: postgres-exporter
...
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres-exporter
  template:
    metadata:
      labels:
        app: postgres-exporter
    spec:
      securityContext:
        runAsUser: 1234 # Ensures both container run as the same user
        fsGroup: 4567 # Ensures both container can write to the shared volume
      containers:
      - name: app
        image: quay.io/prometheuscommunity/postgres-exporter:v0.9.0
...
        env:
          - name: PGPASSFILE
            value: "/postgres/.pgpass"
...
        volumeMounts:
          - name: shared-volume
            mountPath: /postgres
      - name: auth
        image: amazon/aws-cli:2.2.2
        command:
          - /bin/bash
        args:
          - "-c"
          - |
            while true; do
              aws rds generate-db-auth-token --hostname <database>.rds.amazonaws.com --port 5432 --username <some user> | sed "s/:/\\\:/g" | awk "{ print \"<database>.rds.amazonaws.com:5432:*:<some user>:\" \$1}" > /postgres/.pgpass && chmod 600 /postgres/.pgpass # NB the : character in the generated credentials must be escaped with a \
              sleep 300
            done
        volumeMounts:
        - name: shared-volume
          mountPath: /postgres
...
      volumes:
      - name: shared-volume
        emptyDir: {}

prometheus-community / postgres_exporter

Support for short lived dynamic credentials #498