Adding IO manager metrics for AWS

[olivierdupuis]

Getting metrics from IO Manager. Currently limited to S3 storage service and only getting the size metric for a non-partitioned asset. Goal is to get a minimum working solution and then expand.

Config is defined in ~/.damn/connectors.yml file:

io-manager:
  aws:
    credentials:
      access_key_id: "{{ env('AWS_ACCESS_KEY_ID') }}"
      secret_access_key: "{{ env('AWS_SECRET_ACCESS_KEY') }}"
      region: "us-east-1"
    bucket_name: "discursus-io"
    key_prefix: "platform"

Credentials for AWS are set in environment variables.

Once configured, when running the metrics command for an asset, you should now get that additional stored file size metric.

Command: damn metrics gdelt/gdelt_gkg_articles

Results:

[olivierdupuis]

10 - Extract metrics from S3