[BUG] S3 bucket metrics are AWS 'region' dependent "NaN"

[ryanpxyz]

Is there an existing issue for this?

• [X] I have searched the existing issues

Current Behavior

Here is a snippet of data that is presented to our Prometheus for scraping:

aws_s3_bucket_size_bytes_sum{account_id="123456789012",dimension_BucketName="ic-xxxx",dimension_StorageType="StandardStorage",name="arn:aws:s3:::ic-xxxx",region="eu-central-1"} NaN
aws_s3_bucket_size_bytes_sum{account_id="123456789012",dimension_BucketName="id-aaaa",dimension_StorageType="StandardStorage",name="arn:aws:s3:::id-aaaa",region="eu-west-1"} 8.58689413e+08
aws_s3_bucket_size_bytes_sum{account_id="123456789012",dimension_BucketName="id-bbbb",dimension_StorageType="StandardStorage",name="arn:aws:s3:::id-bbbb",region="eu-central-1"} NaN
aws_s3_bucket_size_bytes_sum{account_id="123456789012",dimension_BucketName="id-cccc",dimension_StorageType="StandardStorage",name="arn:aws:s3:::id-cccc",region="us-east-1"} 7140`

• We see currently that eu-central-1 resident buckets are presented with "NaN" values.
• We see identical behaviour for other S3 metrics e.g. aws_s3_number_of_objects_maximum.
• We see consistent and correct values & content for all aws_s3_info bucket metrics.

Expected Behavior

All discovered buckets in all regions should present real numbers for scraping by Prometheus.

Steps To Reproduce

• I am working with the following configuration:

  - type: s3
    regions:
      - eu-central-1
      - eu-west-1
      - eu-west-2
      - us-east-1
      - us-east-2
    addCloudwatchTimestamp: false
    period: 86400
    length: 259200
    metrics:
      - name: NumberOfObjects
        statistics:
          - Maximum
        additionalDimensions:
          - name: StorageType
            value: AllStorageTypes
      - name: BucketSizeBytes
        statistics:
          - Sum
        additionalDimensions:
          - name: StorageType
            value: StandardStorage
      - name: BytesUploaded
        statistics:
          - Average
        period: 60
        length: 300
        additionalDimensions:
          - name: FilterId
            value: EntireBucket

• I have reproduced this behaviour with the following YACE versions:

  • "v0.42.0-alpha"
  • "v0.38.0-alpha"
  • "v0.33.0-alpha"

• I have added debug to logging and can confirm that at least in the logs real number data is extracted from AWS CloudWatch - here is a snippet:

  
  {\n      Id: \"id_7316112639016562625\",\n      Label: \"ic-xxxx AllStorageTypes NumberOfObjects\",\n      StatusCode: \"PartialData\",\n      Timestamps: [2022-11-13 00:00:00 +0000 UTC],\n      Values: [299]\n    }
  {\n      Id: \"id_2276924059018283294\",\n      Label: \"ic-xxxx StandardStorage BucketSizeBytes\",\n      StatusCode: \"PartialData\",\n      Timestamps: [2022-11-13 00:00:00 +0000 UTC],\n      Values: [3.075721e+06]\n    }
  {\n      Id: \"id_7316112639016562625\",\n      Label: \"ic-xxxx AllStorageTypes NumberOfObjects\",\n      StatusCode: \"Complete\"\n    }
  {\n      Id: \"id_2276924059018283294\",\n      Label: \"ic-xxxx StandardStorage BucketSizeBytes\",\n      StatusCode: \"Complete\"\n    }

{\n Id: \"id_7316112639016562625\",\n Label: \"ic-xxxx AllStorageTypes NumberOfObjects\",\n StatusCode: \"PartialData\"\n } {\n Id: \"id_2276924059018283294\",\n Label: \"ic-xxxx StandardStorage BucketSizeBytes\",\n StatusCode: \"PartialData\"\n } {\n Id: \"id_7316112639016562625\",\n Label: \"ic-xxxx AllStorageTypes NumberOfObjects\",\n StatusCode: \"PartialData\"\n } {\n Id: \"id_2276924059018283294\",\n Label: \"ic-xxxx StandardStorage BucketSizeBytes\",\n StatusCode: \"PartialData\"\n } {\n Id: \"id_7316112639016562625\",\n Label: \"ic-xxxx AllStorageTypes NumberOfObjects\",\n StatusCode: \"PartialData\",\n Timestamps: [2022-11-14 00:00:00 +0000 UTC],\n Values: [299]\n } {\n Id: \"id_2276924059018283294\",\n Label: \"ic-xxxx StandardStorage BucketSizeBytes\",\n StatusCode: \"PartialData\",\n Timestamps: [2022-11-14 00:00:00 +0000 UTC],\n Values: [3.075721e+06]\n }

### Anything else?

* I have tried adding, removing & re-adding tags to the affected buckets.
* I have verified the IAM role as being correct.
* I have played with the AWS CloudWatch statistics (sum, maximum etc) and this didn't produce any change.

[ryanpxyz]

Any ideas, updates? Or is this 'cancelled due to lack of interest'? :-)

[oladuwek]

I have also faced this bug. Is there any workaround or information on how to fix it?

config:

apiVersion: v1alpha1
discovery:
  jobs:
    - type: AWS/S3
      regions: [us-west-1]
      searchTags:
        - key: Name
          value: core-archive-prod
      period: 60
      length: 60
      metrics:
        - name: AllRequests
          statistics: [Sum]

root@ip:~# curl -s localhost:9201/metrics | grep s3
# HELP aws_s3_all_requests_sum Help is not implemented yet.
# TYPE aws_s3_all_requests_sum gauge
aws_s3_all_requests_sum{account_id="123456789",dimension_BucketName="core-archive-prod",dimension_FilterId="test-status",name="arn:aws:s3:::core-archive-prod",region="us-west-1"} NaN
# HELP aws_s3_info Help is not implemented yet.
# TYPE aws_s3_info gauge
aws_s3_info{name="arn:aws:s3:::core-archive-prod",tag_Name="core-archive-prod"} 0

[morremeyer]

Can you check again with v0.61.0 or newer? We faced the same issue, which has been resolved with v0.61.0.