cloudspannerecosystem / autoscaler

Automatically scale the capacity of your Spanner instances based on their utilization.
Apache License 2.0
86 stars 33 forks source link

Ignore Bad Values from Google Metrics #355

Open alexlo03 opened 1 month ago

alexlo03 commented 1 month ago

Hello,

Google Metrics is having an outage (https://status.cloud.google.com/incidents/ERzzrJqeGR2GCW51XKFv#RP1d9aZLNFZEJmTBk8e1) and is returning 0s for some metric queries.

Examples:

high_priority_cpu=0%, BELOW the range [50%-60%] => however, cannot scale to 0 because it is lower than MIN 2000 PROCESSING_UNITS
rolling_24_hr=0%, BELOW the range [85%-95%] => however, cannot scale to 0 because it is lower than MIN 1000 PROCESSING_UNITS
storage=0%, BELOW the range [70%-80%] => however, cannot scale to 0 because it is lower than MIN 100 PROCESSING_UNITS

0 is never a real value for any of these metrics. the poller/scaler should just not proceed.

alexlo03 commented 1 month ago

I will amend to note that storage can be zero. The CPU metrics still cannot be from what I can tell.