Open StevenACoffman opened 2 years ago
While I can see that in my GKE Autopilot clusters the event-exporter-gke
pods are running in the kube-system
namespace, there does not appear to be a pre-installed custom-metrics-stackdriver-adapter.
When installing in GKE Autopilot, you do not need need to, and in fact you cannot use node iam.gke.io/gke-metadata-server-enabled: "true"
. Otherwise the directions for how to Use Workload Identity are correct
See GoogleCloudPlatform/k8s-stackdriver#315 and GoogleCloudPlatform/kubernetes-engine-samples#140
gcloud iam service-accounts create custom-metrics-sd-adapter --project "$GCP_PROJECT_ID"
gcloud projects add-iam-policy-binding "$GCP_PROJECT_ID" \
--member "serviceAccount:custom-metrics-sd-adapter@$GCP_PROJECT_ID.iam.gserviceaccount.com" \
--role "roles/monitoring.editor"
gcloud iam service-accounts add-iam-policy-binding \
--role roles/iam.workloadIdentityUser \
--member "serviceAccount:$GCP_PROJECT_ID.svc.id.goog[custom-metrics/custom-metrics-stackdriver-adapter]" \
"custom-metrics-sd-adapter@$GCP_PROJECT_ID.iam.gserviceaccount.com"
kubectl create -f https://raw.githubusercontent.com/GoogleCloudPlatform/k8s-stackdriver/master/custom-metrics-stackdriver-adapter/deploy/production/adapter.yaml
kubectl annotate serviceaccount custom-metrics-stackdriver-adapter \
"iam.gke.io/gcp-service-account=custom-metrics-sd-adapter@$GCP_PROJECT_ID.iam.gserviceaccount.com" \
--namespace custom-metrics
same problem. clearly Autopilot cluster supports the external metrics from Cloud Monitoring -- it even has UI dialog to set them. Though when you try to create a HPA object using the official example https://cloud.google.com/kubernetes-engine/docs/tutorials/autoscaling-metrics#pubsub
it fails:
$ kubectl describe horizontalpodautoscalers.autoscaling
Name: myservice
Namespace: default
Labels: <none>
Annotations: <none>
CreationTimestamp: Mon, 12 Jun 2023 18:50:12 +0300
Reference: Deployment/myservice
Metrics: ( current / target )
"pubsub.googleapis.com|subscription|num_undelivered_messages" (target average value): <unknown> / 2
Min replicas: 1
Max replicas: 5
Deployment pods: 1 current / 1 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True ReadyForNewScale recommended size matches current size
ScalingActive False FailedGetExternalMetric the HPA was unable to compute the replica count: unable to get external metric default/pubsub.googleapis.com|subscription|num_undelivered_messages/nil: unable to fetch metrics from external metrics API: the server could not find the requested resource (get pubsub.googleapis.com|subscription|num_undelivered_messages.external.metrics.k8s.io)
ScalingLimited False DesiredWithinRange the desired count is within the acceptable range
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedGetExternalMetric 1s horizontal-pod-autoscaler unable to get external metric default/pubsub.googleapis.com|subscription|num_undelivered_messages/nil: unable to fetch metrics from external metrics API: the server could not find the requested resource (get pubsub.googleapis.com|subscription|num_undelivered_messages.external.metrics.k8s.io)
I'm pretty sure that the metric exists -- I can see it in Cloud Monitoring
ok. it works after installing custom metric adapter. I thought it should be preinstalled
kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/k8s-stackdriver/master/custom-metrics-stackdriver-adapter/deploy/production/adapter_new_resource_model.yaml
In the tutorial https://cloud.google.com/kubernetes-engine/docs/tutorials/autoscaling-metrics#pubsub it is unclear whether the steps are applicable to an GKE Autopilot cluster.
For instance, is it necessary to install the custom-metrics-stackdriver-adapter in an GKE Autopilot cluster?