search

GoogleCloudPlatform / k8s-stackdriver

Apache License 2.0
391 stars 212 forks source link

How do you configure custom-metrics-stackdriver-adapter to expose external metrics from a different project? #411

Open karlkfi opened 3 years ago

karlkfi commented 3 years ago

The custom-metrics-stackdriver-adapter exposes Stackdriver metrics as Kubernetes Prometheus metrics using the External Metrics API.

But AFAICK, when deployed on GKE, the adapter only exposes metrics from the same project the GKE cluster is in.

I assume the adapter is configurable, and doesn't require using GKE, which means the project ID should probably be configurable, but I can't figure out how to configure it.

  1. Is it even possible to configure the project ID?
  2. Is it possible to monitor multiple projects with the same adapter?
  3. If the adapter can only monitor one project, can multiple adapters by used in parallel to monitor multiple projects?

My use case is that on a multi-tenant GKE, each tenant has their own GCP Project, where they manage their SaaS, like databases and pubsub. So I want to be able to use custom metrics to autoscale based on metrics from multiple projects.

karlkfi commented 3 years ago

If I'm reading the source correctly, it looks like the adapter uses the https://godoc.org/cloud.google.com/go/compute/metadata#ProjectID function from the Go SDK to lookup the Project ID

https://github.com/GoogleCloudPlatform/k8s-stackdriver/blob/master/custom-metrics-stackdriver-adapter/adapter.go#L102 https://github.com/GoogleCloudPlatform/k8s-stackdriver/blob/master/custom-metrics-stackdriver-adapter/pkg/config/gce_config.go#L41

Unfortunately, it sounds like that function doesn't pay attention to any of the standard environment variables that are standard as overrides in the python SDK.

And when a common helper function was requested, the behavior was instead added into specific feature clients, instead of the generic function being used by the adapter. https://github.com/googleapis/google-cloud-go/issues/1294

So it seems like we may need to add explicit support for GOOGLE_CLOUD_PROJECT and/or GOOGLE_CLOUD_PROJECT into the adapter code to allow specifying a different project.

Also, with the new Workspace design that allows projects to be in multiple workspaces, we need to specify the workspace's host project, not just a member project.