GoogleCloudPlatform / prometheus-engine

Google Cloud Managed Service for Prometheus libraries and manifests.
https://g.co/cloud/managedprometheus
Apache License 2.0
189 stars 86 forks source link

fix(export): add support in onGCE + no conn to metadata case #1021

Closed bwplotka closed 2 months ago

bwplotka commented 2 months ago

Fixes b/344740239 (edge case with GKE Metadata Server and GKE sandbox).

Regression test fails on prev flow (no context propagation)

image

Alternatives

Everything we do in FromFlags or constructor is within readines period. We could consider moving potentially "slow" things on slow network or metadata srv to exporter.Run. This could be questionable as for GMP to work we at end need export functionality to work, so delaying that information or making it surface in separation to readiness might not be helpful.

EDIT: Added PR against metadata so we can get rid of custom code in the future: https://github.com/googleapis/google-cloud-go/pull/10370

bwplotka commented 2 months ago

I tested this PR on my repro cluster.

image

Metrics flow for 3 nodes as expected:

image

From this PR POV it's rdy to go I think

bwplotka commented 2 months ago

After I did https://cloud.google.com/stackdriver/docs/managed-prometheus/setup-managed#explicit-credentials all works:

image

bwplotka commented 2 months ago

🤗