search

GoogleCloudPlatform / prometheus

The Prometheus monitoring system and time series database. GCP fork to export to Google Cloud Managed Service for Prometheus. Main branch is kept at parity with upstream - see branches or tags for Google's additions.
https://g.co/cloud/managedprometheus
Apache License 2.0
37 stars 15 forks source link

gmp: Skip WAL (passthrough) if no PRW is specified, normal agent mode otherwise. #207

Open bwplotka opened 1 month ago

bwplotka commented 1 month ago

Before this change GMP fork in agent mode for 2.45.3 will write data to agent DB/WAL and rely on truncation to remove the entries. This is wasteful if no PRW is configured as we have a side channel for GCM export.

This change literally skips all sample appends if no PRW is configured. This effectively keeps agent DB/WAL always empty. If new configuration comes in with PRW entry, the append is enabled and all should work as in vanilla Prometheus.

This should give us maximum efficiency, while keeping agent mode smooth for everyone. This also allows experiments with "passthrough" mode (no WAL/persistence mode) in future of Prometheus.

bwplotka commented 1 month ago

Kind of works, but some statistical bugs are there, to investigate:

image

bwplotka commented 1 month ago

As per https://github.com/bwplotka/prombenchy/pull/14:

CPU is same/similar to srv mode or unchanged agent. Memory is initially much better, but then leaking (for both this PR and unchanged agent, confirming what some cx saw). To be investigated.

image