timescale / promscale_extension

[DEPRECATED] Tables, types and functions supporting Promscale
Other
37 stars 17 forks source link

A data buffer for job metrics and logs #552

Closed sumerman closed 2 years ago

sumerman commented 2 years ago

Description

This PR introduces a Rust-based escape hatch for backend metrics (and, in the future, logs) data backend_telemetry_buffer.

Addresses #495

Merge requirements

Please take into account the following non-code changes that you may need to make with your PR:

sumerman commented 2 years ago

I validated the absence of serious memory leaks by running the following multiple times:

explain analyse SELECT _prom_ext.push_rec(ROW(now(), x)::_prom_ext.backend_telemetry_rec) FROM generate_series(1, 200000) x;
SELECT COUNT(*) FROM _prom_ext.pop_recs();

After executing the first line, the RAM consumption always jumped by about 15 MB and stayed, then reliably returned to its original reading at the end of the test.

sumerman commented 2 years ago

CALL run_job(:job_id); doesn't seem to be using the actual maintenance job infrastructure. In order to prove the buffer works inside an actual job worker I've repeated the integration test on a regular instance.

sumerman commented 2 years ago

Closing in favor of https://github.com/timescale/promscale_extension/pull/555