gnocchixyz / gnocchi

Timeseries database
Apache License 2.0
298 stars 84 forks source link

Clean raw data metrics that do not receive measures #1385

Closed rafaelweingartner closed 3 weeks ago

rafaelweingartner commented 3 months ago

The truncating of the raw metrics data is only done when new measures are pushed. Therefore, if no new measures are pushed, and the archive policy is updated to reduce the back window, the raw data points for metrics that are not receiving new data points are never truncated.

The goal of this PR is to propose a method to identify metrics that are in "inactive state", meaning, not receiving new data points, and execute their raw data points truncation when the archive policy backwindow is changed.

rafaelweingartner commented 1 month ago

Thanks @chungg for the review!

I address the points you raised, and replied to some of the open questions. I am eager to see your feedback on them!

Again, thanks for the support here!

rafaelweingartner commented 1 month ago

Thanks @chungg! Your support is much appreciated!

rafaelweingartner commented 1 month ago

@chungg the error is caused by a numpy upgrade. I cheated a patch to address it via: https://github.com/gnocchixyz/gnocchi/pull/1390. After that is merged, I can rebase this one here.

rafaelweingartner commented 1 month ago

@chungg now this patch seems to be fine. I also squashed all commits to avoid mixing up things, as the patch was depending on others.

rafaelweingartner commented 4 weeks ago

Thank you guys (@pedro-martins and @chungg) for your reviews!

If you think that there is something else to be changed, let me know.

rafaelweingartner commented 4 weeks ago

lgtm

Thanks!

As soon as this one is merged, I will rebase https://github.com/gnocchixyz/gnocchi/pull/1386, and make it ready for review as well.

Then, we will be able to implement a solution for https://github.com/gnocchixyz/gnocchi/issues/1182.

rafaelweingartner commented 3 weeks ago

Hello @chungg and @tobias-urdin can we merge this one. I have some time this week, and I would like to use this time to rebase the other patch as soon as this patch is merged.

rafaelweingartner commented 3 weeks ago

Thanks!