There is no mechanism that removes metrics (after a TTL expired) from the storer file used to save this metric into the disk to calculate the deltas on each execution.
On initialization the integrations checks the TTL of the entire file remove it if it has expired. But there are cases where the metric identifiers are based on ephemeral entities causing that new entries to this file are constantly being created and if the integration keeps it's execution normally the current clean up mechanism is never triggered, making this file grow without control and having the integration loading in on each execution.
Expected Behavior
Metrics on the storer should be garbage collected if they are not being updated after a TTL.
Description
There is no mechanism that removes metrics (after a TTL expired) from the storer file used to save this metric into the disk to calculate the deltas on each execution.
On initialization the integrations checks the TTL of the entire file remove it if it has expired. But there are cases where the metric identifiers are based on ephemeral entities causing that new entries to this file are constantly being created and if the integration keeps it's execution normally the current clean up mechanism is never triggered, making this file grow without control and having the integration loading in on each execution.
Expected Behavior
Metrics on the storer should be garbage collected if they are not being updated after a TTL.
Similar thing is done in the Kubernetes integration
Steps to Reproduce
Your Environment
One example happens on
nri-varnishwherebackendentities could be ephemeral in some environments. more context in this issueAdditional context