search

graphite-project / carbon

Carbon is one of the components of Graphite, and is responsible for receiving metrics over the network and writing them down to disk using a storage backend.
http://graphite.readthedocs.org/
Apache License 2.0
1.51k stars 490 forks source link

The older data captured from collectd is getting lost in graphite server #823

Closed Pratheeshpt closed 5 years ago

Pratheeshpt commented 5 years ago

Hi,

I am capturing CPU usage data from my target board using collectd. In collectd I am using write_graphite plugin to send the date to a Desktop Linux machine. I have setup collectd, graphite, carbon-cache, whisper, grafana etc on desktop machine and successfully getting the collectd data from my target board. The same can be viewed in Grafana. But if I am continuously running the test setup for several days, I noticed that after 2 days the captured data is not displaying in Grafana. I have attached a screenshot of this (cpu_usage_for_7_days.png). I started capturing the data from 2-Nov-2018, 16:30 IST. And I confirmed that the data is captured by viewing the graph in Grafana on same day and monitored it for few hours. But after few days, I cannot see the older data. Now it is showing only last one day's data. I have attached the percent-active.wsp.txt file stored on the desktop machine for reference.

Target board:

OS : ARM Linux Version of collectd : collectd-5.8.0

Desktop Machine:

Operating system : Ubuntu 14.04 graphite-carbon : 0.9.12-3 grafana-server : Version 5.1.4 collectd version : 5.4.0

/etc/carbon/storage-schemas.conf in Desktop machine

[carbon] pattern = ^carbon. retentions = 60:90d

[collectd] pattern = ^collectd.* retentions = 10s:1d,1m:7d,10m:1y

[default_1min_for_1day] pattern = .* retentions = 60s:1d

collectd.conf in target board

Interval 10

Screenshot of CPU usage for 7 days displayed on Desktop machine's Grafana

cpu_usage_for_7_days

percent-active.wsp.txt: percent-active.wsp.txt

Pratheeshpt commented 5 years ago

Hi,

I have one more observation. I checked the whisper-info of that wsp file in which I lost the older data. It is as shown below.

$ whisper-info /var/lib/graphite/whisper/target1/localhost/cpu/percent-active.wsp maxRetention: 86400 xFilesFactor: 0.5 aggregationMethod: average fileSize: 17308

Archive 0 retention: 86400 secondsPerPoint: 60 points: 1440 size: 17280 offset: 28 $

deniszh commented 5 years ago

@DanCech @piotr1212 I think we need to change defaults in https://github.com/graphite-project/carbon/blob/master/conf/storage-schemas.conf.example This question is popping up so often, in issues and across the Internet...

@Pratheeshpt: Default config retention (retentions = 60s:1d) means "Keep last day of 1 minute data (i.e. 86400 points, same as whisper-info shows) and discard everything beyond 1 day to keep disk size limited". Please check the documentation or e.g. http://wiki.xkyle.com/Graphite_Storage_Schemas.html

piotr1212 commented 5 years ago

@Pratheeshpt the name of your collectd metrics does not match the pattern you configured, thus the metrics end up in the default schema. Set Prefix "collectd" in your collectd.conf in the write_graphite section. Or change the pattern in storage-schemas.conf