adrianlzt / collectl

Extending collectl to send process data to graphite
Other
2 stars 3 forks source link

Lost data #1

Open adrianlzt opened 10 years ago

adrianlzt commented 10 years ago

Seems that sending too much data is causing that some data is not being stored in whisper.

adrianlzt commented 10 years ago

/opt/graphite/conf/carbon.conf

Softly limits the number of whisper files that get created each minute. Setting this value low (like at 50) is a good way to ensure your graphite system will not be adversely impacted when a bunch of new metrics are sent to it. The trade off is that it will take much longer for those metrics' database files to all get created and thus longer until the data becomes usable. Setting this value high (like "inf" for infinity) will cause graphite to create the files quickly but at the risk of slowing I/O down considerably for a while. MAX_CREATES_PER_MINUTE = 50

As I can see this parameter is not working. If a delete all files in /opt/graphite/storage/whisper/maquina1/process and then start collectl, with the first data arrives whisper creates 122 files at once.

Others parameter that could affect:

Limits the number of whisper update_many() calls per second, which effectively means the number of write requests sent to the disk. This is intended to prevent over-utilizing the disk and thus starving the rest of the system. When the rate of required updates exceeds this, then carbon's caching will take effect and increase the overall throughput accordingly. MAX_UPDATES_PER_SECOND = 500