search

graphite-project / carbon

Carbon is one of the components of Graphite, and is responsible for receiving metrics over the network and writing them down to disk using a storage backend.
http://graphite.readthedocs.org/
Apache License 2.0
1.5k stars 490 forks source link

Large droppedCreates carbon agents #862

Closed alexmaccabi closed 5 years ago

alexmaccabi commented 5 years ago

Hello, I`m having an issue with carbon whisper based files storage.

Sending about 100K per minute. We have found an issue of large droppedCreates metrics.

droppedcreates

Checking number of creates, and found out that it won't go past 50. event trough: MAX_UPDATES_PER_SECOND and MAX_CREATES_PER_MINUTE set to inf

creates

deniszh commented 5 years ago

I’m not sure that Inf is valid value there. Did you try some sane values for max creates, e.g. 1000 ?

On Mon, 29 Jul 2019 at 22:40, Alex notifications@github.com wrote:

Hello, I`m having an issue with carbon whisper based files storage.

-

10 relay nodes

3 data nodes

whisper and carbon version: 1.1.5

Sending about 100K per minute. We have found an issue of large droppedCreates metrics.

[image: droppedcreates] https://user-images.githubusercontent.com/3840250/62080524-8386e500-b259-11e9-8bb2-6b49910164cd.JPG

Checking number of creates, and found out that it won't go past 50. event trough: MAX_UPDATES_PER_SECOND and MAX_CREATES_PER_MINUTE set to inf

[image: creates] https://user-images.githubusercontent.com/3840250/62080572-9c8f9600-b259-11e9-9875-63c706088267.JPG

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/graphite-project/carbon/issues/862?email_source=notifications&email_token=AAJLTVXASF25RH3KVONIU5TQB5IVPA5CNFSM4IHWBEF2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HCENRDA, or mute the thread https://github.com/notifications/unsubscribe-auth/AAJLTVRARCGQCRDXGXHDA2TQB5IVPANCNFSM4IHWBEFQ .

alexmaccabi commented 5 years ago

Hello, it was a mistake by my part. graphite is running on k8s, there was a config map for carbon.conf that was overwriting the docker config. I tested it with inf and I don't see the droppedCreates. cheers