Closed pawellesniewski closed 7 years ago
"Queue full" messages on startup mostly mean that messages were dropped in order to avoid node crash, which only means you have to increase your queue size in config:
queue:
queue-capacity: 1048576
I'll change the default (which is way to low) today. Sorry about that.
Hi @ifesdjeen , thanks for Your reply.
I have queue-capacity set to 2.000.000, i get this messages but Cyanite hangs after several hours. Is this related with queue size?
No, this is not normal at all. Which commit / RSA are you on? This sounds as if you're not running latest version, as we've already fixed the problem you describe.
Also, please note that config structure for queue has changed, it's not split into write and ingest queues anymore (if you're running latest master)
Cyanite: version 0.5.1, master branch, latest commit: #e5da4e8
build deb with lein tool.
Yes, this is an outdated version.
I am a little bit confused, so what is the current version (commit)?
https://github.com/pyr/cyanite/commit/9d49e8abc6db746577687ae0d865b6351f7c056e (current master).
We're working on publishing artefacts.
Sory, i pasted wrong one (Merge commit id): I build from:
commit 9d49e8abc6db746577687ae0d865b6351f7c056e
Merge: e5da4e8 bf1e05c
Author: αλεx π <alexp@coffeenco.de>
Date: Sat Oct 1 22:51:42 2016 +0200
In this case, your configuration is for the old scheduling system. You should use the new format:
queue:
queue-capacity: 1048576
As there is just one queue now...
Thanks a lot, i will change the configuration and see what will happen during the weekend.
👍 thanks!
We have a problem running Cyanite for more than ~ one day. It gets stuck, i mean it receives metrics:
around 1.3k metrics per second (different metrics, there could be a lot of new metrics every hour). But datapoints are no longer saved to cassandra, no new datapoints are available through api:
tcpdump and graphite (replicated traffic) confirms that metrics are send.
Cyanite storage writer decreases to 0:
PS-Old-Gen.used looks similar to case describe in #251
Cyanite: version 0.5.1, master branch, latest commit: #e5da4e8 Java version:
Run command:
Configuration:
Some GC logs (GC (Allocation Failure)):
We are getting lots of "Queue full":
if you need any additional data/logs please write :)