Open dancb10 opened 6 years ago
We have tried load testing with the following data: hosts timers interval 10 6415 10 - 9600 per 10 seconds OK 10 97515 10 - 146250 per 10 seconds OK 10 195615 10 - 293400 per 10 seconds FAILED
This was done with two Cyanite nodes (16G RAM, 8 Cores and SSD). Cyanite daemon fails while we still have plenty of resources.
@dancb10 you can increase queue capacity.
What's your ingestion rate? How many events per second does Cyanite get approximately?
The number of events are written in the last commit. So there are 10 instances each sending 1956*10 number of metrics every 10 seconds. So there are 293400 number of metrics send every 10 seconds, that's 29340 each second. Note that we are using pretty powerful instances and we have also split the writes and reads into multiple instances. So we have two instances that write two that read each with its own elb. Instances are c3.2xlarge | 8 | 15 | 2 x 80 We are using 2 million queue size so @ifesdjeen you are saying to increase this number? I will try with 20 million.
We are using 2 million queue size so @ifesdjeen you are saying to increase this number?
Hm, no actually 2M should be usually ok...
Hello, We are trying Cyanite for the first time in our testing environment with two Cyanite+Graphite API instances (16G RAM, 8 Cores and SSD) and a three node Cassandra cluster (64G RAM, 8 Cores and SSD) but it seems it cannot handle the load. We see a lot of queue full errors in the Cyanite config file when starting our load test. We are using graphite stress tool to load test the environment:
We have set the heap size of Cyanite to 12G
The cyanite.yaml file also contains the following ingest and write queues settings
It seems that it barely starts the load test and the following errors are sent continuously. NOTE that if even when we stop the load test the log file just keeps sending this type of errors non stop:
Do you have any ides how to fix this? @pyr @ifesdjeen . I see that this issue was addressed here previously but, has it been fixed completely? Note that we have compiled Cyanite until this commit because the latest version does not return multiple metrics in a single query. I have submitted a issue for this bug as well.