search

Stratio / cassandra-lucene-index

Lucene based secondary indexes for Cassandra
Apache License 2.0
600 stars 170 forks source link

Write performance tuning #371

Closed Qnzvna closed 5 years ago

Qnzvna commented 6 years ago

Hi, I've found your plugin really helpful and neat regarding how it can enhance the power of C. The combination of Lucene index and C is a really nice thing to do as it gives you both the high scalability and flexibility do get the data you really want to query.

Currently we are in the phase of testing the solution. From the functional point of view everything works as intended but we are struggling we've performance issues. I have couple questions around tuning the C and plugin parameters. My test C setup is a 3 node, one dc with replication_factor set to one also. The C* version is 3.0.13. On the (3.0.9 we've experienced OOMs possibly regarding https://issues.apache.org/jira/browse/CASSANDRA-12796)

Without Lucene the overall write throughput is around 4.5K writes/node/s. From that 3K writes is going to the indexed table. The latency is around: 12us. After turning on Lucene the writes the overall write is around 3K writes and the Lucene table can handle 1.5K writes. The latency is around: 5ms.

The decrease of the performance is worst as I understand that we need to make some additional work to index the data on the write. But after couple of hours the MutationStage queue in C* is going high and the cluster is unresponsive. I don't see a bottleneck both in the disks or in the CPU. Disk utilization is around 10% while the CPU is not going above 50%.

Everything is running on the default plugin parameters. I've tried to setup the refresh_rate to the higher value but it didn't improve. I've started to look into the GC pauses but they are around 500ms. Currently we are running with the CMS.

I didn't test the reads yet but as our cluster is more write oriented this have higher priority for me.

To sum up:

zjones commented 6 years ago

@Qnzvna - Did you have any luck resolving your performance and OOM issues?

Qnzvna commented 5 years ago

Yes, the problem was actually quite simple and was not related to the Lucene plugin itself. We've didn't use frozen collection on the table. I'm closing it, sorry for bothering.