search

palantir / cassandra

Palantir's fork of Apache Cassandra
Apache License 2.0
10 stars 7 forks source link

Reduce SystemKeyspace truncation record contention #559

Open schlosna opened 1 month ago

schlosna commented 1 month ago

JFR profiles show significant synchronization contention on SystemKeyspace#getTruncationRecord(UUID) as part of committing PaxosState.

The SystemKeyspace truncationRecords map is a lazily initialized unmodifiable snapshot of the local node's truncated_at map. We now guard this with a read/write lock to optimistically allow concurrent reads, while falling back to most costly write lock and re-reading the local node's truncated_at from system tables when there are truncation modifications.

Contention back trace:

  org.apache.cassandra.db.SystemKeyspace.getTruncationRecord(UUID):494
  org.apache.cassandra.db.SystemKeyspace.getTruncatedAt(UUID):488
  org.apache.cassandra.service.paxos.PaxosState.commit(Commit):144
  org.apache.cassandra.service.paxos.CommitVerbHandler.doVerb(MessageIn, int):34
  org.apache.cassandra.net.MessageDeliveryTask.run():70
  java.util.concurrent.Executors$RunnableAdapter.call():539
  org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run():164
  org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run():136
  org.apache.cassandra.concurrent.SEPWorker.run():110