Solr Overseer / Zookeeper single point of contention

[cfournie]

Overseer can become the bottleneck in very large clusters; let's:

1. Look at how Overseer works
2. Define the problem
3. Look at existing solution attempts to see where we could contribute

[cfournie]

Reasons why work can build up on the overseer:

• Large number of events added to the queue (e.g. SOLR-10265)
• Solr node elected as overseer could be experiencing issues (e.g. too much time spent doing GC)
• If an Overseer has not been elected or we're between elections (e.g. SOLR-13416)
• Contention adding events to the global queue in Zookeeper?
• Contention adding events to the per-collection state.json in Zookeeper?

[morissm]

If search/indexation contention on the overseer is the issue, separating overseer and data nodes might be a solution. https://solr.apache.org/guide/solr/latest/deployment-guide/node-roles.html

[cfournie]

Changes proposed to Solr include:

• SOLR-14927: Removing Overseer (ApacheCon Talk)
• SOLR-11465: Process messages in parallel

[cfournie]

How Overseer works:

• Javadoc
• Wiki article

[cfournie]

Mitigation of this by creating separate Overseer nodes using node roles would be interesting, but unfortunately right now if using the k8s solr-operator to manage your cluster it does not support cluster roles yet.