search

apache / lucenenet

Apache Lucene.NET
https://lucenenet.apache.org/
Apache License 2.0
2.24k stars 639 forks source link

[Q] How to create Cluster with help of LUCENE.NET.REPLICATION #791

Open Jeevananthan-23 opened 1 year ago

Jeevananthan-23 commented 1 year ago

It's curious to get an understanding of how to handle auto failover in a cluster with Master - 2 or more replication. With the help of LUCENE.NET.REPLICATION in LuceneNet(NRT-segment replication) we can create a cluster with master-replica clusters but handling failover within the cluster is not possible without any distributed consensus algorithm. By using .NEXT Raft we can achieve the handling clusters failovers. So my question is how to implement the consensus within Lucene.NET.

Thanks!

NightOwl888 commented 1 year ago

This is very similar to the question titled "Is there a way to customize segment names?" on the lucene mailing list: https://lists.apache.org/list?dev@lucene.apache.org:2022-12

It seems that there is nothing built-in that can track segment names for a failover situation, and it is not something that they intend to support. You might want to direct your question to the lucene developer mailing list, as they will likely have a better answer than you can find here.

rclabo commented 1 year ago

It seems like this is the type of functionality that is added by Solr or OpenSearch. So you may want to checkout out their Repos for code that would be helpful. Both are under Apache License I believe.

nzdev commented 1 year ago

I think you would need to backport this feature from the lucene 5.0 codec https://issues.apache.org/jira/browse/LUCENE-5895

Jeevananthan-23 commented 1 year ago

It seems like this is the type of functionality that is added by Solr or OpenSearch. So you may want to checkout out their Repos for code that would be helpful. Both are under Apache License I believe.

Thanks, @rclabo pointing me to the right repos https://github.com/opensearch-project/OpenSearch/issues/2229 great Design Proposal on Segment Replication.

nzdev commented 1 year ago

Interesting link @Jeevananthan-23 . I think this exists already in Lucene 6 in replicator as nrt replication?

nzdev commented 1 year ago

Would be interesting to see the NRT replication from v6 if it can be backported to v4. Here is a start https://github.com/nzdev/lucenenet/tree/feature/nrt-v6/src/Lucene.Net.Replicator/Nrt.

Jeevananthan-23 commented 1 year ago

@nzdev, the question is how to manage a failover kind of distributed consensus. My proposal is to copy the merged segment from the primary node and transfers it to replicas by using consensus Raft. https://github.com/opensearch-project/OpenSearch/issues/2212 discussion on shard promotion inside the cluster.

Jeevananthan-23 commented 1 year ago

Would be interesting to see the NRT replication from v6 if it can be backported to v4. Here is a start https://github.com/nzdev/lucenenet/tree/feature/nrt-v6/src/Lucene.Net.Replicator/Nrt.

Hi @nzdev, I could like to discuss this more can you connect here