search

manticoresoftware / manticoresearch-helm

Helm chart for Manticore Search
Apache License 2.0
34 stars 10 forks source link

Statefulness and cluster syncing issues #100

Open Dlahacz opened 1 month ago

Dlahacz commented 1 month ago

Hi,

We have multiple issues related to the cluster syncing status. I think all of them can be solved with a startup probe that marks the pod as running only after cluster_node_state is synced.

Issue Examples

Adding a New Worker Syncing data to a new node takes a few minutes, but the load balancer starts serving queries to the new worker before all tables are synced. (I'm not sure if this one can be solved without changes in observer.php)

Updating values.yaml (Variables Related to Worker STS, Searchd Configuration, Resources, and Limits)

We have, for example, 3 workers. The STS definition changes. Kubernetes starts with worker-3; the pod is shut down. A new worker-3 pod starts up, and worker-3 begins syncing with worker-2. Since Searchd is running, Kubernetes thinks worker-3 is up and proceeds to restart worker-2. Because worker-3 is not yet fully synced and worker-2 has restarted, worker-3's state is now destroyed. The same happens for worker-2. When worker-1 restarts, there is no primary instance left, leaving the entire cluster in a broken state.

Kubernetes Node Rotation When, for example, I update Kubernetes nodes to a new version or change their instance types, a similar situation to the previous one can occur.

Proposed Solution

Add a startup probe. if /usr/bin/mysql -e "show status;" | grep cluster_node_state | grep -q synced ; then exit 0; else exit 1; fi

Dlahacz commented 1 month ago

We would be happy to do the work, create a pull request, and test it in production etc. We just want to know your opinion on the solution beforehand, so our time isn't wasted.

sanikolaev commented 4 weeks ago

Thanks for the issue @Dlahacz @djklim87 pls look into this.

djklim87 commented 1 week ago

Be aware that replication within Manticore uses worker hosts, such as helm-manticoresearch-worker-1. This host isn’t available until the startupProbe is completed. As a result, your code may encounter a race condition where replication cannot finish because the startupProbe hasn’t completed, and the probe itself cannot complete due to the pending replication process.

Dlahacz commented 14 hours ago

Oh, thanks. I was not aware of that, so what about checking for node status in observer.php?