Closed datadavev closed 11 months ago
One suggested fix for CLUSTERSTATUS timeout is to completely shutdown solr and zookeeper on all nodes, then restart.
This process takes a couple minutes for solr to come back up and so needs to be scheduled for a convenient time on production.
Was able to adjust the nodes using zkCli.sh
as follows:
/var/solr/server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181 -cmd getfile /collections/search_core/state.json search_core_state.json
edit search_core_state.json
, then:
/var/solr/server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181 -cmd putfile /collections/search_core/state.json search_core_state.json
Repeated for the event core. This seemed to be successful, with the cloud view in admin being much more responsive and showing the correct configuration.
There still remains the issue of clusterstatus timeout. The procedure for repairing that is likely:
/overseer/collection-queue-work/
in zkSmall python script for deleting the zk entries:
import kazoo.client
def main():
host = "127.0.0.1:2181"
root = "/overseer/collection-queue-work/"
zk = kazoo.client.KazooClient(hosts=host)
zk.start()
try:
for key in zk.get_children(root):
if key.startswith("qn-"):
_path = f"{root}{key}"
print(f"deleting {_path}")
#zk.delete(_path)
finally:
zk.stop()
if __name__ == "__main__":
main()
After purging the pending clusterstatus requests from zk and restarting solr and zk on each node, normal operations have been restored. It was necessary to kill the zk process on orc as it was unresponsive to shutdown requests.
Despite removal of the UNM nodes and all associated configuration, there remains a "gone" node entry in the solr cloud. This causes spurious error messages to appear in the solr logs and although does not appear to have othe impacts, should be cleaned up.
Listing the nodes can be done with the
CLUSTERSTATUS
command, e.g.:or using zkcli:
Removing a replica is done with the
DELETEREPLICA
command.This operation worked on stage (with parameter adjustment, there the gone node was node2), but times out on production.