Splunk Operator: FAILED - RETRYING: Peer cluster master UDS

[jadamkhhh]

Please select the type of request

Bug

Tell us more

Describe the request

• I used custom Helm values.yml to deploy Splunk components. License manager and cluster manager could start properly. However, other components (monitoring console, heavy forwarders etc.) appear FAILED - RETRYING: Peer cluster master UDS error and could not start the instance.

I looked in to splunkd.log in one of the failing component, it raised error 07-06-2024 08:27:22.099 +0000 ERROR ClusterStatusHandler [1515 TcpChannelThread] - Invalid manager_uri=http+unix://%2Fopt%2Fsplunkforwarder%2Fvar%2Frun%2Fsplunk%2Fcli.socket

Expected behavior

• Splunk monitoring console, heavy forwarders, indexer cluster, search head cluster peer to cluster manager node successfully and start up properly.

[jadamkhhh]

Downgrading splunk image to 9.1.3 fixed the issue.