[Draft]: CSPL-2600: Integrate HashiCorp Vault Support in Splunk Operator

[vivekr-splunk]

Description:

This PR introduces the integration of HashiCorp Vault into the Splunk Operator, providing enhanced security and flexibility in managing secrets. Users can now choose between Kubernetes Secrets or HashiCorp Vault for storing and injecting Splunk secrets. Key changes and features include:

Key Features:

1. Vault Integration in Custom Resource:

   • Added a new VaultIntegration struct in the Splunk CRD, allowing users to enable Vault integration and specify the Vault role and secret path.
   • Fields: enabled, role, and secretPath to configure Vault usage.

2. Vault Injection Annotations for StatefulSets:

   • Implemented the InjectVaultSecret function to add Vault Agent injector annotations to StatefulSets managed by the Splunk Operator.
   • Annotations specify Vault roles, secret paths, file paths, and volume paths for injection.
   • Secrets are mounted at /mnt/splunk-secrets, consistent with the current Kubernetes Secrets path.

3. Vault Client Integration:

   • Added a function using the Vault Go SDK (getVaultSecretVersion) to retrieve the current version of secrets from Vault, ensuring updated secrets are injected.

4. Support for Both Kubernetes Secrets and Vault:

   • Updated the Splunk Operator's reconciliation logic to support both Kubernetes Secrets and Vault.
   • The operator automatically selects between Kubernetes Secrets and Vault based on the custom resource configuration.

5. Predefined Vault Keys:

   • Predefined keys for secrets: hec_token, idxc_secret, pass4SymmKey, password, and shc_secret.
   • Ensures consistency across environments regardless of whether Kubernetes Secrets or Vault is used.

How to Test the Changes:

1. Install and Configure Vault on Kubernetes:

   • Install Vault using Helm and configure Kubernetes authentication.
   • Create secrets in Vault and bind a Vault role to a Kubernetes service account (splunk-service-account).

2. Deploy Splunk Custom Resource:

   • Enable Vault integration in the Splunk CR and deploy it.
   • Verify annotations on StatefulSets and check for successful secrets injection in Splunk pods.

3. Test Secret Rotation:

   • Update secrets in Vault and trigger a rolling restart of StatefulSets to validate secret updates are applied to Splunk pods.

4. Switch Between Kubernetes Secrets and Vault:

   • Test switching Vault integration on and off by updating the Splunk CR and verifying the secrets management behavior.

Summary:

This PR significantly enhances the security of Splunk deployments by integrating HashiCorp Vault into the Splunk Operator, allowing users to seamlessly manage secrets using Vault or Kubernetes Secrets. This flexibility provides improved compliance, security, and operational ease.

Verification:

• [x] Verified with Vault integration enabled.
• [x] Verified with Kubernetes Secrets only.
• [x] Verified secrets rotation and pod restart behavior.

Documentation:

• Updated design document for HashiCorp Vault integration.

Testing:

• Full end-to-end testing performed, including secret injection, role configuration, and secret rotation.

Let me know if there are any questions or if further changes are needed!