The Splunk Operator for Kubernetes currently supports AWS S3 buckets, allowing users to integrate Splunk with AWS storage seamlessly. However, customers utilizing Google Cloud Platform (GCP) and Microsoft Azure are required to use interoperability solutions to access their respective Blob storage services. This pull request introduces native support for GCP storage buckets and enhances Azure integration by implementing Azure AD authentication alongside traditional credential methods.
Key Enhancements:
GCP Storage Support:
GCP SDK Integration: Introduced the Google Cloud Platform (GCP) SDK to enable direct interaction with GCP storage buckets.
GCP SDK Wrapper: Developed a wrapper for the GCP SDK, mirroring the existing AWS S3 client wrapper, to facilitate operations such as listing and downloading objects from GCP storage buckets.
Azure Storage Enhancements:
Azure AD Authentication: Implemented Azure Active Directory (Azure AD) authentication to provide secure and streamlined access to Azure Blob Storage.
Azure SDK Integration: Leveraged the Azure SDK to manage credentials and handle Blob storage operations effectively.
Motivation
Expanding the Splunk Operator's storage capabilities to include GCP and Azure addresses the needs of a broader user base operating in multi-cloud environments. Native support eliminates the complexities and limitations associated with interoperability solutions, offering users a more robust, secure, and efficient way to manage their storage integrations within Splunk.
Changes
GCP Integration
GCP SDK Wrapper:
Created a new module gcpbucketclient that encapsulates GCP SDK functionalities.
Implemented methods for:
Listing objects in GCP storage buckets.
Downloading objects from GCP storage buckets.
Configuration Updates:
Added configuration options in the operator's settings to specify GCP credentials and bucket details.
Updated Kubernetes ServiceAccount and Role definitions to accommodate GCP access permissions.
Documentation:
Updated README and user guides to include instructions for configuring GCP storage integration.
Azure Enhancements
Azure AD Authentication:
Integrated Azure AD authentication mechanisms to secure access to Azure Blob Storage.
Configured the operator to use Azure AD tokens for authenticating storage operations.
Azure SDK Integration:
Utilized the Azure SDK to handle credential management and Blob storage interactions.
Ensured compatibility with existing Azure storage configurations and workflows.
Configuration Updates:
Enhanced the operator's configuration to support Azure AD authentication parameters.
Updated Kubernetes manifests to include necessary Azure RBAC roles and permissions.
Documentation:
Expanded documentation to cover Azure AD authentication setup and troubleshooting steps.
Testing
Unit Tests:
Added unit tests for the new GCP SDK wrapper to ensure reliable interactions with GCP storage.
Enhanced existing tests to cover Azure AD authentication flows.
Integration Tests:
Performed end-to-end testing with GCP storage buckets to validate listing and downloading functionalities.
Verified Azure Blob Storage access using Azure AD authentication, ensuring secure and authorized operations.
Regression Testing:
Ensured that existing AWS S3 functionalities remain unaffected by the new integrations.
Conducted performance testing to confirm that the operator maintains optimal performance with multi-cloud support.
Impact
This enhancement broadens the Splunk Operator's compatibility, making it a more versatile tool for organizations leveraging multi-cloud strategies. By providing native support for GCP and Azure storage services, users can manage their Splunk integrations more effectively, reducing dependency on third-party interoperability solutions and simplifying their infrastructure management.
Related Issues
Issue CSPL-2824: Support for GCP Storage Buckets
Issue CSPL-2825: Implement Azure AD Authentication for Blob Storage Access
Future Work
GCP Storage Operations: Extend GCP storage support to include uploading objects and managing bucket configurations.
Azure Storage Features: Incorporate additional Azure Blob Storage functionalities such as container management and advanced access controls.
Multi-Cloud Optimization: Explore optimizations for simultaneous multi-cloud storage operations to enhance performance and reliability.
User Interface Enhancements: Update the operator's UI/dashboard to provide better visibility and control over multi-cloud storage integrations.
Checklist
[x] Code follows the project's coding standards
[x] Relevant documentation has been updated
[x] All new and existing tests pass
[x] Changes have been tested in a development environment
Signed-off-by: Vivek Reddy vivekrsplunk@github.com
The Splunk Operator for Kubernetes currently supports AWS S3 buckets, allowing users to integrate Splunk with AWS storage seamlessly. However, customers utilizing Google Cloud Platform (GCP) and Microsoft Azure are required to use interoperability solutions to access their respective Blob storage services. This pull request introduces native support for GCP storage buckets and enhances Azure integration by implementing Azure AD authentication alongside traditional credential methods.
Key Enhancements:
GCP Storage Support:
Azure Storage Enhancements:
Motivation
Expanding the Splunk Operator's storage capabilities to include GCP and Azure addresses the needs of a broader user base operating in multi-cloud environments. Native support eliminates the complexities and limitations associated with interoperability solutions, offering users a more robust, secure, and efficient way to manage their storage integrations within Splunk.
Changes
GCP Integration
GCP SDK Wrapper:
gcpbucketclientthat encapsulates GCP SDK functionalities.Configuration Updates:
ServiceAccountandRoledefinitions to accommodate GCP access permissions.Documentation:
Azure Enhancements
Azure AD Authentication:
Azure SDK Integration:
Configuration Updates:
Documentation:
Testing
Unit Tests:
Integration Tests:
Regression Testing:
Impact
This enhancement broadens the Splunk Operator's compatibility, making it a more versatile tool for organizations leveraging multi-cloud strategies. By providing native support for GCP and Azure storage services, users can manage their Splunk integrations more effectively, reducing dependency on third-party interoperability solutions and simplifying their infrastructure management.
Related Issues
Future Work
Checklist