Explore & Optimize: Ensuring Effective Log Storage Management for the Observability Cluster

[schwesig]

Ensuring Effective Log Storage Management for the Observability Cluster

timeline:

• have a solution for the at least first 6 month bei End of Dec 2024
  • updated timeline, timeouts between pods (https://github.com/nerc-project/operations/issues/473) took longer/blocker
• ~have a solution for the at least first 6 month bei End of June 2024 to not being surprised by end of year~
• review regularly every two month to gain insights, if the selected storage is still sufficient

Background:

We have successfully set up an OpenShift cluster dedicated to observability (obs cluster), where log files are collected and stored using Loki and subsequently mirrored to an S3 storage for extended retention. As we aim to maintain a robust observability framework, it is crucial to address and not forget a few topics.

Objectives:

• One Year Retention: Ensure the mirrored S3 archive maintains at least one year of log data, facilitating long-term analysis and compliance with retention policies.
• Controlled Storage Growth: Implement measures to prevent the archive from growing indefinitely, which could lead to unmanageable storage costs and potential performance degradation.
• Full Storage Management: Establish a controlled solution for scenarios where the storage reaches its capacity, potentially involving the deletion of older data to accommodate new entries, ensuring the logging process remains uninterrupted. (having a controlled exit)

Discussion Points:

1. Storage Capacity Planning: Assess the required storage size to accommodate at least one year's worth of log data. This assessment may rely on initial estimates and adjust based on actual usage patterns observed over the first year. Assign an issue to monitor storage trends and plan for capacity adjustments as needed. (Alerts?!)
2. Storage Limit Enforcement: Ensuring the S3 archive does not exceed budget allocations or planned capacity. This may involve setting quotas or using AWS/S3 native features to monitor and cap storage usage.
3. Data Lifecycle Management:
   • xx% Capacity Threshold: Define a protocol for handling scenarios when storage utilization reaches xx%, including potential data lifecycle actions such as archiving (another storage) or deletion of older logs.
   • Lifecycle Tools/Strategies: Investigate tools and mechanisms available within AWS, NooBaa, Loki, or other technologies that support data lifecycle management, focusing on capabilities to automatically expire or migrate data based on age, size, or other criteria.
   • GitOps for Configuration Management: Assess the feasibility of managing lifecycle policies and storage configurations through GitOps practices, or automated changes to the storage and data management configurations.

This issue aims to keep an eye on that topic, and developing a sustainable and cost-effective log storage strategy for the Observability Cluster, ensuring compliance with retention requirements while maintaining energy preserving, operational efficiency, and cost control for a best practice rolemodel idea.

[schwesig]

started in a meeting /cc @computate @bnshr @harshil-codes @jbasu01

We made a general calculation, how much storage we need for a year (requirements). This storage size can be too low or high.

• Too low
  • we may run into uncontrolled errors, that take time to figure out the why
  • just configuring an alert, that may be overseen, is not enough
  • therefore, a contolled, automated solution
• Too high (to be safe, we could just triple the storage)
  • blocking resources
  • too much energy consumption
  • creating unneeded workload and possible costs

Therefore:

• as much as needed, as little as possible
• "controlled error strategy"
• just waiting until an error/alert comes up is NOT a solution
• and as always: automated and gitoped if possible

[schwesig]

ideas and links from the meeting

https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/4.7/html/managing_hybrid_and_multicloud_resources/bucket-policies-in-the-multicloud-object-gateway

https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketLifecycleConfiguration.html

https://nasa.github.io/cumulus/docs/configuration/lifecycle-policies/

https://github.com/noobaa/noobaa-core/blob/d03bfb2c5a3bef6e71610b80ee34dfe785bd81f4/src/api/bucket_api.js#L592-L628

https://github.com/noobaa/noobaa-core/blob/d03bfb2c5a3bef6e71610b80ee34dfe785bd81f4/src/api/common_api.js#L145-L208

Here is an issue where we listed several noobaa and aws commands that we have used on buckets. https://github.com/nerc-project/operations/issues/75

{
    "Rules": [
        {
            "ID": "DeleteAfter365Days",
                  "Expiration": {
                "Days": 365
            },
            "Status": "Enabled"
        }
    ]
}

• currently 50Ti, requesting 2?