Exploration of Additional Personas for Observability Cluster Based on Retention Rate Requirements

[schwesig]

Background

In recent discussions regarding the observability cluster, it has become apparent that our current persona coverage may not fully address all needs, particularly in terms of metrics retention rates. Two key additional personas have been identified as essential to ensuring our observability infrastructure meets all operational and compliance requirements:

1. IRS (Internal Revenue Service) Compliance Persona: This persona focuses on the requirements for storing data necessary to back up invoices and support or facilitate financial audits. Understanding the specifics of data retention for this persona is crucial for compliance with tax laws and regulations.
2. Security Persona: For this persona, the primary concern is the retention of data necessary to investigate security incidents. A minimum retention period of 90+ days has been suggested, but further investigation is required to define the exact requirements.

Goals

Identify Specific Data Retention Requirements for IRS Compliance Persona: Determine the types of data that need to be retained, the duration of retention, and any specific formats or standards that must be adhered to.

Define Data Retention Needs for Security Persona: Establish the minimum data retention period necessary to support effective security incident investigations. This involves identifying the types of data crucial for such investigations and any relevant standards or best practices.

Tasks

• [ ] Research IRS Compliance Requirements
• [ ] Investigate Security Data Retention Best Practices: Ask a security expert (possibly Augustine?)

Discussion Points:

• Are there any existing standards or benchmarks from similar organizations that we can refer to for both personas?
• How will the retention policy impact our current storage and data management infrastructure?
• Coordination with legal and security teams to ensure all recommendations meet regulatory requirements?

[schwesig]

• Follow IRS Publication 583 for detailed recordkeeping requirements.
• Retention Rates:
  • Invoices and Financial Records:
  • Typically, you need to retain these records for at least 7 years. This period ensures you have adequate documentation in case of audits or inquiries from the IRS.

[schwesig]

• Audit Logs: Generally retained for 7 years to comply with regulations like Sarbanes-Oxley (SOX).
• Security Incident Reports: Typically retained for at least 6 years, starting from the date of the last entry in the report​ (ACRMS)​​ (Cribl)​.
• HIPAA: Healthcare organizations must retain data for up to 6 years.
• SOX: Requires financial records to be retained for 7 years.
• GDPR: Personal data should be retained only as long as necessary for the purpose it was collected​ (Securiti)​​ (RSI Security)​.