[Enhancement]: Add Retention Key to Logging Configuration in `aws_mwaa_environment`

[jacobcbeaudin]

Description

The current logging configuration for MWAA (aws_mwaa_environment) does not allow setting the retention period for logs. The default retention period is fixed at 3 months. To provide more flexibility and control over log management, it would be beneficial to include an optional retention key in the logging configuration. This key should allow users to specify their desired log retention period in days.

Proposed Solution

Add an optional retention parameter to each logging configuration block within the aws_mwaa_environment resource. This parameter should accept an integer representing the number of days to retain the logs.

Benefits

• Provides users with greater control over log retention policies.
• Helps in managing storage costs by allowing shorter retention periods for less critical logs.
• Improves compliance with organizational policies that may dictate specific log retention periods.

Impact

This change is backward-compatible as it introduces an optional parameter. Existing configurations without the retention key will continue to use the default 3-month retention period.

Additional Context

Include any references to AWS documentation or community discussions that support the need for this feature.

Affected Resource(s) and/or Data Source(s)

• aws_mwaa_environment

Potential Terraform Configuration

resource "aws_mwaa_environment" "example" {
  name               = "example"
  execution_role_arn = aws_iam_role.example.arn
  source_bucket_arn  = aws_s3_bucket.example.arn
  dag_s3_path        = "dags/"

  network_configuration {
    security_group_ids = [aws_security_group.example.id]
    subnet_ids         = aws_subnet.private[*].id
  }

  logging_configuration {
    dag_processing_logs {
      enabled   = true
      log_level = "DEBUG"
      retention = 30  # Retain logs for 30 days
    }

    scheduler_logs {
      enabled   = true
      log_level = "INFO"
      retention = 90  # Retain logs for 90 days
    }

    task_logs {
      enabled   = true
      log_level = "WARNING"
      retention = 60  # Retain logs for 60 days
    }

    webserver_logs {
      enabled   = true
      log_level = "ERROR"
      retention = 30  # Retain logs for 30 days
    }

    worker_logs {
      enabled   = true
      log_level = "CRITICAL"
      retention = 15  # Retain logs for 15 days
    }
  }
}

References

https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/mwaa_environment

Would you like to implement a fix?

None

[github-actions[bot]]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[acwwat]

It doesn't look like the API supports setting log retention period. The API technically can provide the log group ARNs on read (but not supported by aws_mwaa_environment yet), but there's no way to easily turn that into a aws_cloudwatch_log_group resource where the log retention is set.

Your best bet is to open an AWS support case to request for the CreateEnvironment and UpdateEvnironment APIs to support bring your own CloudWatch log groups. This will then enable creation of aws_cloudwatch_log_group resources with custom log retention periods and provide them to the aws_mwaa_environment resource.