[Enhancement]: Amazon AppFlow support for setting "multiple files" aggregating setting

[nsb413]

Description

Amazon AppFlow supports Aggregate records into multiple files in each partition in Aggregation settings. However Terraform is only supporting None and SingleFile.

Affected Resource(s) and/or Data Source(s)

aws_appflow_flow

Potential Terraform Configuration

resource "aws_appflow_flow" "this" {
  name        = var.name
  description = var.desc

  source_flow_config {
    connector_type         = "SAPOData"
    connector_profile_name = var.source_connector
    source_connector_properties {
      sapo_data {
        object_path = var.object_path
      }
    }
  }

  destination_flow_config {
    connector_type = "S3"
    destination_connector_properties {
      s3 {
        bucket_name   = var.s3_bucket_name
        bucket_prefix = var.s3_bucket_prefix

        s3_output_format_config {
          file_type = "PARQUET"

          aggregation_config {
            aggregation_type = "MultipleFiles"
            target_file_size = "128"
          }
        }
      }
    }
  }

  task {
    source_fields = [""]
    task_type     = "Map_all"

  }

  trigger_config {
    trigger_type = "OnDemand"
  }
  tags = var.tags
}

References

https://docs.aws.amazon.com/appflow/latest/userguide/flows-partition.html

Would you like to implement a fix?

None

[github-actions[bot]]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[jpowell8]

It would be nice to have "MultipleFiles" explicitly included as a configuration option. When I test the module with no aggregation_type set, the result is Aggregate records into multiple files in each partition, so I think all 3 behaviors are available, and MultipleFiles appears to be the default behavior.

My full destination_flow_config block

  destination_flow_config {
    connector_type = "S3"
    destination_connector_properties {
      s3 {
        bucket_name = var.source_bucket_name 
        bucket_prefix = "data"

        s3_output_format_config {
          file_type = "JSON"

          prefix_config {
            prefix_type = "PATH"
          }

          aggregation_config {
            target_file_size = "128"
          }
        }
      }
    }
  }