[Enhancement]: Add `vpc_attachment_options` argument to `aws_osis_pipeline` resource

[ralimardanov]

Terraform Core Version

1.3.1

AWS Provider Version

5.63.0

Affected Resource(s)

aws_osis_pipeline

Expected Behavior

It should create osis pipeline with vpc endpoint.

Actual Behavior

During terraform apply I get an error:

operation error OSIS: CreatePipeline, https response error StatusCode: 400,
RequestID: 672a20cb-bfac-4248-9373-9502a72479c4, ValidationException:
"vpcAttachmentOptions" is a required parameter for AWS DocumentDB source.

Relevant Error/Panic Output Snippet

No response

Terraform Configuration Files

locals {
  pipeline_configuration = {
    markets-collection = {
      documentdb_host                  = "<documentdb_host>"
      documentdb_collection            = "<documentdb_collection>"
      pipeline_ingestion_role          = aws_iam_role.ingestion_pipelines_role.arn
      pipeline_ingestion_bucket        = aws_s3_bucket.ingestion_pipelines_bucket.id
      pipeline_ingestion_bucket_prefix = "<s3_bucket_prefix>"
      opensearch_host                  = ["<opensearch_url>"]
      opensearch_index                 = "<os_index>"
      documentdb_admin_secret          = "<arn_of_secret>"
      max_units                        = 4
      min_units                        = 2
    }
  }
}

resource "aws_osis_pipeline" "opensearch_ingestion_pipelines" {
  for_each = local.pipeline_configuration

  pipeline_name = each.key
  pipeline_configuration_body = templatefile("${path.module}/zero_etl_pipelines/${each.key}.yaml", {
    documentdb_host                  = each.value.documentdb_host,
    documentdb_collection            = each.value.documentdb_collection,
    pipeline_ingestion_role          = each.value.pipeline_ingestion_role,
    pipeline_ingestion_bucket        = each.value.pipeline_ingestion_bucket,
    pipeline_ingestion_bucket_prefix = each.value.pipeline_ingestion_bucket_prefix,
    opensearch_host                  = yamlencode(each.value.opensearch_host),
    opensearch_index                 = each.value.opensearch_index,
    documentdb_admin_secret          = each.value.documentdb_admin_secret
  })

  max_units = each.value.max_units
  min_units = each.value.min_units

  log_publishing_options {
    is_logging_enabled = true
    cloudwatch_log_destination {
      log_group = "/aws/vendedlogs/OpenSearchService/IngestionService/${each.key}"
    }
  }

  vpc_options {
    subnet_ids         = [<subnet_ids>]
    security_group_ids = [<security_group_ids>]
  }
}

Steps to Reproduce

To reproduce you should have DocumentDB cluster and Opensearch domain. Pipeline will connect to DocumentDB collection, get documents, store in s3 bucket and then process to Opensearch domain. The issue is when you try to use VPC endpoints instead of public access. Terraform resource is missing VPC attachment and VPC CIDR options. I assume it wasn't implemented yet. Do you have any workaround for that or any ETA when it will be implemented? Thanks.

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

No

[github-actions[bot]]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[ralimardanov]

https://github.com/hashicorp/terraform-provider-aws/issues/31364 I see in this issue that there is another provider resource that can be used - awscc_osis_pipeline, but it's not recommended to use. VPC attachment is implemented in that resource - https://github.com/hashicorp/terraform-provider-awscc/blob/main/internal/aws/osis/pipeline_resource_gen.go#L601-L623.

[justinretzolk]

Hey @ralimardanov 👋 Thanks for taking the time to raise this! It looks like this ultimately comes down to the need for an additional argument for vpc_attachment_options. We consider additions to existing resources to be an enhancement, rather than a bug. With that in mind, I'm going to update a few things about this issue. There's no further action needed from you at this point, I just like to let people know before making these kinds of modifications.