data "aws_vpc_endpoint_service" is not according the AWS API specification

peter-weiss-prg commented 3 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

The data source "aws_vpc_endpoint_service" is insisting on service_name to be a mandatory as input parameter or filter which will return only exactly one VPC endpoint. Based on official documentation (https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/vpc_endpoint_service): " The given filters must match exactly one VPC endpoint service whose data will be exported as attributes." If it is not defined the terraform end up with the error message "Error: multiple VPC Endpoint Services matched; use additional constraints to reduce matches to a single VPC Endpoint Service".

Well, and this is not aligned with AWS API "describe_vpc_endpoint_services" specification which is used behind the data source "aws_vpc_endpoint_service" where you can query multiple or all VPC endpoints in one query and hence the get list of supported endpoints within particular region.

It is very handful to have such a full list of currently supported VPC endpoints within region, even more for regions which are not enabled by default and support for certain VPC endpoints is usually delivered with delay against default regions.

Reference, e.g. to boto3 implementation of "describe_vpc_endpoint_services" api can be found here https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2.html#EC2.Client.describe_vpc_endpoint_services.

So, in python easy: QUERY:

response = client.describe_vpc_endpoint_services(
    DryRun=False,
    Filters=[
        {
            'Name': 'service-type',
            'Values': [
                'Interface',
            ]
        },
    ]
)

print(json.dumps(response['ServiceNames']))

RESPONSE:

["aws.sagemaker.af-south-1.notebook", "aws.sagemaker.af-south-1.studio", "com.amazonaws.af-south-1.acm-pca", "com.amazonaws.af-south-1.application-autoscaling", "com.amazonaws.af-south-1.appmesh-envoy-management", "com.amazonaws.af-south-1.athena", "com.amazonaws.af-south-1.autoscaling", "com.amazonaws.af-south-1.cloudformation", "com.amazonaws.af-south-1.cloudhsmv2", "com.amazonaws.af-south-1.codebuild", "com.amazonaws.af-south-1.codedeploy-commands-secure", "com.amazonaws.af-south-1.config", "com.amazonaws.af-south-1.datasync", "com.amazonaws.af-south-1.dms", "com.amazonaws.af-south-1.ebs", "com.amazonaws.af-south-1.ec2", "com.amazonaws.af-south-1.ec2messages", "com.amazonaws.af-south-1.ecs", "com.amazonaws.af-south-1.ecs-agent", "com.amazonaws.af-south-1.ecs-telemetry", "com.amazonaws.af-south-1.elasticbeanstalk", "com.amazonaws.af-south-1.elasticbeanstalk-health", "com.amazonaws.af-south-1.elasticfilesystem", "com.amazonaws.af-south-1.elasticfilesystem-fips", "com.amazonaws.af-south-1.elasticloadbalancing", "com.amazonaws.af-south-1.events", "com.amazonaws.af-south-1.execute-api", "com.amazonaws.af-south-1.fis", 
"com.amazonaws.af-south-1.glue", "com.amazonaws.af-south-1.imagebuilder", "com.amazonaws.af-south-1.kinesis-firehose", "com.amazonaws.af-south-1.kinesis-streams", "com.amazonaws.af-south-1.kms", "com.amazonaws.af-south-1.lambda", "com.amazonaws.af-south-1.logs", "com.amazonaws.af-south-1.macie2", "com.amazonaws.af-south-1.monitoring", "com.amazonaws.af-south-1.rds", "com.amazonaws.af-south-1.redshift-data", "com.amazonaws.af-south-1.s3", "com.amazonaws.af-south-1.sagemaker.api", "com.amazonaws.af-south-1.sagemaker.featurestore-runtime", "com.amazonaws.af-south-1.secretsmanager", "com.amazonaws.af-south-1.servicecatalog", "com.amazonaws.af-south-1.sns", "com.amazonaws.af-south-1.sqs", "com.amazonaws.af-south-1.ssm", "com.amazonaws.af-south-1.ssmmessages", "com.amazonaws.af-south-1.storagegateway", "com.amazonaws.af-south-1.sts"]

And I got what I need - full list of VPC endpoints supported within provided region.

Current implementation of data source "aws_vpc_endpoint_service" is not according the AWS API spec. And that is causing the workarounds and hacks e.g. keeping the supported vpc endpoints within the static file, or using null_resource, etc... Instead of having it in comfort nice terraformish way.

Terraform CLI and Terraform AWS Provider Version

terraform v0.14.7 aws provider 3.36.0

Affected Resource(s)

data "aws_vpc_endpoint_service"

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

data "aws_vpc_endpoint_service" "list_all_supported_endpoints_in_region" {
 service_type = "Interface"
}

Expected Behavior

The expected behavior is as described in AWS API spec and it should return the list of all supported VPC endpoints within the particular region. And not throws the error - "Error: multiple VPC Endpoint Services matched; use additional constraints to reduce matches to a single VPC Endpoint Service".

Actual Behavior

The error is thrown "Error: multiple VPC Endpoint Services matched; use additional constraints to reduce matches to a single VPC Endpoint Service".

Steps to Reproduce

terraform apply

References

approximate line of code which is involved: https://github.com/hashicorp/terraform-provider-aws/blob/e1df5707d2847a1c76adbefb0de21f8df4a5bbdf/aws/data_source_aws_vpc_endpoint_service.go#L147
boto3 implementation: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2.html#EC2.Client.describe_vpc_endpoint_services
terraform data source doc: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/vpc_endpoint_service

ewbankkit commented 3 years ago

@peterx1986 Thanks for raising this issue. It seems that what is required is a new "plural" aws_vpc_endpoint_services data source.

peter-weiss-prg commented 3 years ago

yes, it can be plural. it would be very helpful to have such a data source. AWS API is there for it. As I described - now is ugly workaround needed through the bash or python and null_resource or something similar.

Peter

jlepere-everlaw commented 2 years ago

I ended up working around this with an external data source calling decsribe-vpc-endpoint-services. I output a map of service_name => service_name, as <string> => <string> is a requirement for external data sources, from my understanding.

data "external" "available_vpc_endpoint_service_names" {
  program = ["sh", "-c", "aws ec2 describe-vpc-endpoint-services --query ServiceNames --output json | jq -r 'reduce .[] as $i ({}; .[$i] = $i)'"]
}

locals {
 available_vpc_endpoint_service_names = keys(data.external.all_available_vpc_endpoint_service_names.result)
}

I agree a first-class terraform provider would be helpful here!

github-actions[bot] commented 3 months ago

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed. Maintainers can also remove the stale label.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

github-actions[bot] commented 1 month ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

hashicorp / terraform-provider-aws