search

aws / aws-cdk

The AWS Cloud Development Kit is a framework for defining cloud infrastructure in code
https://aws.amazon.com/cdk
Apache License 2.0
11.59k stars 3.89k forks source link

Using InterfaceVPCEndpoint is awkward for custom PrivateLink connections #3627

Closed mitchlloyd closed 1 year ago

mitchlloyd commented 5 years ago

Using InterfaceVPCEndpoint seems oriented toward creating endpoints for AWS-managed services. This makes it awkward to use for connecting to custom VPC Endpoint Services. Specifically:

  1. The privateDNSEnabled: true default doesn't work for custom VPC Endpoint services outside of AWS Marketplace. service com.amazonaws.vpce.us-west-2.vpce-svc-abc123 does not provide a private DNS name.

  2. Getting the primary DNS name (i.e. the DNS entry that points to other AZs) is challenging. Something like this does not work:

this.baseLambdaFunction = new lambda.Function(this, 'Resource', {
    runtime: props.runtime,
    handler: props.handler,
    code: lambda.Code.asset(props.zipPath),
    vpc: props.vpc,
    environment: {
        MY_SERVICE_HOST: vpcEndpoint.vpcDNSEntries[0] // => "#{Token[TOKEN.24]}"
    },
    memorySize: props.memorySize,
});

Instead, getting that value requires several Cfn functions: 

const firstEntry = cdk.Fn.select(0, endpoint.attrDnsEntries);
const entryParts = cdk.Fn.split(':', firstEntry);
const primaryDNSName = cdk.Fn.select(1, entryParts);
  1. The current service property expects one port which is presumptuous because services can listen on many ports.

It may be the case that the intended use of InterfaceVPCEndpoint may be different enough from connecting to custom VPC Endpoint Services to warrant a new class. I don't have any good names but maybe CustomVPCEndpoint. This new construct would address the 3 issues noted above.

At the least, it would be nice if InterfaceVPCEndpoint had a primaryDNSName method so that users don't need to know the exact format of this Cfn value to get a DNS name.

Make using custom PrivateLink connections with CDK more convenient.

nzspambot commented 4 years ago

@mitchlloyd you can chain the Fn.* together as such:

.dnsName(Fn.select(1, Fn.split(":", Fn.select(0, VpcEndpoint.vpcEndpointDnsEntries))))

And also given that I'm using vpc endpoints to build alias records it might be ideal to also expose the HostedZoneId - this is needed because currently the ARecord builder doesn't support regions/continents so need to use CfnRecordSet as such:

.aliasTarget(CfnRecordSet.AliasTargetProperty.Builder() .dnsName(Fn.select(1, Fn.split(":", Fn.select(0, VpcEndpoint.vpcEndpointDnsEntries)))) .hostedZoneId(Fn.select(0, Fn.split(":", Fn.select(0, VpcEndpoint.vpcEndpointDnsEntries)))) .evaluateTargetHealth(true) .build())

flemjame-at-amazon commented 4 years ago

@mitchlloyd the private DNS portion is fixed as of https://github.com/aws/aws-cdk/pull/5987/

bweigel commented 3 years ago

I am wondering if anyone else needs custom domain names for their interface VPC endpoints. Something that is described in detail in the context of private API Gateways with custom domain names here or here.

Generally one could think of an architecture like this:

image

Where the endpoint provider provides a VPC endpoint service (like private AWS API Gateway, RabbitMQ for AmazonMQ etc.) and the consumer would want to use a nice custom domain name like <rabbitmq|apigw>.mycompany.local and not the obscure VPC interface endpoint domains (like vpce-<some-id>.vpce-svc-<some-other-id>.eu-central-1.vpce.amazonaws.com).

Typically that would be accomplished via another proxy (NLB) that balances between the ENIs (IPs) of the interface VPC endpoint. The NLB can have a custom domain name, together with a certificate.

Unfortunately (afaik) the ENIs (ergo IPs) of the interface VPCe are only accessible via some API calls (ec2:describeVpcEndpoints to get ENIs of VPCe and ec2:describeNetworkInterfaces to get IPs of ENIs). Currently we solve that via a Custom Resource, but I think it would be nice to have a method or property like [get]PrivateIPs.

What do you think (aws or cdk-team or aws-customers)? Is the above desirable?

flemjame-at-amazon commented 3 years ago

@bweigel private DNS is a feature of CDK now, as of https://github.com/aws/aws-cdk/pull/10780

bweigel commented 3 years ago

@flemjame-at-amazon yes, if you are the provider, you can create a custom domain for VPC endpoint services that are consumed by your customers. In the case of RabbitMQ on AmazonMQ this would be something along the lines of <broker-id>.mq.eu-central-1.amazonaws.com, which AWS creates as a custom domain for that endpoint service. Or at least that is my understanding of it.

However this is not the problem my suggestion would address. I am talking from the customer perspective, looking at the this domainname (<broker-id>.mq.eu-central-1.amazonaws.com). Sure, it might be a custom name, but still with little meaning to me as a customer. As a customer I might need something along the lines of rabbitmq.dev.my-domain.local or rabbitmq.prod.my-domain.local. To make names mean something.

You could argue to just add a CNAME Record that points to <broker-id>.mq.eu-central-1.amazonaws.com, but then the client cannot verify the certificate, bc. it is only valid for <broker-id>.mq.eu-central-1.amazonaws.com.

VPC endpoint service ≠ interface VPC endpoint

Does this make it clearer?

flemjame-at-amazon commented 3 years ago

@bweigel so you want to alias an endpoint to a user-defined name, and hide the certificate of the original service? Would there be any verification of the original service's cert?

bweigel commented 3 years ago

@bweigel so you want to alias an endpoint to a user-defined name, and hide the certificate of the original service?

Only if there is no other way. What I (or rather the people I talk to inside our org) want is a custom domain name (i.e. a name that can be interpreted in our context) for the interface VPC endpoints.

Would there be any verification of the original service's cert?

I guess not, if the SSL termination moved to another place.

dlants commented 2 years ago

I ran into this while trying to set up privatelink for ElasticSearch Cloud

It seems that ElasticSearch did not set up a default dns, so privateDnsEnabled does not work for the com.amazonaws.vpce.us-east-1.vpce-svc-0e42e1e06ed010238 service. Instead you have to create your own private hostedZone with a Cname that maps * to your private endpoint dns.

Naively, I attempted:

    const hostedZone = new route53.PrivateHostedZone(
      this,
      "ElasticSearchHostedZone",
      {
        zoneName: "vpce.us-east-1.aws.elastic-cloud.com",
        vpc: myVpc,
      }
    );

    new route53.CnameRecord(this, "ElasticSearchRecord", {
      zone: hostedZone,
      recordName: "*",
      domainName: endpoint.vpcEndpointDnsEntries[0]
    });

This failed with:

| CREATE_FAILED        | AWS::Route53::RecordSet
[RRSet of type CNAME with DNS name *.vpce.us-east-1.aws.elastic-cloud.com. does not contain exactly one resource record.]

Replacing the CnameRecord declaration with:

    const firstEntry = cdk.Fn.select(0, endpoint.vpcEndpointDnsEntries);
    const entryParts = cdk.Fn.split(':', firstEntry);
    const primaryDNSName = cdk.Fn.select(1, entryParts);

    new route53.CnameRecord(this, "ElasticSearchRecord", {
      zone: hostedZone,
      recordName: "*",
      domainName: primaryDNSName
    });

worked!

mitchlloyd commented 1 year ago

A lot has changed since the original issue post: PrivateLink services can use private dns now, the default for privateDNS was changed. One port on service property doesn't seem to be a problem.

As far as I can tell there is still not a better way to work with the vpcEndpointDnsEntries, but that has enough nuance it should probably be opened as a new issue.

github-actions[bot] commented 1 year ago

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.