(opensearch): Custom endpoint does not create CNAME

davidmckenzie commented 3 years ago

Attempting to create an OpenSearch domain with customEndpoint set fails with error 'Attribute 'DomainEndpoint' does not exist'

Reproduction Steps

    const esLogGroup = new _logs.LogGroup(this, 'esLogGroup', {
      logGroupName: `/es/${stage}-es-ingest`,
      retention: _logs.RetentionDays.ONE_YEAR
    })

    // elasticsearch cluster
    const esDomain = new _es.Domain(this, 'esDomain', {
      domainName: `${stage}-es-ingest`,
      version: _es.EngineVersion.OPENSEARCH_1_0,
      vpc,
      vpcSubnets: [{
        subnetType: _ec2.SubnetType.PRIVATE_WITH_NAT,
        availabilityZones: ['us-west-2a', 'us-west-2b']
      }],
      capacity: {
        masterNodes: 3,
        masterNodeInstanceType: context?.esMasterSize,
        dataNodes: context?.esDataInstanceCount,
        dataNodeInstanceType: context?.esDataSize,
      },
      customEndpoint: {
        domainName: `es-ingest.${stage}.x.com`,
        certificate: internalCert,
        hostedZone: stageZone
      },
      ebs: {
        volumeSize: context?.esDiskSize
      },
      enableVersionUpgrade: true,
      encryptionAtRest: { enabled: true },
      enforceHttps: true,
      logging: {
        slowSearchLogEnabled: true,
        slowSearchLogGroup: esLogGroup,
        appLogEnabled: true,
        appLogGroup: esLogGroup,
        slowIndexLogEnabled: true,
        slowIndexLogGroup: esLogGroup
      },
      nodeToNodeEncryption: true,
      removalPolicy: cdk.RemovalPolicy.RETAIN,
      securityGroups: [esSecGroup],
      tlsSecurityPolicy: _es.TLSSecurityPolicy.TLS_1_2,
      zoneAwareness: {
        enabled: true,
        availabilityZoneCount: 2
      },
      advancedOptions: {
        "override_main_response_version": "true"
      }
    });

    esLogGroup.grantWrite(new _iam.ServicePrincipal('es.amazonaws.com'))

What did you expect to happen?

OpenSearch domain should create with a custom endpoint configured, and should add a CNAME in Route53 to point to the endpoint address

What actually happened?

1:58:50 PM | CREATE_FAILED        | AWS::Route53::RecordSet                     | esDomainCnameRecordE7D8410D
Attribute 'DomainEndpoint' does not exist

        new RecordSet (/node_modules/@aws-cdk/aws-route53/lib/record-set.ts:119:23)
        \_ new CnameRecord (/node_modules/@aws-cdk/aws-route53/lib/record-set.ts:180:5)
        \_ new Domain (/node_modules/@aws-cdk/aws-opensearchservice/lib/domain.ts:1067:7)
        \_ new IngestStack (/lib/ingest.ts:199:22)
        \_ /bin/cdk.ts:19:3
        \_ Array.forEach (<anonymous>)
        \_ Object.<anonymous> (/bin/cdk.ts:17:6)
        \_ Module._compile (internal/modules/cjs/loader.js:1063:30)
        \_ Module.m._compile (/node_modules/ts-node/src/index.ts:1056:23)
        \_ Module._extensions..js (internal/modules/cjs/loader.js:1092:10)
        \_ Object.require.extensions.<computed> [as .ts] (/node_modules/ts-node/src/index.ts:1059:12)
        \_ Module.load (internal/modules/cjs/loader.js:928:32)
        \_ Function.Module._load (internal/modules/cjs/loader.js:769:14)
        \_ Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:72:12)
        \_ main (/node_modules/ts-node/src/bin.ts:198:14)
        \_ Object.<anonymous> (/node_modules/ts-node/src/bin.ts:288:3)
        \_ Module._compile (internal/modules/cjs/loader.js:1063:30)
        \_ Object.Module._extensions..js (internal/modules/cjs/loader.js:1092:10)
        \_ Module.load (internal/modules/cjs/loader.js:928:32)
        \_ Function.Module._load (internal/modules/cjs/loader.js:769:14)
        \_ Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:72:12)
        \_ internal/main/run_main_module.js:17:47

Environment

CDK CLI Version : 1.123.0
Framework Version: 1.123.0
Node.js Version: v14.16.0
OS : WSLv2, Ubuntu
Language (Version): all

Other

This same code (minus the advancedOptions) worked perfectly in the ElasticSearch module. I opted to delete and recreate the stack rather than go through the migration steps in the docs since there's no data on this particular domain. I have not tested whether the same stack works if migrated instead of recreated from scratch.

This is :bug: Bug Report

kaizencc commented 3 years ago

Hi @davidmckenzie! Thanks for bringing this to our attention. If I'm understanding you correctly, you had code that worked in the Elasticsearch module that you now want to use the Opensearch module? You might still have to use the migration steps even if your code wasn't previously deployed. In particular I'm curious to see if you have updated the context object to have different type suffixes.

davidmckenzie commented 3 years ago

That's correct, and the missing important bit that would have helped is that we just re-used the _es name for the opensearch module, a la:

import * as _es from '@aws-cdk/aws-opensearch';

And yep we used the .search instance types - the instance deployed fine, the only thing that was missing was the CNAME in Route 53 for the customEndpoint.

kaizencc commented 3 years ago

@peterwoodworth can you look into reproducing this bug? I'm adding the guidance tag and removing the bug tag for now, until we can reproduce.

peterwoodworth commented 3 years ago

I've dug into this a little bit, and I think I have a theory on what's happening here

@davidmckenzie, in your synthed template, under the RecordSet which is created there should be a property ResourceRecords that looks like this:

        "ResourceRecords": [
          {
            "Fn::GetAtt": [
              "DOMAIN_LOGICAL_ID",
              "DomainEndpoint"
            ]
          }
        ],

DOMAIN_LOGICAL_ID should be a real logical ID which references the OpenSearchService::Domain resource in the document. According to CloudFormation, this resource should have a DomainEndpont attribute.

Can you please confirm if this is the case for you? If so, I think that this is a bug with CloudFormation not properly implementing the DomainEndpoint Attribute since this is a new resource. I haven't been able to reproduce this because I don't have a certificate to test deployment with, but I'm pretty confident in my theory here. @kaizen3031593 do you think this sounds right? I'll cut CloudFormation team a ticket once I get confirmation from both of ya

peterwoodworth commented 3 years ago

Actually, I've thought of a way to test my theory without needing a certificate. Will update

peterwoodworth commented 3 years ago

Update: the DomainEndpoint attribute works fine for me. I need to get a certificate to see if I can actually reproduce this. @davidmckenzie, sharing your synth output for the Domain and RecordSet would help me to see if there's anything wrong in your template

davidmckenzie commented 3 years ago

This is what it looks like in the old ElasticSearch template, I've cut out most of the ES domain output for brevity, let me know if you need it and I'll sanitise and gist it :)

    "esDomainBCE5E5C3": {
      "Type": "AWS::Elasticsearch::Domain",
      "Properties": {
        "CognitoOptions": {
          "Enabled": false
        },
        "DomainEndpointOptions": {
          "CustomEndpoint": "es-ingest.x.x.com",
          "CustomEndpointCertificateArn": "arn:aws:acm:us-west-2:x:certificate/x",
          "CustomEndpointEnabled": true,
          "EnforceHTTPS": true,
          "TLSSecurityPolicy": "Policy-Min-TLS-1-2-2019-07"
        },
[..]
    "esDomainCnameRecordE7D8410D": {
      "Type": "AWS::Route53::RecordSet",
      "Properties": {
        "Name": "es-ingest.x.x.com.",
        "Type": "CNAME",
        "HostedZoneId": "x",
        "ResourceRecords": [
          {
            "Fn::GetAtt": [
              "esDomainBCE5E5C3",
              "DomainEndpoint"
            ]
          }
        ],
        "TTL": "1800"
      },
      "Metadata": {
        "aws:cdk:path": "infra-ingest-dev/esDomain/CnameRecord/Resource"
      }
    },

When migrating to Opensearch, the RecordSet entry looks exactly the same. The domain looks like:

    "esDomainBCE5E5C3": {
      "Type": "AWS::OpenSearchService::Domain",
      "Properties": {
[..]
          "DedicatedMasterCount": 3,
          "InstanceType": "m6g.large.search",
          "ZoneAwarenessEnabled": true,
          "DedicatedMasterType": "m6g.large.search"
        },
        "CognitoOptions": {
          "Enabled": false
        },
        "DomainEndpointOptions": {
          "CustomEndpointCertificateArn": "arn:aws:acm:us-west-2:x:certificate/x",
          "CustomEndpointEnabled": true,
          "EnforceHTTPS": true,
          "CustomEndpoint": "es-ingest.x.x.com",
          "TLSSecurityPolicy": "Policy-Min-TLS-1-2-2019-07"
        },
[..]
        "EngineVersion": "OpenSearch_1.0",

peterwoodworth commented 3 years ago

Weird, your template looks perfectly fine here, so this leads me to believe the problem lies on CloudFormation's end. Are you able to ask them for support?

github-actions[bot] commented 3 years ago

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

tutNichts commented 8 months ago

Facing the exact same issue, not even using a custom endpoint:

6:27:20 PM | CREATE_FAILED        | AWS::ECS::TaskDefinition                  | FeatureStack/TaskDefinition
Attribute 'DomainEndpoint' does not exist

My configuration:

const openSearch = new Domain(this, 'OpenSearch', {
            capacity: {
                dataNodeInstanceType: 't3.small.search',
                dataNodes: 4,
                masterNodeInstanceType: 't3.small.search',
                masterNodes: 4,
                multiAzWithStandbyEnabled: false,
            },
            domainName: truncate(`${props.stackName}-opensearch`, 28),
            ebs: {
                enabled: true,
                volumeSize: 10,
                volumeType: EbsDeviceVolumeType.GP3,
            },
            enableAutoSoftwareUpdate: true,
            encryptionAtRest: {
                enabled: true,
            },
            logging: {
                appLogEnabled: true,
                auditLogEnabled: true,
                slowIndexLogEnabled: true,
                slowSearchLogEnabled: true,
            },
            nodeToNodeEncryption: true,
            offPeakWindowEnabled: true,
            offPeakWindowStart: {
                hours: 0,
                minutes: 30,
            },
            useUnsignedBasicAuth: true,
            version: EngineVersion.OPENSEARCH_2_9,
            vpc,
            zoneAwareness: {
                enabled: true,
                availabilityZoneCount: 2,
            },
        });

Any ideas?

@davidmckenzie where you able to find a solution?

aws / aws-cdk