search

aws / aws-cdk

The AWS Cloud Development Kit is a framework for defining cloud infrastructure in code
https://aws.amazon.com/cdk
Apache License 2.0
11.5k stars 3.84k forks source link

(msk): associate secret with cluster #20490

Open ianbruton opened 2 years ago

ianbruton commented 2 years ago

Describe the bug

I have 2 stacks

mskStack and opensearchStack

mskStack creates secret containing user credentials for my mskCluster

In kafkaStack:

kafkaCluster.addUser("KafkaUsername");

The secret created in the mskStack is needed again in my opensearchStack as I have defined a lambda function to use mskEventListener to send data from msk to my opensearch domain.

Due to the secret being created behind the scenes by the mskCluster construct, I do not have a secret defined in my mskStack to output, so the secret must be referenced as an existing resource. I tried to accomplish this via the following implementation:

In opensearchStack:

const user = secretsmanager.Secret.fromSecretNameV2(this, "kafkaClusterUser", "AmazonMSK_KafkaUsername")

mskLambdaConsumer.addEventSource(new events.ManagedKafkaEventSource({
  batchSize: 100,
  clusterArn: props.kafkaClusterArn,
  maxBatchingWindow: cdk.Duration.seconds(10),
  secret: user,
  startingPosition: lambda.StartingPosition.TRIM_HORIZON,
  topic: "topicName",
}));

The above code fails, however, if I remove the secret: user, attribute line from the mskLambdaConsumer.addEventSource definition, then the mskEventSource creates without a problem.

Furthermore, if I manually create the EventSource in the AWS console, and select the very same secret to use for the mskEventSource Authentication option, the mskEventSource successfully links to the secret and can access the mskCluster.

This establishes the following:

The breakdown is when cdk uses fromSecretNameV2 to retrieve the secret and then pass the retrieved value to the mskEventSource

Expected Behavior

A valid secret to be retrieved from secretsManager

Current Behavior

OpenSearchStack failed: Error: The stack named OpenSearchStack failed to deploy: UPDATE_ROLLBACK_COMPLETE: Resource handler returned message: "Invalid request provided: The secret provided in 'sourceAccessConfigurations' is not associated with cluster . Please provide a secret associated with the cluster. (Service: Lambda, Status Code: 400, Request ID: , Extended Request ID: null)" (RequestToken: , HandlerErrorCode: InvalidRequest), Resource handler returned message: "Invalid request provided: The secret provided in 'sourceAccessConfigurations' is not associated with cluster . Please provide a secret associated with the cluster. (Service: Lambda, Status Code: 400, Request ID: , Extended Request ID: null)" (RequestToken: 695cbaba-3861-87ad-7c33-8394ea84f506, HandlerErrorCode: InvalidRequest) at prepareAndExecuteChangeSet (/PATH/node_modules/aws-cdk/lib/api/deploy-stack.ts:385:13) at processTicksAndRejections (node:internal/process/task_queues:96:5) at CdkToolkit.deploy (/PATH/node_modules/aws-cdk/lib/cdk-toolkit.ts:209:24) at initCommandLine (/PATH/node_modules/aws-cdk/lib/cli.ts:341:12)

The stack named OpenSearchStack failed to deploy: UPDATE_ROLLBACK_COMPLETE: Resource handler returned message: "Invalid request provided: The secret provided in 'sourceAccessConfigurations' is not associated with cluster . Please provide a secret associated with the cluster. (Service: Lambda, Status Code: 400, Request ID: , Extended Request ID: null)" (RequestToken: , HandlerErrorCode: InvalidRequest), Resource handler returned message: "Invalid request provided: The secret provided in 'sourceAccessConfigurations' is not associated with cluster . Please provide a secret associated with the cluster. (Service: Lambda, Status Code: 400, Request ID: , Extended Request ID: null)" (RequestToken: 695cbaba-3861-87ad-7c33-8394ea84f506, HandlerErrorCode: InvalidRequest)

Reproduction Steps

const user = secretsmanager.Secret.fromSecretNameV2(this, "kafkaClusterUser", <SECRET_NAME>)
<LAMBDA_FUNCTION>.addEventSource(new events.ManagedKafkaEventSource({
  batchSize: 100,
  clusterArn: <CLUSTER_ARN>,
  maxBatchingWindow: cdk.Duration.seconds(10),
  secret: user,
  startingPosition: lambda.StartingPosition.TRIM_HORIZON,
  topic: <EXISTING_TOPIC_NAME>,
}));

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.2.0

Framework Version

2.25.0

Node.js Version

7.15.1

OS

OSX Monterey, 12.4

Language

Typescript

Language Version

~3.9.7

Other information

No response

peterwoodworth commented 2 years ago

Seems to me this is expected, our secret property links to MSK docs on this.

Based on these docs, it looks like we can't do anything about this without a custom resource

To associate the secret with your cluster, use either the Amazon MSK console, or the BatchAssociateScramSecret operation

We can potentially implement this with a custom resource - that would be a feature request. Additionally, we can request CloudFormation to natively accept a secret arn for a cluster to associate them.

ianbruton commented 2 years ago

Hi @peterwoodworth , thanks for the reply but I don't think you are describing the issue that I am having

The secret is already successfully associated with my cluster via the following cdk command in my mskStack: kafkaCluster.addUser("ian");

Verification of association with cluster (image is from cluster details page in AWS Console): image

So the secret is created and already associated with my cluster.

All of this is happening in my mskStack.

After that stack completes deploying I run my opensearchStack and use the following command to get the secret created in the first stack. 

const user: cdk.aws_secretsmanager.ISecret = secretsmanager.Secret.fromSecretPartialArn(this, "kafkaClusterUser", `arn:aws:secretsmanager:${props.env?.region}:${props.env?.account}:secret:AmazonMSK_${props.kafkaClusterName}_${props.kafkaUsername}`)

the immediate next line in the stack creates an output of the secrets name (to test it is getting the correct secret) 

new cdk.CfnOutput(this, "user", { value: user.secretName.toString() });

verification of successful retrieval of secret (screenshot from my terminal): image

As you can see from both images, I am getting the user that is associated with the cluster (another way I can tell that I am getting the correct user is that I only have one secret in this account, and it is this user)

So, all of that seems to be working. The issue I am having though is when trying to pass the secret to the Lambda mskEventSource

mskLambdaConsumer.addEventSource(new events.ManagedKafkaEventSource({
  batchSize: 100,
  clusterArn: props.kafkaClusterArn,
  maxBatchingWindow: cdk.Duration.seconds(10),
  secret: user,
  startingPosition: lambda.StartingPosition.TRIM_HORIZON,
  topic: "someTopic",
}));

Rollback error in cloudformation:

Resource handler returned message: "Invalid request provided: The secret provided in 'sourceAccessConfigurations' is not associated with cluster arn:aws:kafka:<region>:<accountNumber>:cluster/NewKafkaCluster/12345678-1234-1234-1234-123456789123-1. Please provide a secret associated with the cluster.

the example in the CDK docs is essentially the same thing, except they create a new secret: https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_lambda_event_sources.ManagedKafkaEventSource.html

They even mention in a comment that the secret needs to be associated with the cluster, but I have already demonstrated that the secret is indeed associated with the cluster.

So I am quite unsure as to what I could be doing incorrectly as my implementation seems to be accurate as per the documentation AND, if I create the mskEvent Source manually in the AWS Lambda Console with the same secret, it works fine.

The only thing left that I can think of to be incorrect is the implementation in CDK itself. Especially if manually it works fine but not through CDK (with all the same parameters)

Cheers

peterwoodworth commented 2 years ago

Ahhh my apologies, I'm not too familiar with our MSK constructs and I hadn't noticed that our addUser method actually does make use of a custom resource which works around the limitation I described earlier. Thanks for pointing this out.

It's possible that this could be an issue on CloudFormation's end? Check your CloudFormation template - what ends up in the Lambda::EventSourceMapping.SourceAccessConfiguration.URI? Is it an expected value?

ianbruton commented 2 years ago

No worries!

Here is the full section for the event source:

"senddatatoopensearchalphaKafkaEventSourceNewOpenSearchStacksenddatatoopensearch<SUFFIX1>": {
  "Type": "AWS::Lambda::EventSourceMapping",
  "Properties": {
   "FunctionName": {
    "Ref": "senddatatoopensearchalpha<SUFFIX2>"
   },
   "BatchSize": 100,
   "EventSourceArn": {
    "Fn::ImportValue": "NewKafkaStack:ExportsOutputRefKafkaCluster<SUFFIX3>"
   },
   "MaximumBatchingWindowInSeconds": 10,
   "SourceAccessConfigurations": [
    {
     "Type": "SASL_SCRAM_512_AUTH",
     "URI": "arn:aws:secretsmanager:<REGION>:<ACCOUNT>:secret:AmazonMSK_NewKafkaCluster_ian"
    }
   ],
   "StartingPosition": "TRIM_HORIZON",
   "Topics": [
    "SOME_TOPIC"
   ]
  },
  "Metadata": {
   "aws:cdk:path": "NewOpenSearchStack/send-data-to-opensearch-alpha/KafkaEventSource:NewOpenSearchStacksenddatatoopensearch<ID><SOME_TOPIC>/Resource"
  }
 },

It appears to have an error in this section:

"SourceAccessConfigurations": [
    {
     "Type": "SASL_SCRAM_512_AUTH",
     "URI": "arn:aws:secretsmanager:<REGION>:<ACCOUNT>:secret:AmazonMSK_NewKafkaCluster_ian"
    }

I believe this line... "URI": "arn:aws:secretsmanager:<REGION>:<ACCOUNT>:secret:AmazonMSK_NewKafkaCluster_ian"

should look like...

"URI": "arn:aws:secretsmanager:<REGION>:<ACCOUNT>:secret:AmazonMSK_NewKafkaCluster_ian<AWS-APPLIED-SUFFIX>"

OR:

"URI": "arn:aws:secretsmanager:<REGION>:<ACCOUNT>:secret:AmazonMSK_NewKafkaCluster_ian<WILDCARD>"

As the actual arn listed in secrets manager looks like this: arn:aws:secretsmanager:eu-central-1:646797388334:secret:AmazonMSK_NewKafkaCluster_ian-ABC123

Cheers

peterwoodworth commented 2 years ago

fromSecretNameV2() actually won't return the complete arn - it will only return a partial arn.

This is fine in most cases, but this case could potentially be an exception.

Can you try using fromSecretAttributes() or fromSecretCompleteArn() to verify this?

We should call this out better in our secret documentation - it's not clear that this will only return a partial arn

ianbruton commented 2 years ago

okay, so I have done some testing and have some results

I have tried each of these options:

    const secret: cdk.aws_secretsmanager.ISecret = secretsmanager.Secret.fromSecretPartialArn(this, "kafkaClusterSecret", `arn:aws:secretsmanager:${props.env?.region}:${props.env?.account}:secret:AmazonMSK_${props.kafkaClusterName}_${props.kafkaUsername}`)
    new cdk.CfnOutput(this, "MSKSecretName", { value: secret.secretName.toString() });
    new cdk.CfnOutput(this, "MSKSecretValue", { value: secret.secretValue.toString() });
    new cdk.CfnOutput(this, "MSKSecretArn", { value: secret.secretArn });
    // new cdk.CfnOutput(this, "MSKSecretFullArn", { value: secret.secretFullArn! }); This causes an error

    const secret2: cdk.aws_secretsmanager.ISecret = secretsmanager.Secret.fromSecretAttributes(this, "kafkaClusterSecret2", {
      secretPartialArn: `arn:aws:secretsmanager:${props.env?.region}:${props.env?.account}:secret:AmazonMSK_${props.kafkaClusterName}_${props.kafkaUsername}`,
    })
    new cdk.CfnOutput(this, "MSKSecretName2", { value: secret2.secretName.toString() });
    new cdk.CfnOutput(this, "MSKSecretValue2", { value: secret2.secretValue.toString() });
    new cdk.CfnOutput(this, "MSKSecretArn2", { value: secret2.secretArn });
    // new cdk.CfnOutput(this, "MSKSecretFullArn2", { value: secret2.secretFullArn! }); This causes an error

    const secret3: cdk.aws_secretsmanager.ISecret = secretsmanager.Secret.fromSecretAttributes(this, 'kafkaClusterSecret3', {
      secretCompleteArn: `arn:aws:secretsmanager:${props.env?.region}:${props.env?.account}:secret:AmazonMSK_${props.kafkaClusterName}_${props.kafkaUsername}-c71nyl`,
    });
    new cdk.CfnOutput(this, "MSKSecretName3", { value: secret3.secretName.toString() });
    new cdk.CfnOutput(this, "MSKSecretValue3", { value: secret3.secretValue.toString() });
    new cdk.CfnOutput(this, "MSKSecretArn3", { value: secret3.secretArn });
    new cdk.CfnOutput(this, "MSKSecretFullArn3", { value: secret3.secretFullArn! });

their output is as follows:

✨  Deployment time: 198.54s

Outputs:
NewOpenSearchStack.MSKSecretArn = arn:aws:secretsmanager:<region>:<account>:secret:AmazonMSK_NewKafkaCluster_ian
NewOpenSearchStack.MSKSecretArn2 = arn:aws:secretsmanager:<region>:<account>:secret:AmazonMSK_NewKafkaCluster_ian
NewOpenSearchStack.MSKSecretArn3 = arn:aws:secretsmanager:<region>:<account>:secret:AmazonMSK_NewKafkaCluster_ian-c71nyl
NewOpenSearchStack.MSKSecretFullArn3 = arn:aws:secretsmanager:<region>:<account>:secret:AmazonMSK_NewKafkaCluster_ian-c71nyl
NewOpenSearchStack.MSKSecretName = AmazonMSK_NewKafkaCluster_ian
NewOpenSearchStack.MSKSecretName2 = AmazonMSK_NewKafkaCluster_ian
NewOpenSearchStack.MSKSecretName3 = AmazonMSK_NewKafkaCluster_ian
NewOpenSearchStack.MSKSecretValue = {{resolve:secretsmanager:arn:aws:secretsmanager:<region>:<account>:secret:AmazonMSK_NewKafkaCluster_ian:SecretString:::}}
NewOpenSearchStack.MSKSecretValue2 = {{resolve:secretsmanager:arn:aws:secretsmanager:<region>:<account>:secret:AmazonMSK_NewKafkaCluster_ian:SecretString:::}}
NewOpenSearchStack.MSKSecretValue3 = {{resolve:secretsmanager:arn:aws:secretsmanager:<region>:<account>:secret:AmazonMSK_NewKafkaCluster_ian-c71nyl:SecretString:::}}

already at this point we can see that passing the suffix '-c71nyl' seems to fix the error preventing us to retrieve the full ARN, which is pretty redundant because in order to retrieve the full ARN I had to pass the full ARN, and if I alraedy have the full ARN why would I need to make the call to get it?

What I would expect is to pass the secret name 'AmazonMSK_NewKafkaCluster_ian' to a function and have it return the full ARN.

This also "solves" the issue with the EventSource as passing secret3 (the one with the suffix in the ARN) does not error.

props.kafkaTopics.forEach(function (value) {
  mskLambdaConsumer.addEventSource(new events.ManagedKafkaEventSource({
    batchSize: 100,
    clusterArn: props.kafkaClusterArn,
    maxBatchingWindow: cdk.Duration.seconds(10),
    secret: secret3,
    startingPosition: lambda.StartingPosition.TRIM_HORIZON,
    topic: value,
  }));
});
image

The problem with this is that it requires me to know the suffix, something that I can't know before executing the CDK code and must get from the console. This is the exact issue that I am trying to automate.

Desired behaviour:

  1. Create the cluster and users in Kafka stack
  2. Output MSK generated secret name from Kafka stack to OpenSearch stack
  3. Get complete secret ARN returned with suffix
  4. Use Complete ARN to get secret
  5. Pass secret to MSK event source

OR

  1. Create the cluster and users in Kafka stack
  2. Output the complete secret ARN from Kafka stack (currently unsupported to the best of my knowledge)
  3. Pass Complete ARN to OpenSearch stack
  4. Use Complete ARN to get secret
  5. Pass secret to MSK event source

It should be possible to get any associated attribute of a secret with just its name.

ianbruton commented 2 years ago

@peterwoodworth Just wondering if there is any update regarding this issue?

kashif-malik-data-engineering commented 1 year ago

I am having a similar issue, any updates please?