New Pattern: aws-msk-lambda

[dscpinheiro]

Lambda recently added support for MSK (Managed Streaming for Apache Kafka) as an event source (https://aws.amazon.com/about-aws/whats-new/2020/08/aws-lambda-now-supports-amazon-managed-streaming-for-apache-kafka-as-an-event-source/)

Use Case

With this new integration, customers can build Apache Kafka consumer applications with Lambda functions without needing to worry about infrastructure management.

Other

There's an open issue where the new property for Lambda (Topics - https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-lambda-eventsourcemapping.html#cfn-lambda-eventsourcemapping-topics) is not available in the CDK yet. Here's the GitHub issue I created on their repository: https://github.com/aws/aws-cdk/issues/10138

• [x] :wave: I may be able to implement this feature request

---

This is a :rocket: Feature Request

[Jacco]

Am working on this here https://github.com/Jacco/aws-solutions-constructs/tree/aws-mks-lambda

Seems that aws/aws-cdk#10138 is fixed now

I have a few questions about the implementation:

• maybe creating a MSK cluster in this is not so usefull, most cases you would want to attach a Lambda to an existing cluster, but if a clusrer needs to be created following:
  • usually MSK clusters should be in the private subnets but should that me configurable?
  • do we need to open the kafka ports in a SecurityGroup?
  • Currenly msk cluster name and version need to be specified, but the name can be autocreated, if we can do the same for
  • version existingClusterObj and clusterProps can become optional
  • also some alarms could be created (disk, #partitions, offline partitions)
• the lambda does not need to be in the same vpc because MSK can reach lambda through a NAT but should the case of a private route to the Lambda also be possible?
• ManagedKafkaEventSourceProps also supoprts a secret shoud that be passable in a Props property?

I still need to work on my branch a bit moving some stuff to core.