search

aws-samples / serverless-patterns

Serverless patterns. Learn more at the website: https://serverlessland.com/patterns.
https://serverlessland.com
Other
1.48k stars 863 forks source link

New serverless pattern MSK and lambda integration. #1412

Closed Scorch116 closed 11 months ago

Scorch116 commented 11 months ago

Issue #1413

Description of changes: Added Cloudformation template to deploy resources.

Resources VPC, Subnets, and Routing tables and route associations

NAT gateways and elastic IPs are created to allow private subnets to access the internet while maintaining security.

A security group is created with inbound rules allowing access on ports 22, 443, 80, 9094, and 9092. It is associated with the MSK cluster and Lambda function.

IAM roles are created for the MSK cluster and EC2 instances

EC2 instance is launched in a public subnet, associated with the security group, and provided with user data to install Java and Kafka.

An MSK cluster is created with a specified Kafka version, broker node configurations, security groups, and client subnets. It also enables encryption in transit with plaintext.

A Lambda function is defined with a Python 3.9 runtime. It decodes and prints messages from Kafka records received as input.

This template sets up a networking infrastructure with a VPC, subnets, routing, security groups, NAT gateways, and an MSK cluster.

It also provisions an EC2 instance (Client machine) and a Lambda function for interacting with the MSK cluster.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

github-actions[bot] commented 11 months ago

@Scorch116 your 'example-pattern.json' is missing some key fields, please review below and address any errors you have

  1. language: language is not one of enum values: TypeScript,Node.js,Python,Java,Go,Rust,.NET,OpenAPI,YAML

_If you need any help, take a look at the example-pattern file._

Make the changes, and push your changes back to this pull request. When all automated checks are successful, the Serverless DA team will process your pull request.

boyney123 commented 11 months ago

Hey @Scorch116 ,

Looks like an invalid metadata.json file, can you review the changes and change the language?

Thanks!

julianwood commented 11 months ago

Thanks for the work on this. Also, we recently had this PR merged. To avoid duplicates, is your implementation different from: https://github.com/aws-samples/serverless-patterns/tree/main/msk-cfn-sasl-lambda/create-cluster-cfn

Scorch116 commented 11 months ago

Hello Julian,

I have pushed the changes to the metadata json file as asked.

Also, in relation to the other github article you kindly attached, there is some differences

In my template I include a lambda function with python code to print out information on the data process from the Kafka cluster. Also, I use an EC2 instance as the client machine to communicate with the MSK kafka topics. The kafka java dependencies are already included in the my template so the user dose not need to make any further steps here.

My template is designed to go through the full steps of Lambda -> MSK integration. I believe the other template is designed just to create a cluster in MSK with VPC etc..

If you need any further information, please feel free to reach out.

Kind regards

Ed

From: Julian Wood @.> Sent: Monday 19 June 2023 13:15 To: aws-samples/serverless-patterns @.> Cc: Scorch116 @.>; Mention @.> Subject: Re: [aws-samples/serverless-patterns] New serverless pattern MSK and lambda integration. (PR #1412)

Thanks for the work on this. Also, we recently had this PR merged. To avoid duplicates, is your implementation different from: https://github.com/aws-samples/serverless-patterns/tree/main/msk-cfn-sasl-lambda/create-cluster-cfn

— Reply to this email directly, view it on GitHubhttps://github.com/aws-samples/serverless-patterns/pull/1412#issuecomment-1597174753, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AMRKDJMJD54F4AL3AVSK7JTXMBGGXANCNFSM6AAAAAAZIDSC4A. You are receiving this because you were mentioned.Message ID: @.***>

Scorch116 commented 11 months ago

Hello David,

I have pushed the changes to the metadata json file as asked.

If you need any further information, please feel free to reach out.

Kind regards

Ed

From: David Boyne @.> Sent: Monday 19 June 2023 06:18 To: aws-samples/serverless-patterns @.> Cc: Scorch116 @.>; Mention @.> Subject: Re: [aws-samples/serverless-patterns] New serverless pattern MSK and lambda integration. (PR #1412)

Hey @Scorch116https://github.com/Scorch116 ,

Looks like an invalid metadata.json file, can you review the changes and change the language?

Thanks!

— Reply to this email directly, view it on GitHubhttps://github.com/aws-samples/serverless-patterns/pull/1412#issuecomment-1596575938, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AMRKDJPWYS2JN5LWAZUU3Q3XL7VKXANCNFSM6AAAAAAZIDSC4A. You are receiving this because you were mentioned.Message ID: @.***>

julianwood commented 11 months ago

To avoid duplication, going to close this for now. Some of the good work can be added to improve: https://github.com/aws-samples/serverless-patterns/tree/main/msk-cfn-sasl-lambda/create-cluster-cfn