[New repo]: Optimizing Insights with CloudWatch Logs and OpenSearch Serverless

[skyjasonaws]

Description

This project creates an automated pipeline to stream CloudWatch Logs into OpenSearch Serverless for real-time log analytics. CloudWatch Logs collects and streams log data. Kinesis Firehose buffers and delivers to S3. A Lambda function transforms logs into JSON for OpenSearch. S3 and SQS enable durable storage and event notifications. OpenSearch Ingestion pulls logs and loads into a Serverless domain.

language

English

runtime

Python

Level

400

Type

Examples

Use case

Observability

Primary image

https://github.com/skyjasonaws/serverless-patterns/blob/main/cloudwatch-logs-to-opensearch/architecture.png

IaC framework

None

AWS Serverless services used

• [ ] Amazon API Gateway
• [ ] Amazon DynamoDB
• [ ] Amazon EventBridge
• [ ] AWS IoT
• [X] AWS Lambda
• [ ] Amazon Rekognition
• [X] Amazon S3
• [ ] AWS Step Functions
• [ ] Amazon SNS
• [X] Amazon SQS
• [ ] Amazon Transcribe
• [ ] Amazon Translate

Description headline

Stream CloudWatch Logs to OpenSearch Serverless for real-time log analytics using a serverless pipeline

Repo URL

https://github.com/skyjasonaws/serverless-patterns/cloudwatch-logs-to-opensearch

Additional resources

Key Components

• CloudWatch Logs
• Kinesis Data Firehose
• OpenSearch Service
• AWS Lambda
• Amazon S3
• Amazon SQS

Author Name

Jason Salcido

Author Image URL

https://avatars.githubusercontent.com/u/151649141?v=4

Author Bio

Jason Salcido, Senior Startup Solutions Architect at Amazon Web Services, leverages his deep expertise in cloud, analytics, integration, and data security to help clients across industries like financial services, insurance, energy, and utilities transform through technology. With nearly 30 years as a visionary technology leader, he excels at understanding needs and designing elegant solutions. Jason builds trusted advisor relationships through his technical brilliance, strategic perspective, passion, and ability to inspire teams. His comprehensive enterprise architectures and innovative prototypes drive value for global organizations.

Author Twitter handle

No response

Author LinkedIn URL

https://www.linkedin.com/in/salcido/

leave

No response

[jbesw]

Thanks - this will be live at https://serverlessland.com/repos/optimizing-insights-with-cloudwatch-logs-and-opensearch-serverless shortly.