This project demonstrates how to create an automated pipeline to stream CloudWatch Logs into OpenSearch Serverless for real-time log analytics. CloudWatch Logs collects and streams log data. Amazon Data Firehose buffers and delivers to S3. A Lambda function transforms logs into JSON for OpenSearch. S3 and SQS enable durable storage and event notifications. OpenSearch Ingestion pulls logs and loads into a Serverless domain.
Jason Salcido, Senior Startup Solutions Architect at Amazon Web Services, leverages his deep expertise in cloud, analytics, integration, and data security to help clients across industries like financial services, insurance, energy, and utilities transform through technology. With nearly 30 years as a visionary technology leader, he excels at understanding needs and designing elegant solutions. Jason builds trusted advisor relationships through his technical excellence, strategic perspective, passion, and ability to inspire teams. His comprehensive enterprise architectures and innovative prototypes drive value for global organizations.
julianwood
commented
4 months ago
Description
This project demonstrates how to create an automated pipeline to stream CloudWatch Logs into OpenSearch Serverless for real-time log analytics. CloudWatch Logs collects and streams log data. Amazon Data Firehose buffers and delivers to S3. A Lambda function transforms logs into JSON for OpenSearch. S3 and SQS enable durable storage and event notifications. OpenSearch Ingestion pulls logs and loads into a Serverless domain.
language
English
runtime
Python
Level
400
Type
Examples
Use case
Observability
Primary image
https://github.com/aws-samples/cloudwatch-to-opensearch/blob/main/architecture.png
IaC framework
AWS CloudFormation
AWS Serverless services used
Description headline
Stream CloudWatch Logs to OpenSearch Serverless for real-time log analytics using a serverless pipeline.
Repo URL
https://github.com/aws-samples/cloudwatch-to-opensearch
Additional resources
Links to additional information:
Author Name
Jason Salcido
Author Image URL
https://avatars.githubusercontent.com/u/151649141?v=4
Author Bio
Jason Salcido, Senior Startup Solutions Architect at Amazon Web Services, leverages his deep expertise in cloud, analytics, integration, and data security to help clients across industries like financial services, insurance, energy, and utilities transform through technology. With nearly 30 years as a visionary technology leader, he excels at understanding needs and designing elegant solutions. Jason builds trusted advisor relationships through his technical excellence, strategic perspective, passion, and ability to inspire teams. His comprehensive enterprise architectures and innovative prototypes drive value for global organizations.
Author Twitter handle
No response
Author LinkedIn URL
https://www.linkedin.com/in/salcido/
leave
No response