The PBS job status stored in S3 bucket needed further processing such as compute job processing efficiency, fetch number of cpu's used for the job, etc. The processed information was required to be consolidated in a common dashboard such as Kibana. To achieve this, tweak in the existing orchestration architecture was required so that we can automate the entire process.
Proposed solution
The architecture to be used for this setup is as follows:
a) New S3 object created ---Event Trigger----> AWS Lambda ---Processed Data is Logged to--------> AWS CloudWatch
b) New S3 object created ---Event Trigger----> AWS Lambda ---Processed Data is pushed to---> AWS ElasticSearch Domain
Reason for this pull request
The PBS job status stored in S3 bucket needed further processing such as compute job processing efficiency, fetch number of cpu's used for the job, etc. The processed information was required to be consolidated in a common dashboard such as Kibana. To achieve this, tweak in the existing orchestration architecture was required so that we can automate the entire process.
Proposed solution
The architecture to be used for this setup is as follows: a) New S3 object created ---Event Trigger----> AWS Lambda ---Processed Data is Logged to--------> AWS CloudWatch b) New S3 object created ---Event Trigger----> AWS Lambda ---Processed Data is pushed to---> AWS ElasticSearch Domain