search

MAAP-Project / maap-hec-aws

2 stars 1 forks source link

Implement workaround logstash solution for job status and metrics data processing #32

Open mkarim2017 opened 2 years ago

mkarim2017 commented 2 years ago

Job status and metrics information will go from the endpoints to mozart/metrics ES/Kibana through SQS. So there are tow logstash configuration is needed, one from end point to SQS and another from SQS to mozart/metrics.

As we dont have SQS available now, we need to have a workaround solution where the status/metrics information will ssh to mozart/metrics machine and then logstash will process them and send it to mozart/metrics ES/Kibana.

mkarim2017 commented 2 years ago

Created a separate logstash.conf file that reads the logfile which have been ssh-ed map env and insert it to Elastic search.

Screen Shot 2022-06-02 at 1 15 25 PM

Run it : bin/logstash -f logstash_metrics.conf --log.level debug

Screen Shot 2022-06-02 at 12 06 32 PM Screen Shot 2022-06-02 at 12 17 56 PM