Currently, we employ the following approach for sending logs to OpenSearch: receiving application logs in Gzip format stored in S3. We then input this S3 Gzip file(collection of logs) into FluentD, where the file is uncompressed, processed as chunk files, and subsequently forwarded to OpenSearch.
Now, I am attempting to retrieve metrics detailing the count of logs in the uncompressed file and the count forwarded to OpenSearch.
We are using the following design to obtain metrics from FluentD
0
Currently, we employ the following approach for sending logs to OpenSearch: receiving application logs in Gzip format stored in S3. We then input this S3 Gzip file(collection of logs) into FluentD, where the file is uncompressed, processed as chunk files, and subsequently forwarded to OpenSearch.
Now, I am attempting to retrieve metrics detailing the count of logs in the uncompressed file and the count forwarded to OpenSearch.
We are using the following design to obtain metrics from FluentD
FluentD-> Monitoring_Agent -> DatadogAgent-> DataDog
However , we are facing issues aligning the logs count or file count form these metrics(fluentd.emit_records,fluentd.emit_count)
Please advise on the FluentD metrics property is giving correct count for logs