AWS Data Firehose HTTP Endpoint Input Plugin

syedmhashim commented 2 months ago

Use Case

I would like to request for an AWS Data Firehose HTTP Endpoint Input Plugin. I'd be using this in our monitoring infrastructure to transfer CloudWatch Logs from an AWS Data Firehose stream to a Telegraf server where I would be doing some processing and aggregation of the logs data and create metrics from the logs. These would then be pushed to a Loki server using the loki output plugin, from where these metrics would be pulled by our AWS Managed Grafana Server.

Expected behavior

Successful transfer of CloudWatch Logs from AWS Data Firehose stream to a Telegraf server.

Actual behavior

Initially, I decided to use the existing _http_listenerv2 input plugin with the Data Firehose's HTTP Endpoint destination option. However, HTTP Endpoint destination option in Data Firehose have some requirements which I was not able to achieve using the _http_listenerv2 input plugin. Some of the constraints are as follow:

Firehose expects the response body to be in a specific format. If the format is not met, it mark it as an error and retries the request and continues to do so until the threshold is reached.
For authentication, it uses a specific header i.e X-Amz-Firehose-Access-Key, and expects the recipient server to handle authentication using this header.
The request body sent by Firehose is in json format, and the data field is base64 encoded.

For more details, please see here

Additional info

As an alternative, we instead push the logs from the Firehose to an S3 bucket. These logs then get pulled from the S3 bucket to an EC2 server and placed in folder from where Promtail scrapes and pushes to loki. We want to eliminate the S3 bucket and Promtail server and instead use Telegraf. I'm already working on the Data Firehose Input Plugin for this purpose and would be more than happy to share a PR.

srebhan commented 2 months ago

Next steps: Someone needs to implement this plugin.

syedmhashim commented 2 months ago

Hi @srebhan. I would be more than happy to work on this plugin. Infact, I already have a working solution for this plugin. I can share a draft PR for early review

srebhan commented 1 month ago

@syedmhashim sorry for the late reply! I would be very happy to review your work!

syedmhashim commented 1 month ago

@srebhan Thanks for your response. Here's a draft PR. I still need to write the tests but thought it'd be good to have some early feedback. Looking forward to your comments

influxdata / telegraf