AWS Data Firehose HTTP Endpoint Input Plugin

[syedmhashim]

Use Case

I would like to request for an AWS Data Firehose HTTP Endpoint Input Plugin. I'd be using this in our monitoring infrastructure to transfer CloudWatch Logs from an AWS Data Firehose stream to a Telegraf server where I would be doing some processing and aggregation of the logs data and create metrics from the logs. These would then be pushed to a Loki server using the loki output plugin, from where these metrics would be pulled by our AWS Managed Grafana Server.

Expected behavior

Successful transfer of CloudWatch Logs from AWS Data Firehose stream to a Telegraf server.

Actual behavior

Initially, I decided to use the existing _http_listenerv2 input plugin with the Data Firehose's HTTP Endpoint destination option. However, HTTP Endpoint destination option in Data Firehose have some requirements which I was not able to achieve using the _http_listenerv2 input plugin. Some of the constraints are as follow:

• Firehose expects the response body to be in a specific format. If the format is not met, it mark it as an error and retries the request and continues to do so until the threshold is reached.
• For authentication, it uses a specific header i.e X-Amz-Firehose-Access-Key, and expects the recipient server to handle authentication using this header.
• The request body sent by Firehose is in json format, and the data field is base64 encoded.

For more details, please see here

Additional info

As an alternative, we instead push the logs from the Firehose to an S3 bucket. These logs then get pulled from the S3 bucket to an EC2 server and placed in folder from where Promtail scrapes and pushes to loki. We want to eliminate the S3 bucket and Promtail server and instead use Telegraf. I'm already working on the Data Firehose Input Plugin for this purpose and would be more than happy to share a PR.

[srebhan]

Next steps: Someone needs to implement this plugin.

[syedmhashim]

Hi @srebhan. I would be more than happy to work on this plugin. Infact, I already have a working solution for this plugin. I can share a draft PR for early review