Issue: Stream logs using Kinesis Data Firehose -- Parameters / custom metadata

[jmm]

Hello,

The AWS documentation on configuring New Relic as the destination for a Kinesis Data Firehose says:

Parameters - optional Kinesis Data Firehose includes these key-value pairs in each HTTP call. These parameters can help you identify and organize your destinations.

That seems to fall under this general instruction from the New Relic documentation:

Configure and review the remaining settings.

Are those Kinesis Data Firehose destination Parameters used or manifested in any way in New Relic?

I'm wondering if they can be used to associate attributes or some other kind of custom metadata with the logs. Otherwise it seems the only way to do that via this integration would be to configure a transformation Lambda, convert logs to JSON, and add attributes to them.

Doc information (don't delete this section)

• https://docs.newrelic.com/docs/logs/enable-log-management-new-relic/enable-log-monitoring-new-relic/stream-logs-using-kinesis-data-firehose/

Thanks

[rhetoric101]

@jmm, thanks for your question about Kinesis Data Firehose parameters. I'll check to see if there's anything we need to add to our documentation to clarify this.

[jmm]

Thanks @rhetoric101! It doesn't make too much sense for the AWS documentation to mention that in the context of the New Relic integration if New Relic doesn't do anything with it.

And https://github.com/newrelic/aws-log-ingestion has an NR_TAGS environment variable that can be used to associate custom metadata with the logs being shipped. (Despite the fact that it refers to "tags" I assume it actually means attributes, though I've asked for clarification.) Even though that's of limited usefulness, it would be an advantage over the Firehose integration if it doesn't even support that level of metadata configuration.

[rhetoric101]

@jmm, it may be a while before we can get this documented properly, but when defining a Kinesis Firehose Delivery Stream destination, you can define parameters, as you noted. This is the purpose of the Parameters section. These parameters are key-value pairs that will be added to every log message that is sent using the delivery stream.

I'm afraid I don't have more details for you at the moment about what we'll be adding to our documentation, but you could contact New Relic Support or post on Explorers Hub if you need help with your specific situation.

[jmm]

@rhetoric101 thanks for your reply. Ok, so they must be added in the form of attributes. It will be good to document that of course, but I'll proceed on that premise for now.

[rhetoric101]

@jmm, I'm going to go ahead and close this issue and pull your helpful details into our internal issue tracking system. Thanks again for highlighting this gap in our docs!

[rhetoric101]

This will be followed up in DOC-6702.

[austin-schaefer]

Reopening per our decision to keep GH issues open until the relevant Jira is closed

[pizzaz93]

@barbnewrelic Is this fixed with #5826?

[barbnewrelic]

@pizzaz93 Yes, thanks for checking!

[austin-schaefer]

Looks like this was fixed in #5826 and just wasn't closed