Feature Request: Add Diagnostics Logs Properties Fields to Input

[aj-devops-wi]

I am looking at implementing a solution that uses shared blob storage for multiple applications, and there does not appear to be an easy way to extract the ApplicationName, InstanceID, etc. from the Application Log? The CSV formatted logs appear to import fine, it's the Append Blob logs (that are just .txt files) that do not appear to be imported with the proper header format.

To clarify, this is for Application Logs configured under an Application Service (Web Site), under Diagnostic Logs. example_app_log_txt.txt

example_app_log_csv.txt

[brahmnes]

Hi @aj-devops-wi,

I am not sure what you are asking for. My guess is you are using the logstash-input-azureblob plugin to import log blobs from Azure blob containers written by Azure App Service?

The input makes no assumption about the format of the blob. You can see if you can write a filter that can parse each line and add the application name before the data reaches the output.

[aj-devops-wi]

That's a good point, but it would be great if this plugin supported the Web and App Diagnostics logs natively.

At a minimum, it would be great if some metadata about the blob was available to ELK, even as part of the message. In order to differentiate among App diagnostics log sources (for instance) I need to create a separate container for each in the Storage Account, and add my own tag. It would be great if the source could be auto extracted from a path, or even if the path was just made available as part of the message.

[brahmnes]

Hi @aj-devops-wi

For getting the blob path as part of the metadata, you are actually not the first one to ask about it. I think that addition is reasonable. You can submit a PR for it. It can look something like this:

• a new parameter to turn this on
• in the process function, make sure the path is passed to the enqueue_content function
• in enqueue_content, add the path as an additional property for the event object

As for supporting the web and app diagnostics logs natively, since this plugin is named simply as azureblob plugin, we probably cannot add the native support to it. Using a filter is actually quite simple. You can take a look at the grok filter: https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html

It seems to handle space delimited fields pretty well.