Support template syntax in hostname and tags field for LogDNA sink

[romatthe]

I'm currently trying to set up vector to ship logs from the docker source to the LogDNA sink. The LogDNA sink has a field hostname that it will use to determine the "source" of the log event. I tried using the template syntax to set the hostname based on the container_name field set by the docker source, but it seems this field does not accept the template syntax.

Here's a simple example of what I'm trying to set up

[sources.in]
  type = "docker"

[transforms.extract_app]
  # General
  type = "regex_parser"
  inputs = ["in"]
  drop_field = true
  field = "image"
  patterns = ['(.*/)*(?P<app>[a-zA-Z-]*):']

  # Types
  types.app = "string"

[sinks.console]
  # Encoding
  encoding.codec = "json"

  # General
  inputs = ["extract_app"]
  type = "console"

[sinks.logdna]
  type = "logdna"
  inputs = ["extract_app"]
  api_key = "${LOGDNA_API_KEY}"
  hostname = "{{ container_name }}"      # <---- This specifically...
  tags = ["{{ container_id }}"]                   # <---- ... and this

Without the ability to set this dynamically, it seems to me that there is no reliable way to indicate to LogDNA what instance this log is coming from.

The same goes for tags. I wanted to apply some tags that that take in some of the fields from the docker log output.

Would it be possible to support the use of templates in these fields?

[juchiast]

As I understand, template's values come from events, LogDNA batches many events together so each event could render a different hostname value. Should we group events that have the same hostname and tags?

[binarylogic]

Yes, this will require partitioning. The aws_cloudwatch_logs sink demonstrates this. Because these values are at the request level we'll need to partition requests across their unique values.

Let us know if you have any other questions. The sink partition code can be somewhat complex and we want to simplify that over time.