kinesis_stream_name as token for config reference

[nickmaccarthy]

Dont know if its possible or not, but it would be nice have the kinesis_stream_name as a token we can reference in the rest of the logstash config. In my particular case, it would allow us to more easily specify which elasticsearch index this stream data should go to.

For example, assuming we are pulling from our stream name foobar we could add this in our logstash config to ensure it goes to the right index in elasticsearch:

input {
    kinesis {
         kinesis_stream_name => "foobar"
    }
}

output {
    if [kinesis_stream_name] == "foobar" {
        elasticsearch {
            hosts => ['esh.mycompany.com']
            index => "kinesis_foobar-%{+YYYY.MM.dd}"
        }
    }
}

In fact, it would be great if all of the input config options could be referenced.

[codekitchen]

This should be possible using the new @metadata functionality: https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#metadata

I'm sure there's a way to have this input plugin add these config values as metadata fields, so that they can be referenced by config but don't pollute the output.

[nickmaccarthy]

@codekitchen - that would be great! Do you know if you can reference those fields in outputs too, specifically it he elasticsearch index config?

example:

elasticsearch {
   hosts => ['esh.mycompany.com']
   index => "kinesis_[@metadata][kinesis_stream_name]-%{+YYYY.MM.dd}"
}

[codekitchen]

I think that would work with syntax "kinesis_%{[@metadata][kinesis_stream_name]}-%{+YYYY.MM.dd}"

[codekitchen]

@nickmaccarthy thinking about this some more, it seems like something that should probably be implemented in logstash itself, rather than in this plugin, so that all plugins get this enhancement. Would you mind moving this feature request over to https://github.com/elastic/logstash/issues ?

[dichenli]

As a similar request, it would be great if application_name can be referenced. Kinesis stream names may contain upper case letters (not allowed as ES index name) and sometimes not customizable. I've tried "%{[@metadata][application_name]}-%{+YYYY.MM.dd}" or "%{application_name}-%{+YYYY.MM.dd}", they won't work.

[codekitchen]

I'm not sure if this ever got filed against logstash core. I still think it makes the most sense to implement this there, it'd be silly for every plugin to implement it individually.