Override output block configuration

[axelaris]

In Logstash 1.5 there is a new, special field, called @metadata. As you can see on example, It gives me ability to alter ES's index name depend on log message type. But ES output configuration is monolithic by default, so I've used additional output_override variable to give me ability to override output block to something like this:

properties:
  logstash_parser:
    output_override: |
     if [@metadata][log_type] == "app" {
       elasticsearch {
         host => "127.0.0.1:9200"
         protocol => "http"
         flush_size => 10
         idle_flush_time => 3
         index => "logs-app-%{+YYYY.MM.dd}"
         document_type => "%{@type}"
         manage_template => false
       }
     } else {
       elasticsearch {
         host => "127.0.0.1:9200"
         protocol => "http"
         flush_size => 10
         idle_flush_time => 3
         index => "logs-cf-%{+YYYY.MM.dd}"
         document_type => "%{@type}"
         manage_template => false
       }
     }

Would do you like it?

[dpb587]

I'd be a bit wary of merging this given we currently have the default elasticsearch output properties, along with the logstash_parser.outputs property. This would add a third output property which is extremely similar to .outputs. I think a more correct solution would be a property which allows disabling the the default elasticsearch output. Maybe something like logstash_parser.disable_builtin_output which defaults to false to preserve historical behavior.

I think an alternative to this PR which utilizes the current release would be to:

1. rely on filters to add a metadata field for your index partial name, e.g. @metadata[log_type]
2. update logstash_parser.elasticsearch_index property to be cf-%{[@metadata][log_type]}-%{+YYYY.MM.dd}

@mrdavidlaing, thoughts?

[axelaris]

Hi @dpb587, thank you for suggestion. However, use of logstash_parser.elasticsearch_index is reasonable for elasticsearch backend only. It will not help me if I want to push some logs to Cassandra, for example.

[axelaris]

ping?

[mrdavidlaing]

@axelaris, I'm sorry; but after consideration we've decided we not going to support this feature request at this time.

Feel free to submit an alternate PR to add a logstash_parser.disable_elasticsearch_output flag (defaulted to false).