Allow dynamic outputs (for sharding/partitioning)

[GoogleCodeExporter]

ElasticSearch and other backends might benefit from sharding, that is, be able 
to write to

elasticsearch://localhost:9200/logstash-YYYY-mm-dd/mylogs

In the above example, this would effectively partition your logs by day.

Further, each log type should probably get its own type in elasticsearch rather 
than being stored in the same /logstash/all index+type.

Some format string would be required here. Something similar to grok's syntax 
like:

elasticsearch://localhost:9200/logs-%{date}/%{type}

We'd have to figure out how to let folks specify the date format, but %{type} 
would become the LogStash::Event#type ('@type' in the event json)

This would be beneficial to other possible outputs like mongodb, mysql, hdfs, 
local files, etc.

Original issue reported on code.google.com by jls.semi...@gmail.com on 15 Feb 2011 at 8:25

[GoogleCodeExporter]

Original comment by jls.semi...@gmail.com on 15 Feb 2011 at 8:25

• Added labels: scalability

[GoogleCodeExporter]

%{} format works now and is a method on the event:

LogStash::Event#sprintf(format)

I'm going to also add time format support, so you can do: %{+STRFTIMESTRING} 
like %{+%Y-%m-%d} or maybe java date format like %{+YYYY-mm-dd} ? This date 
format would be based on the event's @timestamp

Original comment by jls.semi...@gmail.com on 21 Feb 2011 at 9:44

[GoogleCodeExporter]

Added %{+TIME FORMAT} support just now.

Original comment by jls.semi...@gmail.com on 19 Apr 2011 at 4:42

[GoogleCodeExporter]

Original comment by jls.semi...@gmail.com on 19 Apr 2011 at 4:47

• Changed state: Fixed