Heka multiple config

[DonHarishAxe]

Hey guys. I have ten configuration files which executes ten different scripts using the ProcessInput plugin and dumps it to a different index and type in ElasticSearch. Now in these configuration files, most of the lines are common regarding the Decoder and Splitter being used. So I am thinking of having a common config file having these common plugins and importing this to all of the files as it would reduce the lines and also make the process of changing the configurations easier. But however I could'nt find a suitable Heka feature to do this. Help me with this. Thanks.

[robison]

Use the configuration directory option?

[DonHarishAxe]

How come? I need to use the common config file with all the other config files. So i must import the common file and all the config files into the same directory. But that will load all of the plugins and clash unnecessarily. Or else i must make copies of common file and put it in a seperate directory with each config file correspondingly. That will not serve its purpose. Btw thanks for replying. Hope you prove me wrong and come up with a solution.

[robison]

This discussion is most likely better suited for the mailing list, as it really isn't an issue with the software, per se.

[DonHarishAxe]

To be simple, I am actually asking if there is an option in Heka to specify two configuration files together instead of specifying it in a directory. Or if there is any provision in toml format itself to import another toml file.

[robison]

Use ProcessDirectoryInput?

[DonHarishAxe]

ProcessDirectoryInput can dump into a single output plugin and into the same index and type in my case as far as I know. But as i have already told, I need it to be dumped onto different indices. Moreover I do not know why you brought this into this discussion. Because ProcessDirectoryInput just takes the ProcessInput block of the config files. So I feel this isn't related to my problem.

[robison]

Any Heka messages that are created from decoded inputs can be sent via any output. I feel as though your issue may be one of misunderstanding how the Heka message router works. You have the option of setting up message matchers on your output plugin configuration to route Heka messages towards indexes. e.g.:

[ElasticSearchOutputDailyIndices]
type = 'ElasticSearchOutput'
connect_timeout = 5000
encoder = 'ESJsonEncoderDailyIndices'
flush_count = 10000
flush_interval = 60000
http_disable_keepalives = true
message_matcher = '(Type != "nginx" && Type != "httpd" && Type != "stats")'
server = 'http://localhost:9200'
use_buffering = true

[ElasticSearchOutputDailyIndices.buffering]
cursor_update_count = 1
full_action = 'drop'
max_buffer_size = 8589934592
max_file_size = 67108864

[ElasticSearchOutputDailyIndices.retries]
delay = '250ms'
max_delay = '8s'
max_retries = -1

[ESJsonEncoderDailyIndices]
type = 'ESJsonEncoder'
es_index_from_timestamp = true
fields = [ 'DynamicFields', 'Hostname', 'Logger', 'Payload', 'Pid', 'Severity', 'Timestamp' ]
id = '%{UUID}'
index = 'heka-%{colo}-%{Type}-%{%Y.%m.%d}'
raw_bytes_fields = [ 'audit_details', 'audit_log_trailer', 'request_headers', 'response_headers' ]
timestamp = '%Y-%m-%dT%H:%M:%S.%f'
type_name = '%{Type}'

[ESJsonEncoderDailyIndices.field_mappings]
Severity = 'level'
Timestamp = '@timestamp'
Type = '_type'
Uuid = '_id'

To answer your questions, there is no option at present to specify two configuration files, nor is there an option to have one TOML configuration file call in another TOML configuration file.

[michaelgibson]

@DonHarishAxe: Could you post your configs? Perhaps that would make it clearer what you are trying to do.