fluentd and rsyslog support; journal input support; use rsyslog conta…

[richm]

…iner

Run openshift-test.sh with USE_FLUENTD=true to test fluentd, use USE_FLUENTD=false to test rsyslog. USE_JOURNAL=true will use journal input to log collector for the system logs, otherwise, read from /var/log/messages style files. Because rsyslog-collector docker container configuration is not as flexible as fluentd container, we build a special rsyslog-perf-test container with our specific configuration, layered on top of the rsyslog-collector.

[richm]

@portante I removed that debug setting from rsyslog.conf.save - it wasn't being used in the test

[richm]

@t0ffel you are correct about the journal tag - the updated patch uses system.journal for messages read from the journal - this simplified the config somewhat As far as reading messages from fluentd - they will go through the kubernetes.* tag flow - not sure what they look like in elasticsearch, if they have some special identifier other than the pod name having fluentd in it.

[t0ffel]

@richm there are internal messages generated by fluentd. These messages have tags like 'fluentd.warn' or 'fluentd.info' etc. I'm not sure if they are logged to stdout, they probably are, but they'll have crippled metadata in this case.

[richm]

@t0ffel are those messages automatically generated and inserted into the event stream? I guess I would need to filter them in order to put them in the .operations index format.

[t0ffel]

@richm yes, they appear automatically. I actually hope to present something next week that will align various k8s/ose logs in a single data model. for the moment I usually just shove it as a separate ES document type in a separate <match **> section in the end.

[richm]

@t0ffel excellent. I'm currently working on using the journal to read logs from k8s in openshift rather than /var/log/containers/*.log

[lukas-vlcek]

Merged, closing... Thanks!