I am using heka to push the logs to flume.
In heka I am able to see the logs getting tagged event by event(line by line)
But in flume I am not getting line by line. It merges 2 or 3 lines of log together and prints it as a chunk of message.
I am using source as multiport_syslogtcp and channel as memory channel, sink as file_roll.
Example log line:
ggggggggggggg
hhhhhhhhhhhhh
Example output in heka:
2016/08/24 09:11:57 {"message":"ggggggggggggggggggggggggggggggggggggggggggggggggg","@timestamp":"2016-08-24T09:11:57.129Z","subsystem":"cls","host":"cls-1","severity":7,"component":"ce","logger":"mos-mce","logsource":"event"}
2016/08/24 09:11:57 {"message":"hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh","@timestamp":"2016-08-24T09:11:57.129Z","subsystem":"cls","host":"cls-1","severity":7,"component":"ce","logger":"mos-mce","logsource":"event"}
The output log in flume does not print logs line by line as received by heka.
Could you please suggest if anything has to be changed to in flume source or memory channel
I am using heka to push the logs to flume. In heka I am able to see the logs getting tagged event by event(line by line) But in flume I am not getting line by line. It merges 2 or 3 lines of log together and prints it as a chunk of message. I am using source as multiport_syslogtcp and channel as memory channel, sink as file_roll.
Example log line: ggggggggggggg hhhhhhhhhhhhh
Example output in heka: 2016/08/24 09:11:57 {"message":"ggggggggggggggggggggggggggggggggggggggggggggggggg","@timestamp":"2016-08-24T09:11:57.129Z","subsystem":"cls","host":"cls-1","severity":7,"component":"ce","logger":"mos-mce","logsource":"event"} 2016/08/24 09:11:57 {"message":"hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh","@timestamp":"2016-08-24T09:11:57.129Z","subsystem":"cls","host":"cls-1","severity":7,"component":"ce","logger":"mos-mce","logsource":"event"}
Equivalent messages in file in flume: {\"channelName\":\"ahdenclinearwf\/53560_abr\",\"eventDetail\":\"RepID: 2000000_audio_frag_302 SIDX: {reference_ID: 302; PTS: 323565332463; first_offset: 72970420204; size: 821936; duration: 194400; timescale: 90000\",\"eventName\":\"SIDXDropped\",\"level\":\"MAJOR\",\"profileName\":\"1864k\",\"serviceType\":\"LIVE\",\"subType\":\"CIFConverter\",\"type\":\"RECORDER\"}}}, \"alarmop\": \"\" }","@timestamp":"2016-08-24T09:11:57.129Z","subsystem":"cls","host":"cls-1","severity":7,"component":"mce","logger":"mos-mce-211.208.250.30","logsource":"event"}{"message":"ggggggggggggggggggggggggggggggggggggggggggggggggg","@timestamp":"2016-08-24T09:17:39.379Z","subsystem":"cls","host":"cls-1","severity":7,"component":"mce","logger":"mos-mce-211.208.250.30","logsource":"event"}{"message":"hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh","@timestamp":"2016-08-24T09:17:39.379Z","subsystem":"cls","host":"cls-1","severity":7,"component":"mce","logger":"mos-mce-211.208.250.30","logsource":"event"}{"message":"llllllllllllllllllllllllllllllllllllllllllllllllll","@timestamp":"2016-08-24T09:17:39.379Z","subsystem":"cls","host":"cls-1","severity":7,"component":"mce","logger":"mos-mce-211.208.250.30","logsource":"event"}
The output log in flume does not print logs line by line as received by heka. Could you please suggest if anything has to be changed to in flume source or memory channel