search

mozilla-services / heka

DEPRECATED: Data collection and processing made easy.
http://hekad.readthedocs.org/
Other
3.4k stars 531 forks source link

LogstreamerInput replays log data when heka was restarted. #1905

Open affinity226 opened 8 years ago

affinity226 commented 8 years ago

The LogstreamerInput plugin will replay log data on every restart. also, the seek point in journal file is just 0, it doesn't change.

elemoine commented 8 years ago

You need to provide a test case demonstrating the issue. We use LogstreamerInput plugins and rely on Heka's journaling functionality, and we've had no issues with it. So the problem is probably related to the specific configuration you use.

affinity226 commented 8 years ago

post toml configuration file :

[test_input] type = "LogstreamerInput" log_directory = "/home/jiwoncha/LOG" file_match = 'log(?P\d+).txt' decoder = "test_decoder" priority = ["idx"] rescan_interval = "10s"

[test_decoder] type = "PayloadRegexDecoder" match_regex = '^(?P\S+)' timestamp_layout = "2006/01/02 15:05:05"

[test_decoder.message_fields] Type = "TestLogFile" Payload = "Message"

[test_encoder] Type = "PayloadEncoder" Append_newlines = true

[RstEncoder]

[test_output] type = "FileOutput" message_matcher = "Logger == 'test_input'" path = "/home/jiwoncha/out.txt" encoder="test_encoder" perm = "666" flush_count = 100 flush_operator = "OR"