search

mozilla-services / heka

DEPRECATED: Data collection and processing made easy.
http://hekad.readthedocs.org/
Other
3.39k stars 530 forks source link

LogstreamerInput fails to read appended data #947

Closed trink closed 9 years ago

trink commented 10 years ago

The FxA basic metrics cron job appends to its log file once a day, these appends are not detected until a Heka restart. (as per @whd)

rafrombrc commented 10 years ago

Is this still happening? If so, is rsync being used to append to the log files? If yes, then the workaround might be to use rsync --inplace; for more details please see this thread: https://mail.mozilla.org/pipermail/heka/2014-August/000172.html

whd commented 10 years ago

This still happens occasionally but I haven't been keeping track of when or how often. The process on cron is Python opening the log file in append mode.

I'll make a concerted effort to track this down if it happens again. In theory it should be very simple to reproduce. I've also updated heka to 0.7.2 on the box. If it doesn't happen in the next week or so I'd say it's safe to close this issue.

whd commented 9 years ago

Confirmed still happening. I'll work on a simple test to reproduce this.

whd commented 9 years ago

It turns out the logstreamer was failing to read every other line from the file. However, I was never able to reproduce and it started working a few weeks ago (reason unknown), so I'd say it's safe to close this.

rafrombrc commented 9 years ago

Apparently my sacrifice of 3 chickens and a goat to the Logstreamer gods was sufficient.