MiguelAlho
commented
7 years ago I believe I may have found the problem that causes the above behaviour - The repetition of events is related to constantly restarting the read of the buffer file. The following seems to occur:
- The bookmark file is read, and start position and file name are read.
- Events are read and the book mark eventually advances.
- On the next iteration, the bookmark e reread, with the stream positioned at the beginning, but for some reason, the first line is not read - instead a second, empty line may have been read OR a possible BOM may influence the read, OR a StreamReader disposal problem.
- Because the bookmark file has been considered empty, a new bufferfile is considered and choosen. Typically, in this scenario, it will be the same file previously read, but the buffer is repositioned at the start of the file.
- The batch of events are re-read from the buffer file.
It's somewhat hard to replicate, but is possible using the sample file and some buffer / book mark files from a production instance. On top of this, I may have fixed a "second message dropped" problem related to the BOM in the buffer file,
CDargis
One behaviour that has popped up infrequently is related to the durable Logger implementation sending the same message frequently / repeatedly to Loggly. This eventually creates a massive accumulation of messages on Loggly and many times quickly maxes out any cap in place.
We've detected the problem through the following symptoms:
We suspect that it is the durable logger implementation we use that some how cannot update the buffer (read messages) correctly and in failing to move forward, stays and repeats the same message(s).
Looking at the buffer.bookmark file, there is a record for a byte pointer to the first buffer.json file (oldest) and many invalid-Error files in the folder.
I'll be looking in to this to fix it since it is causing problems for us, but if anyone has seen this before, any input is appreciated.