This PR fixes several issues regarding SingleLineJsonParser:
Currently, if there is a log record that is invalid JSON, the parser is not able to handle the parsing Exception and DirectorySource is not able to update the log file's position. As a result, the agent will keep reading the same record and failling over and over again. This PR catches the Exception, log the error message, and skip the invalid record.
Some customers write log records larger than NLog's buffer size (1024), and so a record line may not be written completely at once. (e.g. NLog writes {"key": first, then write the rest "value"} after a while. When that happens, Kinesis agent thinks that {"key": is the entire record and fails, because the SingleLineJsonParser uses StreamReader.ReadLine(), which returns a string when EndOfStream is reached EVEN IF there is no new line sequence. This PR adds a new class FileLineReader, which returns a line ONLY IF it ends with a new line sequence. This allows SingleLineJsonParser to always parse the complete JSON record line.
Testing
Added 2 new sets of tests: FileLineReaderTest to test the behavior of FileLineReader, and SingleLineJsonParserTest to test the behavior of SingleLineJsonParser using the new ReadLine functionality.
Description
This PR fixes several issues regarding
SingleLineJsonParser
:DirectorySource
is not able to update the log file's position. As a result, the agent will keep reading the same record and failling over and over again. This PR catches the Exception, log the error message, and skip the invalid record.{"key":
first, then write the rest"value"}
after a while. When that happens, Kinesis agent thinks that{"key":
is the entire record and fails, because theSingleLineJsonParser
usesStreamReader.ReadLine()
, which returns a string whenEndOfStream
is reached EVEN IF there is no new line sequence. This PR adds a new classFileLineReader
, which returns a line ONLY IF it ends with a new line sequence. This allowsSingleLineJsonParser
to always parse the complete JSON record line.Testing
Added 2 new sets of tests:
FileLineReaderTest
to test the behavior ofFileLineReader
, andSingleLineJsonParserTest
to test the behavior ofSingleLineJsonParser
using the newReadLine
functionality.