At the moment, Tailviewer streams the entire contents of a log file into memory. This is fine as long as you're viewing small text log files or have enough main memory to waste. Due to storing the text log contents as strings, things are even more serious as every character takes up 2 bytes: A typical ASCII or UTF-8 log file will require at least twice it's size in memory when viewed (excluding indices).
In order to allow viewing huge log files (several Gb in size), Tailviewer needs to only stream those contents of the file into memory which are actually being viewed.
This MR lays the foundation for streaming text log files: Full streaming will probably be implemented over several MRs.
Detailed description
Introduce new StreamingTextLogFile implementation (will replace TextLogFile at a later date)
IoScheduler is responsible for providing read-only, out-of-order, random access to a text file
StreamingTextLogFile is responsible for building an index and delegating read calls to either its cache or the IoScheduler when not cached
The ILogFile.GetColumn, ILogFile.GetEntries and ILogEntries.CopyFrom overloads which took a LogFileSection parameter have been removed due to the amount of complexity introduced in the various implementations. There are only a few cases where having a contiguous source region is advantageous and in those cases, the implementation can perform a simple check if the source is of type LogFileSection.
Any expected problems concerning backwards compatibility of existing plugins?
Yes
This breaks every plugin which implements ITextLogFile (see below)
This breaks most plugins which use the ILogFile interface
Added Types
LogFilePropertiesView
Modified Types
Removed ILogFile.GetColumn overload which took a LogFileSection parameter
Removed ILogFile.GetEntries overload which took a LogFileSection parameter
Added parameter to ILogFile.GetColumn: LogFileQueryOptions queryOptions
Added parameter to ILogFile.GetEntries: LogFileQueryOptions queryOptions
Removed ILogFile.GetValues: Copying a subset of values is no standard use case
Added ILogFile.GetAllValues: Copies all values from the log file to the destination. Copying a subset can still be achieved by using a LogFilePropertiesView
Removed property ILogFile.Progress: the progress is now reported via LogFileProperties.PercentageProcessed
Removed property IlogFile.OriginalCount: This is now reported via TextLogFileProperties.LineCount
Any expected problems concerning backwards compatibility of existing user settings?
Defects
Fixes #277 Fixes #278 Fixes #279 Fixes #280 Fixes #281 Fixes #282
Reason for this change
At the moment, Tailviewer streams the entire contents of a log file into memory. This is fine as long as you're viewing small text log files or have enough main memory to waste. Due to storing the text log contents as strings, things are even more serious as every character takes up 2 bytes: A typical ASCII or UTF-8 log file will require at least twice it's size in memory when viewed (excluding indices).
In order to allow viewing huge log files (several Gb in size), Tailviewer needs to only stream those contents of the file into memory which are actually being viewed.
This MR lays the foundation for streaming text log files: Full streaming will probably be implemented over several MRs.
Detailed description
StreamingTextLogFileimplementation (will replaceTextLogFileat a later date)IoScheduleris responsible for providing read-only, out-of-order, random access to a text fileStreamingTextLogFileis responsible for building an index and delegating read calls to either its cache or theIoSchedulerwhen not cachedThe
ILogFile.GetColumn,ILogFile.GetEntriesandILogEntries.CopyFromoverloads which took aLogFileSectionparameter have been removed due to the amount of complexity introduced in the various implementations. There are only a few cases where having a contiguous source region is advantageous and in those cases, the implementation can perform a simple check if the source is of typeLogFileSection.Any expected problems concerning backwards compatibility of existing plugins?
Yes
Added Types
Modified Types
ILogFile.GetColumnoverload which took aLogFileSectionparameterILogFile.GetEntriesoverload which took aLogFileSectionparameterILogFile.GetColumn:LogFileQueryOptions queryOptionsILogFile.GetEntries:LogFileQueryOptions queryOptionsILogFile.GetValues: Copying a subset of values is no standard use caseILogFile.GetAllValues: Copies all values from the log file to the destination. Copying a subset can still be achieved by using aLogFilePropertiesViewILogFile.Progress: the progress is now reported viaLogFileProperties.PercentageProcessedIlogFile.OriginalCount: This is now reported viaTextLogFileProperties.LineCountAny expected problems concerning backwards compatibility of existing user settings?
No
Does this break existing user workflows?
No