HisFileWriter: waiting_writes is now emptied before getting events from the main thread instead of writing a limited amount of events at a time.
This gets rid of the large wait time and the memory bloat that was associated with the write queue. Before, it was not uncommon to see 5GB of memory in use as reported by htop.
Memory usage now tops out at around 600MB, but has some periodic hitching when the main thread cannot get the lock for the write queue. This could probably be cured through the use of a spin lock or atomics, but in favor of being able to support systems that don't have C++14 or newer available, I think it's unwise. In the case that someone less familiar with concurrent computing ever has to touch this code, it'd also be less obvious why things are being done that way.
Overall, finalize() times for the threaded HisWriter are now on the order of 10's of microseconds, down from 1-5 minutes depending on the speed of the storage, so I consider this a win.
utkscan CmakeLists: Add option for not using the threaded his file writer. Off by default.
OutputHisFile: Add compile time condition to not use the threaded version of the his file output.
This will allow utkscan to function in an environment that is very memory constrained, such as 55kb of memory in use on Ubuntu 20 LTS.
Runtimes with this are on the order of 750 seconds on my laptop, so faster but results may vary depending on the system.
Implements the following changes
HisFileWriter: waiting_writes is now emptied before getting events from the main thread instead of writing a limited amount of events at a time.
utkscan CmakeLists: Add option for not using the threaded his file writer. Off by default.
OutputHisFile: Add compile time condition to not use the threaded version of the his file output.