syang0
commented
5 years ago Hello Balaji,
I think the answer depends on what you mean by a "truncated'' file. If you mean that the file has all bytes present and in order except for the very tail, then the decompressor should be able to interpret most of the log and then error out towards the very end.
If "truncated" means that bytes are missing in the middle (like say form a UDP packet drop), then you need to make sure that at least the NanoLog "dictionary" is sent in full to your remote location and that the log data present is aligned with the start a "BufferExtent". Once you have those two requirements, you should be able to hack the NanoLog Decoder accept the two separately and interpret the binary stream. Below is some information that may be helpful in your endeavor.
Good Luck.
Log File Structure and The Dictionary
How the log file looks and how you go about obtaining the dictionary are different depending on whether you’re working with Preprocessor NanoLog or C++17 NanoLog.
For Preprocessor NanoLog, the log file looks similar to Figure 4 in the NanoLog Paper. The log file contains a header, the dictionary for the entire application, and “Buffer Extents” which are groupings of log messages. The entire dictionary is built statically at compile-time and can be obtained by invoking the insertCheckpoint() function in runtime/Log.cc:118.
For C++17 NanoLog, the dictionary is built up piece-wise as the application runs. The format again resembles Figure 4 from the NanoLog Paper, except the “Dictionary” segment can appear in-between Buffer Extents, and each segment contains a fragment of the full dictionary. For your purposes, if you are missing dictionary entries, then you need to run logic similar to what’s found in runtime/RuntimeLogger.cc:393 to build it. And again, the dictionary is built up incrementally in C++17 NanoLog, so you will need to keep updating your remote dictionary as log messages are encountered during runtime. This is different from Preprocessor NanoLog which has the complete dictionary at application start.
Building Buffer Extents
To make sure log messages aren't cut randomly, I'd recommend you send only complete BufferExtents in UDP packets. The BufferExtents are built in this function. You can change the logic to build smaller extents that fit within the MTU of a UDP packet.
Of course, all this means you’ll also need to change the Decoder to accept the dictionary and BufferExtents separately, but it should be doable. Good Luck.
bkannadassan
Hi All,
Instead of logging locally I am planning to stream it over network. What I would like to know is, since this is a binary information if I pickup a truncated file will I loose all the information ?. If so can this be modified to extract the logs till which its written ?.
rgds Balaji Kamal Kannadassan