Open jwbonner opened 2 months ago
PeterJohnson
commented
2 months ago Some options for compression could be zstd or LZ4. zstd seems to work better on 64-bit, and LZ4 seems to be lower overhead, although zlib with Z_BEST_SPEED seems to be very fast? Need to do some benchmarks on the Rio.
Would it be better to use a different file extension than .wpilog for compressed files?
jwbonner
commented
2 months ago I would tend towards keeping the wpilog extension, which would align with how MCAP handles compression (the algorithm used to encode each chunk is part of the header). The main benefit I can think of for using a different extension is that it increases visibility for users who might want to turn it off. But assuming that the performance tradeoff is reasonable, I think most users shouldn't need to worry about whether compression is enabled. I could see adding some text to AdvantageScope displaying whether or not the log is compressed though, so users who are interested could check it there.
Many teams are using the WPILOG format to store hundreds or thousands of fields, and often without removing duplicate values. This can result in enormous log files that are difficult to download and can quickly fill the RIO's storage. Adding an option to compress data before writing to disk would alleviate this issue, and could remain optional by adding a compression flag to the WPILOG header.
Of course, choosing an algorithm and configuration that can perform well on the RIO is an important consideration. However, we already have evidence that this is reasonable given CTRE's (apparent) use of compression for the Hoot log format without issue.
I ran a couple of tests using zlib (with the same configuration as gzip) to determine how effective compression might be on a typical WPILOG. This is based on a ~4.5 minute log file of a real match with 533 fields, including a variety of data types (arrays, structs, simple values, etc).