Open m3d opened 5 years ago
Can we just wrap the whole log into a gz stream? Is there any particular advantage of working per stream which would outweigh the complications with regards to its configuration and such?
Just a note: Jirka solved similar problem by mounting a file as a zfs filesystem which supports lz4 compression. The file is sparse so at the beginning takes no space and as the logs are saved there, everything is lz4 compressed.
I still thing that if the need arises, wrapping the whole stream with lz4 would be my first test (before changing the format of the logs).
The need is still valid in SubT context - it is otherwise "unreasonable" to log X3 (drone) RGBD data and the same holds for PrimeSense. The zipped stream has problem with regular flush while internal zip each large packet does not.
There is a need to compress some external raw data like ROS
sensor_msgs/PointCloud2
message (see https://robotika.cz/competitions/subtchallenge/cs#190105 for motivation). This should be probably on the very low level ofserialize/deserialize
. Any suggestion? Should we provide option line "raw|zip" or "raw.zip" on definition of node output, where "raw" would remain not compressed? p.s. later I realized that I had to deal with the same problem a year ago when logging stereo data from Naio Oz robot (https://github.com/robotika/naio/blob/master/myr2017.py#L266).