search

robotika / osgar

Open Source Garden (Autonomous) Robot
MIT License
13 stars 12 forks source link

Zipped sparse data #125

Open m3d opened 5 years ago

m3d commented 5 years ago

There is a need to compress some external raw data like ROS sensor_msgs/PointCloud2 message (see https://robotika.cz/competitions/subtchallenge/cs#190105 for motivation). This should be probably on the very low level of serialize/deserialize. Any suggestion? Should we provide option line "raw|zip" or "raw.zip" on definition of node output, where "raw" would remain not compressed? p.s. later I realized that I had to deal with the same problem a year ago when logging stereo data from Naio Oz robot (https://github.com/robotika/naio/blob/master/myr2017.py#L266).

zwn commented 5 years ago

Can we just wrap the whole log into a gz stream? Is there any particular advantage of working per stream which would outweigh the complications with regards to its configuration and such?

zwn commented 5 years ago

Just a note: Jirka solved similar problem by mounting a file as a zfs filesystem which supports lz4 compression. The file is sparse so at the beginning takes no space and as the logs are saved there, everything is lz4 compressed.

I still thing that if the need arises, wrapping the whole stream with lz4 would be my first test (before changing the format of the logs).

m3d commented 5 years ago

The need is still valid in SubT context - it is otherwise "unreasonable" to log X3 (drone) RGBD data and the same holds for PrimeSense. The zipped stream has problem with regular flush while internal zip each large packet does not.