search

mawson-rovers / hestia

Hestia is a space-ready heater & temperature measurement system developed by Mawson Rovers in Sydney. Its first mission will demonstrate a novel UTS heatsink design, aboard the Waratah Seed cubesat mission. It is due to launch on a SpaceX rocket in July 2024.
MIT License
1 stars 1 forks source link

Compress data log files for downlinking #73

Closed cowen-mawson closed 7 months ago

cowen-mawson commented 10 months ago

At the moment, Hestia log files are written to disk as uncompressed csv files. As bandwidth for downlinking from the satellite will be at a premium (and the downlink protocol not having built in compression support) we should choose a compression format for the files which can be downlinked from the satellite efficiently.

mryall-mawson commented 8 months ago

Based on advice from CUAVA, we need to switch to gzip encoding, so I've made the change and opened a PR here: https://github.com/mawson-rovers/hestia/pull/79

But while testing this change, I realised that the compression for the log files isn't very effective in its current implementation because it is only compressing 2 lines at a time, because we open and close the file for every write.

For a 120-line log file with 19 KB of (dummy) data, the current streaming implementation resulted in a compressed output of 13 KB. Compressing the whole file with gzip -1 results in a file size of 1 KB.

So I think we should change the way it works to compress log files in bulk. Probably easiest to do it in uts-log when it starts up, and whenever a day ticks over. We can just go through all the files in the UTS_LOG_PATH directory and gzip them up.

I'll take a look at this tomorrow, and put all the changes on that PR against this ticket.

mryall-mawson commented 7 months ago

Tested manually and works on WS-1.

Added to the flight schedule to be run before downlink as uts-cli zip.