search

TheTorProject / lepidopter

lepidopter: raspberry pi image for conducting OONI network measurements
https://ooni.torproject.org/
GNU General Public License v3.0
47 stars 20 forks source link

Emergency deletion of non submitted OONI reports #53

Open anadahz opened 8 years ago

anadahz commented 8 years ago

The image should have enough free disk space for new ooniprobe measurements and system files in order for the OS to operate properly.

Currently we don't enforce any emergency deletion process, if for any reason the disk space fills up (cannot communicate with backend) the OS will not be able to recover and Raspberry Pi will turn in a boot cycle or became a useless brick that doesn't perform/submits any ooniprobe measurements.

On the other hand we may lose important measurement reports that occurred in a specific event i.e a country's network completely blocking outgoing/incoming connections for a long time. If we don't automatically delete these reports a person can manually recover the reports (from the SD card) and extract useful information about this incident from the OONI measurements.

cc: @hellais @willscott @bassosimone @agrabeli

bassosimone commented 8 years ago

I'd check whether we're short in disk space and avoid running measurements in that case, so to avoid filling up all the disk. Does it make sense?

anadahz commented 8 years ago

@bassosimone preferably we should not stop performing measurements in lepidopter, since lepidopter is suppose to do OONI measurements...

bassosimone commented 8 years ago

@anadahz I agree, there is clearly a trade-off between filling the whole disk and running new measurements. What is more important in case the collector is unreachable and the disk is quickly filling up? Older or newer reports? Would it make sense, instead of pausing, to delete older reports?

anadahz commented 8 years ago

@bassosimone another idea will be to compress these reports and try to upload them (with on the fly extraction) at regular intervals, when the disk space fills up to a critical low (say 500M) we could delete there reports.

bassosimone commented 8 years ago

@anadahz yes, this could be a good way to solve the problem!

anadahz commented 8 years ago

Since the OONI reports deletion is going to be handled by the disk quota implementation in ooniprobe. I have made #75 that deletes log files and other lower priority files triggered by critical (98%) or warning (95%) disk usage percentage.

anadahz commented 7 years ago

I'm going to move this to future releases.