Project-CETI / whale-tag-embedded

Source code for the software that is used to build the image for the embedded computer inside the Whale Tags to be deployed onto sperm whales for project CETI
Apache License 2.0
13 stars 5 forks source link

Harden uSD storage against unexpected power loss #15

Closed MattCummings0517 closed 2 years ago

MattCummings0517 commented 2 years ago

Since the uSD card storage media, which contains the Raspberry Pi OS along with Tag audio and other recorded telemetry, is encapsulated in resin, a corrupt OS and/or filesystem may result in loss of data or rendering the Tag functions inaccessible.

It is reasonably foreseeable that the Tag may lose power or restart/reboot unexpectedly in many scenarios. The device is exposed to extremes of pressure and other environmental stresses and operates in an uncontrolled environment. The device is also powered from batteries that may fail at any time. It is very difficult to guarantee a complete and orderly shutdown via the main application in all circumstances.

It is desirable to ruggedize the system such that it can handle unexpected loss of power and application crashes without corrupting the basic platform so that the device can be restored subsequently.

PeterMalkin commented 2 years ago

The most important part is maintain the bootability of the device. Given that whale tags are potted in a bulk of epoxy and the only way to access the system is to ssh in - it is important that the devices can boot at all times, even if it was rebooted unexpectedly.

That means we need to make sure that the boot partition and the root partition are never corrupted. To do this, it is good to make them read only.

I followed this article: https://medium.com/swlh/make-your-raspberry-pi-file-system-read-only-raspbian-buster-c558694de79 And implemented the changes here: https://github.com/Project-CETI/whale-tag-embedded/blob/main/build/setup_image.sh#L86

However, there are three more things to do to make it really work: 1) Make sure the root system read only. There is a convenience script for this: https://github.com/Project-CETI/whale-tag-embedded/blob/main/overlay/etc/bash.bashrc However, the proper way is to mark the filesystems to be mounted ro at boot. The article gives examples of that.

2) Today ceti-tag-data-capture logs everything to syslog. That means that logging folders should be mounted as a separate filesystem. In the article they use tmpfs for it, which resides in memory. THis is not going to work for us. So we need to create a different partition at build time and format it, mount it writeable and store all logs there.

3) Finally, we need to create a separate partition for data. It is already assumed to have label "cetiData" and be mounted at /data by default through /etc/fstab. See here: https://github.com/Project-CETI/whale-tag-embedded/blob/main/build/setup_image.sh#L77

PeterMalkin commented 2 years ago

We should also utilize a better filesystem for the data partition that would be more ameanable to recovery than generic ext2. Can probably consider btrfs

PeterMalkin commented 2 years ago

I think I implemented everything there was to do here: https://github.com/Project-CETI/whale-tag-embedded/compare/main...feature/v2_1/data_partition

Please take a look, and if this seems reasonable, I can issue a PR into "main"