Considering that the Performance option of raspi-config allows a minimal configuration of the Overlay File System, this is a proof of concept for making the mount points of the OverlayFS visible via configuration option, based on the idea in this thread. A new step in the wizard is added for this; the default is to make the mount points invisible, as per the current setting; when they are visible, their root directory is /run, like the following (referring to a Raspberry Pi Zero W):
This configuration is still safe because the physical mount point (lower) is in read-only mode.
The /run/upper mount point allows inspection of the changes, for instance, to monitor their growth (via du or standard commands). Any changes here are not persistent, as the /run/upper filesystem is a tempfs in RAM. The /run/lower mount point is a read-only filesystem, so any possible access is safe for the SD card.
The new step in the wizard is the following:
┌──────────────────────────────────────────────────────────┐
│ │
│ Would you like the mount points of the overlay file │
│ system to be visible under '/run/upper' (rw) and │
│ '/run/lower' (ro)? │
│ │
│ │
...
│ │
│ <Yes> <No> │
│ │
└──────────────────────────────────────────────────────────┘
If No is selected (default), the output (after the reboot) is the standard overlay mode, with no visible mount points:
The following command will monitor the growth of the upper filesystem (e.g., files changed after a reboot) by listing the first 10 largest directories:
cd /run/upper/data && sudo du -h | sort -hr | head
This PR also includes some improvements to the batch (command line) mode:
"status" can be 0, to enable the overlayfs, or 1 to disable it (1 is the default);
"writable" can be 0 for setting a read-only boot partition, or 1 if the boot partition has to be writable (0 is the default);
"invisible" is used when enabling the overlays; it can be 0 for visible mounts or, 1 for invisible mounts (1 is the default).
If status is missing, a usage summary is printed.
(Notice that best practices suggest that the whole SD card is in read-only mode, including the boot partition, as physically consumer SD cards have no partition concept (and no fault tolerance), so any write inconsistency (also to the boot partition) can drive an SD corruption if not appropriately completed.)
Even if all preliminary tests of this patch appear to be fine, long-term stability needs monitoring for some additional time.
Usage notes
Formally, OverlayFS does not support merging changes from an upper filesystem to a lower filesystem and changes to the lower filesystem are not allowed. This premised, empirically I succeeded in mounting the lower filesystem in read-write via the following command:
sudo mount -o remount,rw /run/lower
Then I could make permanent changes to the SD card by copying (cp -p) files from the upper part to the lover part; if a file is changed in the lower part, it persists to a reboot. After making changes, I could return back to read-only mode with the following command:
sudo mount -o remount,ro /run/lower
Also, I could change the /run/upper filesystem, provided that after the changes the overlayfs is remounted through
sudo mount -o remount -t overlay -olowerdir=/run/lower,upperdir=/run/upper/data,workdir=/run/upper/work overlay /
Interesting to note that an adequate application of changes to the /run/upper filesystem followed by the remount of the OverlayFS allows reverting changes.
Example:
# Creating a new file under /home/pi (login pi):
cd
cat foo # failure
ls -l /run/upper/data/home/pi/foo # failure
ls -l /run/lower/home/pi/foo # failure
echo test > foo
cat foo # success
ls -l /run/upper/data/home/pi/foo # success
ls -l /run/lower/home/pi/foo # failure
mount | grep "lower type" # read-only
# Making the new file persistent:
sudo mount -o remount,rw /run/lower
cp -p foo /run/lower/home/pi
sudo mount -o remount,ro /run/lower
rm /run/upper/data/home/pi/foo
sudo mount -o remount -t overlay -olowerdir=/run/lower,upperdir=/run/upper/data,workdir=/run/upper/work overlay /
cat foo # success
ls -l /run/upper/data/home/pi/foo # failure
ls -l /run/lower/home/pi/foo # success
# Updating the new file:
echo hello >> foo
cat foo # success
ls -l /run/upper/data/home/pi/foo # success
ls -l /run/lower/home/pi/foo # success
# Reverting updates:
rm /run/upper/data/home/pi/foo
sudo mount -o remount -t overlay -olowerdir=/run/lower,upperdir=/run/upper/data,workdir=/run/upper/work overlay /
cat foo # previous data
ls -l /run/upper/data/home/pi/foo # failure
ls -l /run/lower/home/pi/foo # success
# Removing the file and making this persistent:
rm foo
ls -l /run/upper/data/home/pi/foo # whiteout existing, as a character device with 0/0 device number
sudo mount -o remount,rw /run/lower
rm /run/lower/home/pi/foo
sudo mount -o remount,ro /run/lower
rm -f /run/upper/data/home/pi/foo # remove the whiteout
sudo mount -o remount -t overlay -olowerdir=/run/lower,upperdir=/run/upper/data,workdir=/run/upper/work overlay /
cat foo # failure
ls -l /run/upper/data/home/pi/foo # failure
ls -l /run/lower/home/pi/foo # failure
Notice that the above methods have been empirically tested; they appear to work, even if the official documentation does not report them.
Maintenance notes
By periodically monitoring /run/upper with df -h|sort -k5 -h, cd /run/upper/data&&sudo du -h|sort -hr|head - and sudo tree -dh /run/upper I noticed that the growth of two resources have to be controlled: packagekit and journald: when the OverlayFS is active, the first one need to be stopped and disabled, while the second one might benefit from a configuration to limit its size.
The following script can be added to crontab@reboot:
if ! test -d /run/upper; then exit 0; fi # Only run if the overlayfs is active
sudo systemctl stop packagekit.service
sudo systemctl mask packagekit
sudo systemctl stop apt-daily.service
sudo systemctl stop apt-daily.timer
sudo systemctl stop apt-daily-upgrade.timer
sudo systemctl stop apt-daily-upgrade.service
sudo systemctl mask apt-daily.service
sudo systemctl mask apt-daily.timer
sudo systemctl mask apt-daily-upgrade.timer
sudo systemctl mask apt-daily-upgrade.service
sudo apt-get clean
if ! grep -q '^SystemMaxUse=' /etc/systemd/journald.conf; then
echo "SystemMaxUse=50M" | sudo tee -a /etc/systemd/journald.conf >/dev/null
echo "SystemKeepFree=50M" | sudo tee -a /etc/systemd/journald.conf >/dev/null
sudo service systemd-journald restart
fi
As per the journald service, in the hypothesis that only user-1000 (pi) and system user ids are used, the script should result in making journald produce up to 10 files under /var/log/journal/ (including rotated files), where each one has size of 6,3 MB, with a total consumption of max 63 MB.
More generally, it is necessary to avoid the memory full condition of the OverlayFS by periodically monitoring the “/run/upper” usage size. Typical elements to check are:
journald (size of /var/log/journal directory)
size of /tmp directory
size of the user directories
mailing system
apt and /var/lib/dpkg
packagekit
On a Raspberry Pi Zero W, the max size of /run/upper will be about 215 MB.
Notice that when the usage size reaches 100%, a crash might occur; on a Raspberry Pi Zero W, this can be detected by monitoring the ACT led, intercepting when it repeatedly blinks with the following pattern: about 200 short blinks within 0.5 secs + 0.5 sec blink (constantly repeated).
To avoid the OverlayFS saturation condition, a periodic reboot can be issued when the OverlayFS usage exceeds 90% via the following script, which can be added to crontab, to be run every day (e.g., 0 2 * * * /home/pi/OverlayFSMonitor):
USAGE_PERCENT=90
size=`df| awk '/\/run\/upper/{ gsub("%",""); print $5 }'`
test "$size" || size=0
if [ "$size" -gt "$USAGE_PERCENT" ] ; then
echo "OverlayFS bigger than $size%: performing a reboot." | systemd-cat -t OverlayFSMonitor
df -h | systemd-cat -t OverlayFSMonitor
sudo shutdown -r now
fi
When the growth of the previously mentioned resources is correctly maintained, the above memory full condition in the script (producing the reboot) should occur very rarely, for example after many months of uninterrupted operation.
Considering that the Performance option of
raspi-configallows a minimal configuration of the Overlay File System, this is a proof of concept for making the mount points of the OverlayFS visible via configuration option, based on the idea in this thread. A new step in the wizard is added for this; the default is to make the mount points invisible, as per the current setting; when they are visible, their root directory is /run, like the following (referring to a Raspberry Pi Zero W):This configuration is still safe because the physical mount point (lower) is in read-only mode. The
/run/uppermount point allows inspection of the changes, for instance, to monitor their growth (viaduor standard commands). Any changes here are not persistent, as the/run/upperfilesystem is a tempfs in RAM. The/run/lowermount point is a read-only filesystem, so any possible access is safe for the SD card.The new step in the wizard is the following:
If No is selected (default), the output (after the reboot) is the standard overlay mode, with no visible mount points:
The following command will monitor the growth of the upper filesystem (e.g., files changed after a reboot) by listing the first 10 largest directories:
This PR also includes some improvements to the batch (command line) mode:
If status is missing, a usage summary is printed.
(Notice that best practices suggest that the whole SD card is in read-only mode, including the boot partition, as physically consumer SD cards have no partition concept (and no fault tolerance), so any write inconsistency (also to the boot partition) can drive an SD corruption if not appropriately completed.)
Even if all preliminary tests of this patch appear to be fine, long-term stability needs monitoring for some additional time.
Usage notes
Formally, OverlayFS does not support merging changes from an upper filesystem to a lower filesystem and changes to the lower filesystem are not allowed. This premised, empirically I succeeded in mounting the lower filesystem in read-write via the following command:
Then I could make permanent changes to the SD card by copying (
cp -p) files from the upper part to the lover part; if a file is changed in the lower part, it persists to a reboot. After making changes, I could return back to read-only mode with the following command:Also, I could change the
/run/upperfilesystem, provided that after the changes the overlayfs is remounted throughInteresting to note that an adequate application of changes to the
/run/upperfilesystem followed by the remount of the OverlayFS allows reverting changes.Example:
Notice that the above methods have been empirically tested; they appear to work, even if the official documentation does not report them.
Maintenance notes
By periodically monitoring /run/upper with
df -h|sort -k5 -h,cd /run/upper/data&&sudo du -h|sort -hr|head -andsudo tree -dh /run/upperI noticed that the growth of two resources have to be controlled: packagekit and journald: when the OverlayFS is active, the first one need to be stopped and disabled, while the second one might benefit from a configuration to limit its size.The following script can be added to crontab
@reboot:As per the journald service, in the hypothesis that only user-1000 (pi) and system user ids are used, the script should result in making journald produce up to 10 files under /var/log/journal/ (including rotated files), where each one has size of 6,3 MB, with a total consumption of max 63 MB.
More generally, it is necessary to avoid the memory full condition of the OverlayFS by periodically monitoring the “/run/upper” usage size. Typical elements to check are:
On a Raspberry Pi Zero W, the max size of /run/upper will be about 215 MB.
Notice that when the usage size reaches 100%, a crash might occur; on a Raspberry Pi Zero W, this can be detected by monitoring the ACT led, intercepting when it repeatedly blinks with the following pattern: about 200 short blinks within 0.5 secs + 0.5 sec blink (constantly repeated).
To avoid the OverlayFS saturation condition, a periodic reboot can be issued when the OverlayFS usage exceeds 90% via the following script, which can be added to crontab, to be run every day (e.g.,
0 2 * * * /home/pi/OverlayFSMonitor):When the growth of the previously mentioned resources is correctly maintained, the above memory full condition in the script (producing the reboot) should occur very rarely, for example after many months of uninterrupted operation.