Request: read-only SD card

[tomfanning]

I'm looking to get the SD card read-only after I'm done commissioning an install, so as to increase SD card lifetime, make it more power-failure tolernt etc.

Ideal situation is that I'd have a one-line command to make things temporarily writeable (install/update packages, tweak config etc) and another command (or a reboot) to go back to read-only.

Simply, ~/remount-ro and ~/remount-rw scripts to handle everything, with the system being set to read-only at boot.

I've given this a go but I'm getting weird issues like installed packages disappearing after reboot so I think there's some kind of incompatibility there, but I'm not knowledgeable enough to diagnose and fix this myself.

[Mausy5043]

There's a problem with /var/spool which needs to be rw and able to survive reboots. For example: it stores the user's cron jobs (from crontab -e). :thought_balloon: Probably other parts of the filesystem have similar problems when they are/become read-only?

Have you tried using the readonly-switch on the side of the SD-card? Unfortunately, on microSD's (for B+) there is no switch available. :-1:

Can you give an example of a package that "disappears"?

[diederikdehaas]

A fully read-only system is indeed not possible (afaik), but what tomfanning is asking for (iiuc) is for a (Debian) Live type system, with the addition of (a) script(s) for persistence.

[JedMeister]

My 2c... TBH I'm not sure whether this is something that should be included in raspbian-ua-netinst by default. It's a potentially useful thing, but I'm not sure how many people would actually want it. Also IMO @diederikdehaas you can''t include every cool idea or otherwise raspbian-ua-netinst will end up bloated. Having said that, if a working process could be documented (and how you could include it if that's what you wanted) that would be pretty awesome.

FWIW from my research, wearing out SD cards from normal usage is a non-issue. Yes they will wear out eventually, but with wear-levelling and 10k+ write cycles (some have reported 40k+ and still going strong on quality card) the effective lifespan is probably similar to any modern electronics (i.e. the lifespan of the card is probably equivalent or greater than it's usefulness). Also the more free space the card has, the longer it will last.

Despite that, this idea would still be useful to protect against fs corruption from power failure. IMO if your power is unreliable then a hardware solution is probably better (e.g. build a mini UPS for your Pi). Then it won't go off or if it does it will shutdown properly.

@tomfanning - As I said above, IMO SD cards wearing out is a non-issue. Regardless, what you want should be possible but it seems to me that your fs is staying read-only (e.g. installed packages disappearing after reboot). If you look in the comments of the tutorial you link to, someone says that the instructions are insufficient for making the fs writeable and suggests another way... Perhaps you should explore that... Also perhaps rather than using a RAM disk for the UnionFS overlay you could use a USB stick instead. Then changes to the fs will always be persistent.

@Mausy5043 - The instructions linked to use UnionFS (IMO a very cool copy-on-write (cow) fs) with a RAM disk so there should be no issue with any fs locations. The system doesn't realise that it is using a read-only fs (it can write to the RAM disk fs - which is overlaid over the real fs). Obviously there are times (new cron jobs, adding/removing/updating packages, adjusting config, etc) when you would want to really write to the fs (hence the need to occasionally mount the real fs read/write).

@diederikdehaas - No a read only fs should be possible (using UnionFS) although a live system with some persistence would be another way of doing it...

[diederikdehaas]

No a read only fs should be possible (using UnionFS) although a live system with some persistence would be another way of doing it...

I think we're saying/meaning the same thing. The base Debian Live system is read-only, but when you start it up, the system will use a RAM disk and 'union' mechanism to enable writing files within the running system. Since it's a RAM disk holding the changes, they would be lost when you shutdown/reboot ... unless you use persistence with that.

You may very well be right about (not) implementing this in the installer, although it probably would be a useful program for a raspbian(/debian) system in general. But I do think that SD cards in general are an issue with the RPi, as I experience myself (with Samsung EVO cards, which I think/thought are rather good). (OTOH implementing this feature likely wouldn't help with that though)

[Mausy5043]

Have to agree with @JedMeister about the "feature-creep" warning.

[goranche]

A fully read-only system is indeed not possible (afaik)

depends on your definition of "fully read-only" :smile: I've done a project for a client where the debian image (official raspbian soft-float based, but should work with "anything") boots from a read-only partition, with some dirs mounted on a RAM disk. in addition to this a non-critical read-write partition is layered over the root file system, which allows for normal read-write work, apt-get upgrades, ... ... ... the main thing is that the system boots from a initramfs, which upon boot checks the consistency of the read-write partition and if any problem is found, just reinitializes ("formats") it. this means that the system will boot no matter what (well, within the realm of reason, of course)

this was needed because their product was installed in environments where power was randomly cut, so shutdowns were pretty much out of the question.

I've already thought of how to incorporate this into the installer, but it's not an easy task (actually, I doubt it's doable in a fully automated fashion, maybe with a different installer, not sure)... (also, I'm not sure if the client would allow the solution to be published, I'll have to check on that)

I'm not sure whether this is something that should be included in raspbian-ua-netinst by default

I agree with @JedMeister here

[diederikdehaas]

In @tomfanning's 'defence', I asked him to create a GitHub issue for this.

[goranche]

I don't really think any defence is needed, it's an interesting topic (although not an easy one)... also... we have a topic on the raspperypi forum? :grin:

[diederikdehaas]

I think we can hereby conclude that I was right (on the forum) LOL

[goranche]

yeah... I guess... :angel: :wink:

[tomfanning]

Hi all, thanks for the discussion, I'm sorry I've not been able to more actively contribute.

@JedMeister Without fail every single Pi I've put into an unattended role has had the SD card fail after approximately 6 months. Variety of Pi models, SD card vendors, software, environment, role. The sample size is about 5 units.

The solution I linked to applies cleanly to vanilla Raspbian, leaving me with the following:

root@pi:~# mount
/dev/root on / type ext4 (ro,noatime,data=ordered)
devtmpfs on /dev type devtmpfs (rw,relatime,size=219744k,nr_inodes=54936,mode=755)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=44784k,mode=755)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /run/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=89560k)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
/dev/mmcblk0p1 on /boot type vfat (ro,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro)
ramdisk on /etc_rw type tmpfs (rw,relatime)
unionfs-fuse on /etc type fuse.unionfs-fuse (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
ramdisk on /var_rw type tmpfs (rw,relatime)
unionfs-fuse on /var type fuse.unionfs-fuse (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other)
none on /tmp type tmpfs (rw,relatime)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)

and after a mount -o remount,rw /:

root@pi:~# mount
/dev/root on / type ext4 (rw,noatime,data=ordered)
devtmpfs on /dev type devtmpfs (rw,relatime,size=219744k,nr_inodes=54936,mode=755)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=44784k,mode=755)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /run/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=89560k)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
/dev/mmcblk0p1 on /boot type vfat (ro,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro)
ramdisk on /etc_rw type tmpfs (rw,relatime)
unionfs-fuse on /etc type fuse.unionfs-fuse (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
ramdisk on /var_rw type tmpfs (rw,relatime)
unionfs-fuse on /var type fuse.unionfs-fuse (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other)
none on /tmp type tmpfs (rw,relatime)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)

So this meets my needs and has been reliable so far. But it completely sucks to have to clean up a straight Raspbian system (remove the GUI, all the bundled stuff like Mathematica etc) before it's suitable for use in my typical environment.

So my requirements are:

• headless install - the netinst mechanism in raspbian-ua-netinst is perfect, since the result is a perfectly up to date system with no need to upgrade packages after OS install
• fully read-only root, which we can see from above is perfectly possible, to prevent SD card write issue and copy with random power-offs
• minimal packages installed

raspbian-ua-netinst seems like an ideal starting point for my needs

Hope this helps explain where I'm coming from - and thanks for your consideration.

[Mausy5043]

I have different requirements and I don't want to expect this repository to comply with every one of them. (no offense intended). But I think it's a great starting-point. So, what I did was git clone the v1.0.x branch and modify that to my own needs. Making the appropriate changes in installer-config.txt for making the necessary changes in root-filesystem, packages to install, cmdline, hostname and stuff and using post-install.txt to bend the installation to my needs, rolling-back or "correcting" things where needed. Being careful not to tamper with the core of the installer, so I can easily stay in sync with the debian-pi master.

[ralin3]

I do this all the time, I have readonly_fs=yes in in my installer-config.txt. And then I have this in post-install.txt:

### Readonly-FS
if [[ $readonly_fs == "yes" ]] ; then
        echo 'echo "/dev/mmcblk0p1 /boot vfat ro 0 0" > /etc/fstab' >> /rootfs/etc/rc.local
        echo 'echo "/dev/mmcblk0p2 / ext4 ro 0 0" >> /etc/fstab' >> /rootfs/etc/rc.local
        echo 'echo "tmpfs /var/log tmpfs defaults,noatime,nosuid,nodev,noexec 0 0" >> /etc/fstab' >> /rootfs/etc/rc.local
        echo 'echo "tmpfs /tmp tmpfs defaults,noatime,nosuid,nodev 0 0" >> /etc/fstab' >> /rootfs/etc/rc.local
        echo 'echo "tmpfs /var/lib/dhcp tmpfs defaults,noatime,nosuid,nodev,noexec 0 0" >> /etc/fstab' >> /rootfs/etc/rc.local
        echo 'echo "tmpfs /var/tmp tmpfs defaults,noatime,nosuid,nodev,noexec 0 0" >> /etc/fstab2' >> /rootfs/etc/rc.local
        ###### dailyreboot.cron
        echo '#!/bin/bash' > /rootfs/root/dailyreboot.cron
        echo 'mount -o remount,rw,noatime,data=writeback,nobarrier,noinit_itable /dev/root /' >> /rootfs/root/dailyreboot.cron
        echo 'mount -o remount,rw /dev/mmcblk0p1 /boot' >> /rootfs/root/dailyreboot.cron
        echo '/usr/bin/apt-get update' >> /rootfs/root/dailyreboot.cron
        echo '/usr/bin/apt-get upgrade -y -d' >> /rootfs/root/dailyreboot.cron
        echo '/sbin/reboot' >> /rootfs/root/dailyreboot.cron
        /bin/chmod +x /rootfs/root/dailyreboot.cron
        echo '07 03 * * * root /root/dailyreboot.cron' >> /rootfs/etc/crontab

        ###### remount.sh
        echo '#!/bin/bash' > /rootfs/root/remount.sh
        echo 'mount -o remount,rw,noatime,data=writeback,nobarrier,noinit_itable /dev/root /' >> /rootfs/root/remount.sh
        echo 'mount -o remount,rw /dev/mmcblk0p1 /boot' >> /rootfs/root/remount.sh
        /bin/chmod +x /rootfs/root/remount.sh
fi

This remounts every night, updates and then reboots (I've had issues with only remounting) I'm running this on several PI's and it works great with those applications. Obviously other applications might need more mountpoints in tmpfs, ymmv.

[Mausy5043]

@tomfanning : I've found that since I started mounting /var/log and /tmp in tmpfs the life expectancy of the SD-card has gone up.

[goranche]

/tmp should be tmpfs anyway... there are a few additional things that make sense on a readonly system... I'm almost ready to release my scripts... just need a little more time :blush:

but yes, the SD card will survive longer, and what's more important, the system will survive a cold powerdown

[niun]

I managed to get a read-only root working with the kernel's overlayfs. This is done by this initramfs script. It should work without additional packages. Install instructions are in the script. I will make some more tests with additional writable data partitions and I will try to install this via post-install.txt. I don't know yet, what the most elegant way is to update a system configured like that. It would require some root juggling or two reboots.

[nagualcode]

Is this ready? How do I make the root fs Read Only after the install is completed?

I am using some solutions based on tutorials, but I would prefer some automated solution coming from the installer.

[diederikdehaas]

No, this functionality is not available in this installer.

[bbinet]

+1, read-only support would be super useful!

@goranche in your last comment, you said that you were almost ready to release your scripts: are they actually available somewhere?

[goranche]

yeah, I was, but then systemd happened... :unamused: and now they don't work anymore... :-1:

I'll create a repo with a the post_install.txt script I used for a video player device... it does work with the latest installer, if you're willing to overlook a bunch of error messages... :innocent:

[bbinet]

Ok, thanks. I'd rather fix the reported errors ;)

[goranche]

so would I, but don't really have the time... :unamused:

still, you can take a look here

[goranche]

you'll need to use your own username in the script, of course, and create a salted hash for the password, that is set on the username in the installed system... should be easy to google, if you have problems with that, let me know

[bbinet]

Thanks, I will have a look and report back the results.

On 15 January 2016 at 14:52, Goran Blažič notifications@github.com wrote:

you'll need to use your own username in the script, of course, and create a salted hash for the password, that is set on the username in the installed system... should be easy to google, if you have problems with that, let me know

— Reply to this email directly or view it on GitHub https://github.com/debian-pi/raspbian-ua-netinst/issues/174#issuecomment-171966370 .

[nagualcode]

Yeah :) Read only file system with last raspbian/systemd is a must. Many of the Raspberry projects like robots and such deal with power-failures. I am still learning the basics, but will try to catch with you guys and help with all I can.

[goranche]

deal with power-failures

don't get me started... I worked on a project where many RPis were deployed into street lighting fixtures, and got their power cut on a daily basis... I was invited to the project after they lost several SD cards (and of course time to send people out to change them)...

but now there's systemd... :unamused: will try to find some time, though, and have a go at it, just can't promise anything

[JedMeister]

A little off topic, but FWIW if SystemD is an issue you can always revert back to SySVIinit instead... I have had some issues with nested SystemD (in LXC containers) and found that reverting to SysVInit (inside the container; host still uses SystemD) works a treat... Just purge all SystemD packages (apt-get purge systemd*) and install sysvinit-core and systemd-shim...

[goranche]

yeah, good luck with truly purging systemd from your installation... this is the problem (as I see it) with systemd... sure, you can add sysvinit as the init system, and the logging system, and ..., but you'll still have a huge dependency on systemd... (I mean, the fact you need systemd-shim says it all)

but this isn't the place for this debate ;)

[goranche]

I'm happy to report it seems I've "cracked" it... I have a standard install (from the last released image) that's running with a fully (both /boot and /) ro SD card, while having rw access to /etc and /var with the writes going to a mounted tmpfs (mounted using unionfs)...

haven't seen a single error and / or warning during boot, systemd seems to be happy as well, so I actually think it's ready for "prime time" :smile:

I'll set up a post-install.txt script that'll set everything up (it's actually quite simple and should be very clean when done during installation) and put it into a repository for all to see (and tear apart :grin: )

later (not today, might take way more time) we might discuss including the option into the installer itself :thought_balloon:

This is the output from mount after booting: mount-output.txt

[goranche]

I should note, though, that this will be a stateless system, nothing gets preserved (unless you do it explicitly, of course, which is possible without too much hassle), including /var/spool, which was mentioned in this comment...

it suits my needs, I need a system that'll boot up readonly after installation, since I want the installation to take care of everything to set up a system that does what I need (I mean, that's what post-install.txt is for, after all)...

as with everything, YMMV :wink:

[diederikdehaas]

Nice :+1:

[nagualcode]

"A great step for mankind". Thank you sir. I will test it a lot and report back everything :)

[goranche]

will test it a lot and report back everything

thanks a lot, since I'll be using it for a project soon (it'll be deployed in about a month, 40 devices), so any help and bug reports will be really appreciated... :wink: this project is the reason I was suddenly able to find the time to do it :innocent:

[bbinet]

Thanks a lot. I will also test it. On my side, I still need a way to temporarily remount the partitions as rw so that I can install new packages for example, and I want those changes to persist. I don't really know unionfs, but I guess this is not compatible with this use case?

On 23 January 2016 at 14:07, Goran Blažič notifications@github.com wrote:

will test it a lot and report back everything

thanks a lot, since I'll be using it for a project soon (it'll be deployed in about a month, 40 devices), so any help and bug reports will be really appreciated... [image: :wink:] this project is the reason I was suddenly able to find the time to do it [image: :innocent:]

— Reply to this email directly or view it on GitHub https://github.com/debian-pi/raspbian-ua-netinst/issues/174#issuecomment-174185223 .

[nagualcode]

Could you not command:

mount -o remount,rw /

so to remount root partition as rw?

On Saturday, 23 January 2016, Bruno Binet notifications@github.com wrote:

Thanks a lot. I will also test it. On my side, I still need a way to temporarily remount the partitions as rw so that I can install new packages for example, and I want those changes to persist. I don't really know unionfs, but I guess this is not compatible with this use case?

On 23 January 2016 at 14:07, Goran Blažič <notifications@github.com javascript:_e(%7B%7D,'cvml','notifications@github.com');> wrote:

will test it a lot and report back everything

thanks a lot, since I'll be using it for a project soon (it'll be deployed in about a month, 40 devices), so any help and bug reports will be really appreciated... [image: :wink:] this project is the reason I was suddenly able to find the time to do it [image: :innocent:]

— Reply to this email directly or view it on GitHub < https://github.com/debian-pi/raspbian-ua-netinst/issues/174#issuecomment-174185223

.

— Reply to this email directly or view it on GitHub https://github.com/debian-pi/raspbian-ua-netinst/issues/174#issuecomment-174187159 .

[goranche]

you'd still have /etc and /var mounted with unionfs, so any writes to those locations will go to the underlying tmpfs... there will be another way to access stuff, though, but I'm guessing it'll require a bit of script magic :wink:

[goranche]

well... this should be it... I've only done a single install (on a RPi2) with it (I have additional stuff in both the config and post-install files, but those should have no impact on the read-only "functionality"), but it seems to work...

Any questions / issues / whatever, can be posted here or in that repository. after some testing (and possible contributions), when it's "proved" stable, I'll pull it back to this organisation, maybe even start thinking (after discussing with others, of course) of adding a simple config option to enable readonly mode

[kpfleming]

You can 'bind mount' (using mount --bind) the underlying root filesystem in another location, so you'll have access to it without going through the unionfs mounts on /etc and /tmp.

[goranche]

true, but executing "normal" operations (apt-get update | upgrade | install) will not use the bound location...

if access to the file systems is all you need, then you already have it in the form of /etc_orig and /var_orig, these will be writable after remounting /, but as stated, it has it's quirks

[Mausy5043]

/etc_orig and var_orig ? Why not mount them under /opt or /srv?

[goranche]

@Mausy5043 uhm... yes? as described in the README of my repo

[Mausy5043]

I'd expect a tree under /srv

[goranche]

I disagree, especially for this particular use case

[Mausy5043]

no problem

[goranche]

FYI, I've (already?) found an issue with the read only stuff, it's described as an issue on the repository I linked to (here)

[Mausy5043]

Since no usable PR is forthcoming I'm closing this issue. If you feel the closure is in error, please feel free to re-open and add new information.