search

ptx2 / gymnasticon

Make obsolete and/or proprietary exercise bikes work with popular cycling training apps like Zwift, TrainerRoad, Rouvy and more.
https://ptx2.net/posts/unbricking-a-bike-with-a-raspberry-pi
MIT License
297 stars 38 forks source link

Enable overlayfs on Raspberry Pi via pi-gen process #58

Closed chriselsen closed 3 years ago

chriselsen commented 3 years ago

This change enables read-only mode on the Gymnasticon Raspberry Pi image via the overlayfs support already built into Pi OS. It also creates a file overctl, which allows checking the current overlayfs mode (RO or RW) and changing of the mode for the next reboot.

All changes are part of the pi-gen process:

ptx2 commented 3 years ago

Looking good!

This covers / but I guess we'd also want to update fstab to mount /boot ro?

The wpa_supplicant.conf and ssh files get rm'd from /boot too. The ssh file is an interesting one with read-only mode as a new host key is generated each time. For on-going ssh access it might be worth booting once in r/w mode, generate host keys and persist them, optionally install your own key, remove the ssh file from /boot and boot back up in r/o mode.

chriselsen commented 3 years ago

Looks like enable_overlayfs in raspi-config also takes care of mounting /boot in RO:

pi@gymnasticon:~ $ mount | grep boot
/dev/mmcblk0p1 on /boot type vfat (ro,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro)

But I need to see how Pi OS handled the wpa_supplicant.conf and ssh files in that case.

chriselsen commented 3 years ago

I guess the question whether to keep the gymnasticon.json file in /boot comes down to user experience: With the wpa_supplicant file it makes sense to remove that file, as it contains the Wifi passport and it's a good idea to not leave that around. But the gymnasticon.json file doesn't include that. Here it might be helpful if a user could open the existing config file from /boot on the SD card and make changes to it.

I tried to test this pull request on my side end-to-end. But I can't get your deploy process running on my side. Even without these changes, pi-gen for me errors out with:

install: cannot stat 'files/gymnasticon.json': No such file or directory

On the build-machine I did:

sudo apt-get update && sudo apt-get upgrade -y
sudo apt-get install -y docker.io git libudev-dev
cd
git clone https://github.com/ptx2/gymnasticon.git
cd gymnasticon
npm install
npm run build
cd deploy/pi-sdcard
./build.sh

Do you see anything obvious missing?

ptx2 commented 3 years ago

Good point on wpa_supplicant.conf! I guess it would be world-readable in /boot but only root-readable once copied into place.

Re: that build error. Sorry if you lost time on this. In hindsight I should have put a note in the repo. There is an issue there I haven't had a chance to look into yet. When you hit that error, you can cd into the pi-gen directory and run CONTINUE=1 ./build-docker.sh and it should continue and succeed.

ptx2 commented 3 years ago

I opened #61 to track that issue.

chriselsen commented 3 years ago

I managed to get a Gymnasticon SD-card image build. In my case using CONTINUE=1 ./build-docker.sh wasn't enough, but I also had to specify (as outlined in the pi-gen doc):

touch ./stage3/SKIP ./stage4/SKIP ./stage5/SKIP
touch ./stage4/SKIP_IMAGES ./stage5/SKIP_IMAGES

I also bought a Raspberry Pi Zero W as "dev" device to not mock around with my "production" unit. Good news is: The resulting image indeed comes up with RO enabled and swap disabled. But the wpa_supplicant file is moved to the overlayfs and removed from /boot. Therefore after a reboot the Pi looses network connectivity. Looks like a bug in Pi OS itself.

To work around that we could:

Also, I'll keep the Pi Zero running for a day or so with bot bike running to make sure it doesn't run out of memory or disk.

chriselsen commented 3 years ago

Turns out that by default "/boot" is still RW this way. As a result Pi OS moves the wpa_supplicant.conf file out of it to the non-permanent overlayfs partition. This means that Wifi connectivity does not "stick" beyond a reboot. Same for the SSH access.

In other words: If you want to have access via SSH to Gymnasticon you would have to re-create these two files before every boot. This might be acceptable given the headless embedded system goal and also given that a password change for the user won't "stick" beyond a reboot either.

On the other side: We could also mount /boot as RO from the get-go. This would prevent the wpa_supplicant.conf and ssh file to be moved and the config would "stick" beyond a reboot. But this also means that the wpa_supplicant.conf file - with the Wifi password in it - remains visible in the /boot directory.

There is no way to access the lower level of the overlayfs without a reboot and therefore persist the wpa_supplicant.conf files somehow.

What path should we take?

ptx2 commented 3 years ago

Nice digging here!

I think allowing wpa_supplicant.conf and ssh files to remain in /boot is my preference of those two options. But as you said it leaves wpa_supplicant.conf more exposed than it would be otherwise. It will be world-readable in /boot rather than root-readable in /etc. It also means ssh keys will be regenerated on every boot which could be seen as both a security and ux issue.

That said, it is simple to implement and gets us pretty far so I think it's worth considering.

It would be ideal if:

Couple ways I could see that working (both of these are more complicated to implement)

1) Write a startup script that automates the reboot flow required to toggle overlayfs on/off when handling the wpa_supplicant.conf and ssh files.

Or,

2) Add another partition that is not under overlayfs so it can be easily toggled from ro/rw at runtime without a reboot, and configure wpa_supplicant and sshd to use that partition for whatever files that need to be written.

I think option 1 is more contained and maintainable.

Example pseudo code:

# Startup script to handle wpa_supplicant.conf and ssh on read-only /
# Note: /boot is mounted RO by default in fstab
# Note: overlayfs is ON by default

if exists /boot/wpa_supplicant.conf or exists /boot/ssh or if we need write access to / for any other reason:
  if overlayfs is enabled:
    mount -o remount,rw /boot
    modify /boot/cmdline.txt to disable overlayfs
    touch /boot/enable_overlayfs
    reboot (the next boot will handle wpa_supplicant and/or ssh and re-enable overlayfs and reboot again)
  else:
    handle wpa_supplicant and/or ssh as normal (these files get removed from /boot here)

if exists /boot/enable_overlayfs:
  if overlayfs not enabled:
    mount -o remount,rw /boot
    modify /boot/cmdline.txt to enable overlayfs
    rm /boot/enable_overlayfs
    reboot (the next boot will be with overlayfs)

If the user wants to do anything else that requires read/write (e.g. change pi user password, ssh-copy-id, etc.) they can just login and use the CLI aliases to toggle rw/ro mode.

It might be useful to show a reminder in .login and/or modify PS1= to remind the user when they're in read-write mode, e.g.

OverlayFS disabled:

(rw-mode) pi@gymnasticon $

OverlayFS enabled:

pi@gymnasticon $

Thoughts?

ptx2 commented 3 years ago

Forgot to add:

I think allowing wpa_supplicant.conf and ssh files to remain in /boot is my preference of those two options. But as you said it leaves wpa_supplicant.conf more exposed than it would be otherwise. It will be world-readable in /boot rather than root-readable in /etc.

It's possible we could mount /boot with umask=0077 to address this. I'm not sure what that might break though. I'd guess it's mounted world-readable for a reason.

It also means ssh keys will be regenerated on every boot which could be seen as both a security and ux issue.

The UX aspect could be worked around by disabling strict host key checking but it's then a security issue.

chriselsen commented 3 years ago

To clarify: My concern around keeping wpa_supplicant.conf in /boot is not so much around the run-time access from the Pi OS. You are right, that this can easily be fixed with fixing permissions. It's more that someone can pop out the SD card, place it into any PC / Mac / phone and mount that FAT-based partition to read out the Wifi password in clear text. To be fair, the attacker could do the same with the ext4 filesystem that is mounted to / and read /etc/... instead. But the barrier to reading from FAT is much lower than from ext4. :-) Possible that I'm overthinking this attack vector though.

With regards to displaying the current RO/RW status. Here is how the login with user pi will look like after the change:

Linux gymnasticon 5.10.11+ #1399 Thu Jan 28 12:02:28 GMT 2021 armv6l
  ____                                 _   _
 / ___|_   _ _ __ ___  _ __   __ _ ___| |_(_) ___ ___  _ __
| |  _| | | | '_ ` _ \| '_ \ / _` / __| __| |/ __/ _ \| '_ \
| |_| | |_| | | | | | | | | | (_| \__ \ |_| | (_| (_) | | | |
 \____|\__, |_| |_| |_|_| |_|\__,_|___/\__|_|\___\___/|_| |_|
       |___/
Last login: Fri Feb 26 13:47:20 2021 from 192.168.0.123

SSH is enabled and the default password for the 'pi' user has not been changed.
This is a security risk - please login as the 'pi' user and type 'passwd' to set a new password.

/ is currently mounted RO and will be mounted RO on next boot
pi@gymnasticon:~ $

We can change the text itself to something else. Modifying $PS1 is an excellent idea!

I learned that the Pi OS image performs a FS resize on the first boot within the Raspberry itself. Therefore setting /boot as read-only from pi-gen won't work, unless we disable that resize. That means we have to set /boot to RO at first boot. If that needs to be done via /etc/fstab it requires two reboots to get / out of RO and back in. The alternative would be to use a startup script that sets /boot to RO via mount -o remount,ro /boot.

With that I was thinking about an approach very similar to what you write above. It would use two services at startup and look like this:

At this point the SSH and Wifi settings are part of the underlay and gymnasticon settings are part of the overlay. Gymnasticon setting changes do not require any additional reboots to change the overlay fs.

In case someone wants to change the Wifi settings in the future:

Therefore in that particular case, we have to go through a bunch of reboots. But the benefit would be that you can create a "no-readonly" file in /boot and basically turn off the read-only mode.

The much simpler alternative would be:

This has the benefit that we minimize the number of reboots for config changes to none, but SSH cert would be regenerated every time (we can actually fix that by removing the SSH file before putting /boot into RO), and the wpa_supplicant file would stay in place and all setting changes (Wifi, SSH, Gymnasticon) remain in the overlay and /boot only.

I can implement either of these approaches and the work is probably marginally more for the first one above.

chriselsen commented 3 years ago

I spent some more time on this and had to do some more changes:

A few lessons learned:

In addition a few decisions:

I've been running a version of Gymnasticon with these changes (as well as the HW watchdog from #66) for > 1 week now, with average daily usage of 1-2 h of biking. So far, so good. No problems popped up whatsoever. It just works.

I also added some more documentation. At this point this PR should be ready to go, along with #66 for the HW watchdog.

ptx2 commented 3 years ago

Hey @chriselsen, again sorry for the slow reply here.

It's more that someone can pop out the SD card, place it into any PC / Mac / phone and mount that FAT-based partition to read out the Wifi password in clear text.

Fair point! I thought about it and decided it was in the clear on the ext4 already. But you're right, it is easier to snatch the file from a vfat partition.

To clarify: My concern around keeping wpa_supplicant.conf in /boot is not so much around the run-time access from the Pi OS. You are right, that this can easily be fixed with fixing permissions.

I should stress I don't know if we can fix that with permissions. It'd require tightening perms for all of /boot as far as I can tell. I'm not sure if that would break something else?

I learned that the Pi OS image performs a FS resize on the first boot within the Raspberry itself. Therefore setting /boot as read-only from pi-gen won't work, unless we disable that resize.

Good catch! I hadn't thought about this fs resize. It seems like we could skip this for the typical read-only user? But also seems fine to keep it and enable RO on first boot.

I can implement either of these approaches and the work is probably marginally more for the first one above.

I was leaning toward the more complicated one because it provides a less-experienced user an easy way to put their Pi on the network with the same security and UX as a regular read-write system. That said, my guess is most users fall into one of two camps: 1) won't set up networking ever, or 2) knows what they're doing and can persist their wpa/ssh stuff themselves.

It looks like you have the simpler option ready-to-go here. Maybe we can move ahead with that and revisit if we see a need.

I noticed you added to the README about the wpa_supplicant.conf security. What do you think about having a similar note shown on .login if we detect a /boot/wpa_supplicant.conf ? Maybe we could also add some pointers on how to set things up securely e.g. "For on-going wifi/ssh access, boot read-write mode, set things up how you want, set a password, etc., remove your wpa_supplicant.conf and ssh files from /boot, and boot read-only mode."

ptx2 commented 3 years ago

Just saw your latest message

This has the added benefit that we can run this service e.g. after the SSH daemon is started as a reaction to the ssh file in /boot. Therefore the SSH key is not regenerated upon further boots. Instead it is made part of the underlay.

Interesting OK I didn't realize that was happening. I assumed we were just going to accept that the keys would generate on each boot. This is a nice side-effect as long as you have the ssh file there for the first boot, which I expect would be the most common case.

Could we not have the same behavior for wpa_supplicant.conf? i.e. copy it instead of move, but also try to remove it, and don't worry if the remove fails.

This would mean we won't have any of the security issues we've been talking about, so long as the files are installed on the first boot?

I've been running a version of Gymnasticon with these changes (as well as the HW watchdog from #66) for > 1 week now, with average daily usage of 1-2 h of biking. So far, so good. No problems popped up whatsoever. It just works.

Great news! Looking forward to getting this out.

chriselsen commented 3 years ago

We can change the order of the script that enables overlayfs on first boot to be executed after the wpa_supplicant file is moved. The result will be that the behavior will be like a standard Pi OS image: wpa_supplicant.conf gets applied and removed.

Without the changes in this PR, if the user wants to make changes to wpa_supplicant.conf by creating an updated file in /boot, that will not work. The file would not be applied and also not removed.

But luckily we can do both, which would actually be a nice compromise:

All that needs to be done for this is change Before=raspberrypi-net-mods.service to After=raspberrypi-net-mods.service. What do you think?

I like the idea of providing some more warnings upon login. We can use /etc/profile.d/sshpwd.sh as a template for that. The SSH Password warning probably also needs to be changed, as a change of password will not "stick".

I do agree with you that the average user (who should not configure Wifi and SSH anyways) will probably never care about all of this and the users who do enable Wifi and SSH probably know how to work around it. I guess for that second set we need to explain sufficiently well what's going on.

ptx2 commented 3 years ago

All that needs to be done for this is change Before=raspberrypi-net-mods.service to After=raspberrypi-net-mods.service. What do you think?

Sounds good!

chriselsen commented 3 years ago

I think now we are really ready for this PR. :-)

Here's what I changed with the latest commit:

A few minor changes to the README file:

ptx2 commented 3 years ago

Looks great!