Better Secure Remount

[monsieuremre]

Ding dong. Known problems with our current remount implementation are:

• There is a hard dependency on dracut, without it system breaks
• System startup is quite noticeably slower
• There are errors when unmounting on shutdown

I come with a brand new approach. No dracut, no hook. No slow startup, just as fast as normal boot. No errrors in shutdown. So it is all the good stuff without the issues. We get all the hardneing benefits, and even more for some partitions. A systemd oneshot service executes this script, that does the remounting. Well, how do we know what to remount and what to bind? These can change radically depending on the specific system. The answer is, we generate a very solid config file for that very specific system on package install. We use this config file as our fstab to the remounting. The good old 'normal' fstab is still there and functions just a fine, does its thing on startup. We do our thing right after the normal fstab. And since it is a fstab file that we do the remounting with, tho a custom one, unmounting goes just in order perfectly.

Please do test. If you have any problems, let's see them. But as I see it, this is the same protection and a better experience. Ready to merge, I believe, because I tested it and adjusted the startup order so that all services start a ok, and real fast.

[adrelanos]

First, I need to set expectations right. I am not sure this will get merged. There's some good and some issues. Going to reply inline.

[adrelanos]

Terminology:

• the existing implementation (as in git master)
• the suggested implementation (this PR)

So yeah, I see a lot of (potential) issues with the suggested implementation as mentioned above inline.

Other issues:

• This implementation does not support configuration yet as it already does in git master.
• It does not remove the implementation in git master.

Why not iterate on top of the git master implementation?

• 1) Rename the dracut module folders. Append -disabled. So these are ignored. We keep them for now as I expect implementing this is really complex, error prone and will need iterations.
• 2) Modify the existing remount-secure script.
  • Add support for creating a file for use with mount --fstab instead. (So we can test either option.)
  • Or create the file for mount --fstab and at the end run mount --fstab. (I am not sure yet this is possible with reverting user options such as noatime.)

A lot of work already went into remount-secure:

• command line parsing
• pipefail, nounset, shellcheck
• research which folder
• configuration
• user documentation
• testing

The remaining issue (umout at shutdown) of the git master implementation might be already fixed in the systemd version in Debian trixie. I think this much more important to test but I didn't get to it yet.

That's why I am hesitant to completely throw away the existing implementation and start over.

I am however eager to get rid of dracut and replace it with early systemd if that is possible without race conditions.

• There is a hard dependency on dracut, without it system breaks

• Might be fixed by moving from dracut to systemd.

• System startup is quite noticeably slower

• Might be fixed by creating a file for use with mount --fstab.
• Might be fixed by moving from dracut to systemd.

• There are errors when unmounting on shutdown

• Might be fixed in Debian trixie.
• Might be fixed by moving from dracut to systemd.

[monsieuremre]

That's why I am hesitant to completely throw away the existing implementation and start over.

This completely understandable. A lot of work went into the current implementation. It is significantly more tested. It doesn't make sense to throw it all away. Seems a little risky.

We can find a middle ground here I think. First of all, I can iterate on the old implementation to modernize it with the new approach. I can:

• Strip all the dracut stuff, or mark them disabled
• Make the systemd service use the old executable we have
• Modify the old executable to not 'remount' manually, but to generate a fstab manually, and then do the remounting all at once using the custom fstab file. Generating this fstab can be done on every boot if wished, and findmnt can be used instead of parsing /etc/fstab, which as you pointed out, is not optimal.

But I think I can better port all of these to the current implementation because it would pretty much be a rewrite of a good chunk either way. And by that, I mean I already did. I already fixed all the stuff you pointed out as problematic, which they actually were. I already tested using findmnt instead of parsing /etc/fstab with the new implementationf and it works. I also got rid of the generate once on install approach and now we generate the fstab on each startup. This adds a very miniscule extra latency, but is still quite considerably faster than dracut. I had already fixed enforcing file systems, so thats even on the tree.

The others, I didn't push yet. Should I push? Or should I reiterate on the old base and do a more or less complete rewrite, which would yield the same result.

Does this option sound reasonable to you? If there are more problems with the new approach, please point out, I would be more than happy to fix them to comply with the expectations of this package.

[monsieuremre]

This implementation does not support configuration yet as it already does in git master.

This is quite easy to add too. I did this on purpose because I don't believe the current settings will result in any breakage whatsoever. How do I know? Well, I have been daily driving these options on Debian for 3 months and I have been doing basic development and all the other daily stuff. And nothing broke. I think defining more levels to this options introduces more complexity that is not really needed. So I think, an option to disable would more than suffice. But If you wish, I can easily respect the boot paramaters as in the old implementation, no problem. Please do tell if you definetely still want this.

[adrelanos]

That's why I am hesitant to completely throw away the existing implementation and start over.

This completely understandable. A lot of work went into the current implementation. It is significantly more tested. It doesn't make sense to throw it all away. Seems a little risky.

We can find a middle ground here I think. First of all, I can iterate on the old implementation to modernize it with the new approach. I can:

• Strip all the dracut stuff, or mark them disabled

Disabled for now.

• Make the systemd service use the old executable we have

Ok.

• Modify the old executable to not 'remount' manually, but to generate a fstab manually, and then do the remounting all at once using the custom fstab file.

Yes. (If not messing with defaults, noatime, file systems, and whatnot. I am not sure yet it is even possible to create an fstab file without running into corner case issues.)

Generating this fstab can be done on every boot if wished, and findmnt can be used instead of parsing /etc/fstab, which as you pointed out, is not optimal.

Yes.

But I think I can better port all of these to the current implementation because it would pretty much be a rewrite of a good chunk either way. And by that, I mean I already did. I already fixed all the stuff you pointed out as problematic, which they actually were. I already tested using findmnt instead of parsing /etc/fstab with the new implementationf and it works. I also got rid of the generate once on install approach and now we generate the fstab on each startup. This adds a very miniscule extra latency, but is still quite considerably faster than dracut. I had already fixed enforcing file systems, so thats even on the tree.

Nice.

The others, I didn't push yet. Should I push? Or should I reiterate on the old base and do a more or less complete rewrite, which would yield the same result.

Can push at any time to see what changed.

Iterate based on old one seems best since it has nice declarations [1] and tested configuration by kernel parameters.

Does this option sound reasonable to you? If there are more problems with the new approach, please point out, I would be more than happy to fix them to comply with the expectations of this package.

I will check now again inline.

---

Variable could be unconditionally set to NEWROOT="" for now with the new implementation.

[1]

Old implementation remount-secure seems very cleanly implemented using the snippets such as:

_boot() {
   mount_folder="$NEWROOT/boot"
   ## https://lists.freedesktop.org/archives/systemd-devel/2015-February/028456.html
   intended_mount_options="nosuid,nodev,noexec"
   remount_secure
}

That would be good to keep if reasonable.

[adrelanos]

This implementation does not support configuration yet as it already does in git master.

This is quite easy to add too. I did this on purpose because I don't believe the current settings will result in any breakage whatsoever. How do I know? Well, I have been daily driving these options on Debian for 3 months and I have been doing basic development and all the other daily stuff. And nothing broke.

Things change if it gets deployed (on a supposedly stable version) for 1000s of users.

Even the very old implementation before worked in principle for me but then some users complained about random unbootable systems if I remember right. This was due to race conditions when exactly the remount happened.

And then the "no answer" on the systemd mailing list didn't increase my confidence doing this using systemd either.

Therefore I am still worried the systemd approach needs to be reverted later but I guess at least the "write an fstab file instead of multiple calls to mount" can be kept even in that case.

I think defining more levels to this options introduces more complexity that is not really needed. So I think, an option to disable would more than suffice. But If you wish, I can easily respect the boot paramaters as in the old implementation, no problem. Please do tell if you definetely still want this.

There's some weird installers that still do weird stuff such as executing files in /tmp and also noexec for the home folder is controversial. So best to have this configurable.

[adrelanos]

related:

• https://github.com/Kicksecure/security-misc/issues/157

[adrelanos]

I am disabling the remount-secure dracut module.

[monsieuremre]

I am disabling the remount-secure dracut module.

In favor of the new implementation, or planning to drop the functionality?

[adrelanos]

monsieuremre:

I am disabling the remount-secure dracut module.

In favor of the new implementation, or planning to drop the functionality?

In favor for the upcoming systemd solution.

(And to prevent boot slowdown in case I move this to the testers repository before that.)

[adrelanos]

I renamed the dracut module only. This is no change of plan about what I said above at all.

Looking forward to the systemd based solution as discussed.

[adrelanos]

I reverted (restored) the old systemd unit for remount-secure.

https://github.com/Kicksecure/security-misc/blob/master/lib/systemd/system/remount-secure.service

Please edit it to make it work without race conditions.

Note there is still a systemd-present to disable it by default. (Was also git reverted by the revert.)

https://github.com/Kicksecure/security-misc/blob/master/lib/systemd/system-preset/50-security-misc.preset

But that does not hinder development. Enabling it is simple using:

sudo systemctl enable remount-service

[adrelanos]

There was a merge conflict with the systemd unit that I resolved but the systemd unit being the critical part here needs your review anyhow.

[adrelanos]

• https://github.com/adrelanos/security-misc/blob/master/lib/systemd/system/sysinit-post.target
• https://github.com/adrelanos/security-misc/blob/master/lib/systemd/system/remount-secure.service

[monsieuremre]

Ok. The service looks nice and better than mine anyway. It is only the binary secure-remount that we have to port from my patch. Because what you have there is the same with dracut, so slow and problematic to unmount. Should I fork the new base to change the binary secure-remount to use a special fstab like my own patch (this pr) and create a new pull?

[adrelanos]

Ok. The service looks nice and better than mine anyway.

Nice. Though, not yet tested.

It is only the binary secure-remount that we have to port from my patch.

Because what you have there is the same with dracut, so slow and problematic to unmount.

Right.

Should I fork the new base to change the binary secure-remount to use a special fstab like my own patch (this pr) and create a new pull?

Two options:

• A) Add a new mode (command line option) to the existing secure-remount script so it can operate in both modes, current (multiple mount commands) and suggested (create fstab file and mount --fstab).
• B) Don't touch remount-secure). Keep it as is. Copy/paste the existing remount-secure script to remount-secure-fstab (or better name), keep the options parsing / config as much as possible but modify for the new approach (create fstab and mount --fstab). (Then in the systemd service it could be easily swapped from remount-secure(-mount) to remount-secure-fstab.)

That way makes it easier to compare the two different approaches and easier to go back and forth between the two.

[monsieuremre]

Hmm. I'm not sure why we're not just going for the new thing and ditching the old one. It gets the unmount order wrong and results in errors.

[adrelanos]

monsieuremre:

Hmm. I'm not sure why we're not just going for the new thing and ditching the old one. It gets the unmount order wrong and results in errors.

Because I am not sure it will lead to race conditions during boot, mysterious broken boot, the reason why the very first attempt stalled for years.

[monsieuremre]

Is there a Kicksecure ISO repo? There should be. We might be better of creating a secure debian ISO, which guarantees everything installs securely by default. Especially for stuff like this, where there is no direct solution unless we touch /etc/fstab.

[adrelanos]

monsieuremre:

Is there a Kicksecure ISO repo?

ISO repo? No, there's no separate one. I don't think there needs to be a separate one. It will just be integrated with derivative-maker. The APT repository will also be using the same packages just made compatible with the ISO (mostly inventing calamares config files and what else comes up).

There should be. We might be better of creating a secure debian ISO, which guarantees everything installs securely by default.

Once there's an ISO, fstab can be hardened as well.

Especially for stuff like this, where there is no direct solution unless we touch /etc/fstab.

dracut /etc/fstab.sys might be the alternative fstab file we're looking for. dracut prefers /etc/fstab.sys over /etc/fstab. See dracut source code.

[monsieuremre]

This makes no difference still because that file is also meant to be edited by system administrators. Also handling things without extra liabilities like dracut is more preferable. I think we can reasonably eliminate all possible race conditions with this implementation.

[monsieuremre]

We are a package and we can't own /etc/fstab, ok. But what is stopping us from dynamically editing fstab exactly? We can backup the old config and very safely update the fstab to be hardened and not break anything on the system. This would by no means would be a 'clean' solution, but it would be the one and only non-band-aid direct solution, in my opinion.

This can be managed with post pre install scripts. It can get really dirty, true. But i think what I have here is a rather 'clean' solution. Clean in a sense it does not break anything and is completely reversable and it is just a piece of script so we don't really 'own' any file, but of course still not 'clean clean'. So in theory it should also be compatible with Qubes. I went and thought about all the possible edge cases there can be. I believe everything is covered. Tho still this is not a config file, it is a dynamic adjustment script for the fstab.

Would a similar solution be in the scope of this package with adjustments?

[adrelanos]

I wrote this elsewhere but it depends what we try to accomplish: upgrade mount options for:

1) new VM builds (easiest) 2) existing VM builds (maybe doable after 1. is done) 3) distro morphed non-VM builds or systems that only install security-misc (hardest since the most diverse fstab configs)

Doing 1. seems straight forward to contribute a hardened fstab to grml-debootstrap and Qubes as much as hardened mount options are accepted by default or as an opt-in.

The others I have other ideas which goes too far for now which can be looked into when 1. is done.

We are a package and we can't own /etc/fstab, ok. But what is stopping us from dynamically editing fstab exactly?

It's an unclean solution. These tend to create follow-up mess and time required to clean it up. Example:

It seems simple such as https://github.com/Kicksecure/security-misc/pull/147 which seems like "just add this 1 line to add an extra dependency to add libpam-tmpdir" but the follow-up issues were impossible to foresee, huge and time consuming to debug and fix. It broke 2 build tools which are chroot based, cowbuilder and grml-debootstrap. One could argue cowbuilder is unnecessary (which I would disagreed with) but grml-debootstrap is necessary for the build process, not easily replaced and even if replaced the replacement likely could have the same issues. Now still derivative-maker has to carry a lightweight patched/forked version of grml-debootstrap until patches are merged upstream to fix libpam-tmpdir compatibility at the root in a clean way.

And even after it's done, this would still cause follow-up issues, hard to remember and debug in the future. Somebody using Kicksecure and attempting to use some build tool that uses chroot such as cowbuilder, grml-debootstrap would be broken. Things broken default in Kicksecure but not in Debian. And seemingly mysterious. Requiring probably tool-specific workarounds. Awful.

We can backup the old config and very safely update the fstab to be hardened and not break anything on the system.

For example for the Kicksecure or Whonix web server a backup version of fstab wouldn't help much. If the server doesn't boot anymore, I cannot even easily access the backup to restore it. After server reboot, if it fails to boot, there is no more ssh access. That's not a great way to spend my day. Yeah, it can be recovered somehow but until that's done that's another 4-8 hours lost only for that.

So in theory it should also be compatible with Qubes.

But if it's not and the VM fails to boot then it's a headache to debug. Hence, hardening options should be suggested upstream first as much as possible so any issues caused by it are noticed earlier during upstream development.

I wouldn't mind so much about these follow-up issues if I knew someone else would magically take care of these but I know this won't happen and it will all be blamed on me.

[adrelanos]

So meanwhile what would be most helpful as contribution:

1) Take a look at /etc/fstab from a Kicksecure (or Whonix VM).

Harden it as much as possible. Send a PR for /usr/share/doc/security-misc/fstab or so. Based on that I could send PRs to grml-debootstarp (and maybe even Qubes).

This also I might use during build of new VMs. That could become the new default fstab for new VM builds.

(ISO to be worries about later once that is done.)

2) Open tickets at grml-debootstrap / send PRs to harden their auto generated fstab.

This is currently blocked for me because I don't have a hardened fstab file yet.

[monsieuremre]

I see no reason for it to break because the dynamic editing is done very carefully. And tho it is not the cleanest, I can't think of anything else that is more direct, simple, easy to manage and would not randomly break. The backup is there to reverse if so wished, it is not there as a fallback when things break, the idea is that things won't break, in an ideal scenario. I honestly can't find any direct and viable alternative other than this.

[monsieuremre]

Actually, nevermind. I think I found a solution that you want. I'm gonna create a new pull and close this one.