Open adrelanos opened 1 year ago
A little off topic:
By the way the guide you provided, I have also gone through it several times before, but it should be noted that it is super duper old. Mounting /tmp
with noexec
will not cause any problems almost ever. What they are mentioning is a legacy issue, that some old old software used to install itself by running scripts in /tmp. This is super prone to exploits and no legit program should ever run anything on /tmp
, ever, there is no legit reason for this. The software that breaks because of this is the one thing that would need fixing. And also, no one does this anymore anyway. As such, this has not been a problem for like ever. The issue they mention on that link is from 2001. So I think it isn't much of a stretch to assume this is no more relevant.
The only case where running some kind of anything on /tmp
that I can think of is, maybe, software development and debugging. Like some sort of compilation stuff while debugging, I don't know, maybe then, it might be used legitimately. But even then I think it is not the only way. But this won't be the only thing that will break given our hardened kernel parameters. So no, we should go nexec for tmp all the way, which I think we are, so nice.
By the way, while we are at it, I wanna refer to #178, where you said:
Compatibility with the kernel version in Debian stable is interesting but not a blocker for getting merged as long as nothing breaks.
In that case I would suggest we set the following parameters in grub, as they will work for the newer systemd versions. systemd.mount-extra=udev:/dev:devtmpfs:defaults,nosuid,noexec systemd.mount-extra=tmpfs:/dev/shm:tmpfs:defaults,nosuid,noexec,nodev systemd.mount-extra=tmpfs:/tmp:tmpfs:defaults,nosuid,noexec,nodev
This has the same behavior as adding new lines to fstab. And since these are universal and system independent, they can be set this way. They won't change based on the users partitioning scheme.
I am also surprised we're the first ones inventing a hardened fstab file and publishing it.
There are some problems.
First of all,
/run
is mounted with noexec by default on debian 12.
What is "Debian"? I mean, where is it down, how, source file? This could be specific to initramfs-tools vs dracut vs other init systems vs booting a system for real vs systemd-nspawn etc.
If it is different on a debian qube, that is a qubes os thing and not a debian thing.
Also Qubes is also a legitimate user of Debian and Qubes most likely didn't purposefully downgrade /run. Rather something else (something similar to initramfs-tools vs dracut).
Secondly, anything that is not on disk should never be bound. Not to itself, not to anywhere.
Why?
For example hardening
/dev
would be like this:udev /dev devtmpfs defaults,nosuid,noexec 0 0
Added and then from the running system sudo mount -a
. Error message:
mount: /dev: udev already mounted or mount point busy. dmesg(1) may have more information after failed mount system call.
Works for me only when using bind
.
Could say your syntax works when booting but not from the running system. But using bind seems easier to to work with since then no reboot is required every time.
Maybe the current syntax would fail to boot (or actually do nothing).
In any case, a completely hardened fstab file that is fully tested would be most helpful.
I would recommend using the
defaults
option at the beginning.
Ok.
Also nofail should not be used when not debugging. We probably want the mounting to happen for sure. If not happening, we want it to fail and not boot.
If mounting /var as noexec,nosuid fails, I'd prefer a booting system. Way easier to debug and fix than a non-booting system.
Unit testing can be done in other places such as in systemcheck.
In any case, nofail can stay for the first iteration and maybe removed later when this is stable.
Also, is this planned to be a complete fstab file?
Yes, a complete fstab that can be used in Kicksecure VMs. Would a great start.
Because there are no lines for the regular real partitions, like the root partition
/
.
There are no partitions in the VM except for:
/dev/disk/by-uuid/26ada0c0-1165-4098-884d-aafd2220c2c6 / auto nofail,defaults,errors=remount-ro 0 1
And also /boot/efi but the latter is simply not done yet. I don't know if it has a stable uuid yet or maybe blocked by migration to (maybe) mkosi which creates stable uuids presumably.
A little off topic:
By the way the guide you provided, I have also gone through it several times before, but it should be noted that it is super duper old. Mounting
/tmp
withnoexec
will not cause any problems almost ever. What they are mentioning is a legacy issue, that some old old software used to install itself by running scripts in /tmp. This is super prone to exploits and no legit program should ever run anything on/tmp
, ever, there is no legit reason for this. The software that breaks because of this is the one thing that would need fixing. And also, no one does this anymore anyway. As such, this has not been a problem for like ever. The issue they mention on that link is from 2001. So I think it isn't much of a stretch to assume this is no more relevant.The only case where running some kind of anything on
/tmp
that I can think of is, maybe, software development and debugging. Like some sort of compilation stuff while debugging, I don't know, maybe then, it might be used legitimately. But even then I think it is not the only way. But this won't be the only thing that will break given our hardened kernel parameters. So no, we should go nexec for tmp all the way, which I think we are, so nice.
Great to know!
By the way, while we are at it, I wanna refer to #178, where you said:
Compatibility with the kernel version in Debian stable is interesting but not a blocker for getting merged as long as nothing breaks.
In that case I would suggest we set the following parameters in grub, as they will work for the newer systemd versions. systemd.mount-extra=udev:/dev:devtmpfs:defaults,nosuid,noexec systemd.mount-extra=tmpfs:/dev/shm:tmpfs:defaults,nosuid,noexec,nodev systemd.mount-extra=tmpfs:/tmp:tmpfs:defaults,nosuid,noexec,nodev
This has the same behavior as adding new lines to fstab. And since these are universal and system independent, they can be set this way. They won't change based on the users partitioning scheme.
I want to test a hardened /etc/fstab first over several releases. Then everything is in 1 expected place and easy for users to customize.
When nothing breaks, maybe systemd kernel parameters can be used.
Maybe /etc/fstab could contain a comment to mention how /dev/, /dev/shm, /tmp is hardened (by using kernel parameters). That would be good too. Feel free to send a PR. That could remain open until the next Debian release.
Compatibility with the kernel version in Debian stable is interesting but not a blocker for getting merged as long as nothing breaks.
True, I said that. However, in this case I would make an exception. This is the second step. Would be rushing ahead and confusing myself to merge that right now. Also because of the potential regressions when porting to the next major version of Debian. Could take a while to remember and figure out what is causing it.
Because /etc/fstab hardening is realistic in this Debian release cycle so that should be done as first step.
I am also surprised we're the first ones inventing a hardened fstab file and publishing it.
I don't think we are. Debian already does some hardening for api file systems. For example run is fully hardened and /dev/shm is partially hardened on debian, which would not be the case by default if debian devs didn't set it that way.
What is "Debian"? I mean, where is it down, how, source file?
On a debian system run cat /usr/share/initramfs-tools/init
. This is where the mount options for api file systems are set. You can manually edit this script file to set whatever you want. In this file for example, there is a line mount -t sysfs -o nodev,noexec,nosuid sysfs /sys
. This line is the choice of debian devs. By the way, the difference in /run
hardening may be different on a system with dracut, but on debian dracut is not the default.
Also, I have discovered that, api file systems are still remounted if we modify the fstab. Because when these file systems are mounted the first time, there is no universal way of accessing the fstab file, so the init script just runs and does its thing. If there is a line for an api file system in fstab, they are remounted by systemd-remount-fs.service. So the ultra clear solution would be modifying the init script, which I think is possible for packages to do as well.
You asked
Why? not to use bind mounts for non disks things. Because, first, these file systems are on ram for a reason. They are volatile and fast. Binding them to themselves or to somewhere else introduces unneccessary extra mounts and binds. This is extra complexity. Not to mention that some sub-directories under these paths are already bound to other places for legacy reasons, so when we bind the parent or the child, this may introduce a lot of multiple bind mounts and one path being accessible through several ways. Which, at best is just superfluous functionality, at worst is a vector that slow us down for no reason.
Could say your syntax works when booting but not from the running system. But using bind seems easier to to work with since then no reboot is required every time.
Because it is meant to work on boot only. Bind mounting provides no solution because we never use fstab to mount things midway. But it creates new problems on its own. This would also be true if we had a line for proc there. We can also use bind, but then if a process was accessing that path before we bind, it won't be affected by our hardening at all, including init I think, which obviously can't be good if anything.
So for the ultimate solution, I suggest we find a way to solve the api file systems in initramfs. For other, real partitions, we already have a working solution that uses drop in config files, from my old pull request. So combining these would be, I think, optimal.
What is "Debian"? I mean, where is it down, how, source file?
On a debian system run
cat /usr/share/initramfs-tools/init
.
This is only 1 way to use Debian. That is initramfs-tools, which Kicksecure doesn't use. At time of writing, Kicksecure uses dracut by default. Reference: https://forums.whonix.org/t/replacing-initramfs-tools-with-dracut/4487
It is also possible to have an initrdless system (a kernel with all required modules compiled-in), perhaps other init systems (in the future maybe mkosi-initrd, which is a systemd based initrd without special init queues by initramfs-tools or dracut, which might simplify initrd a lot), and certain types of virtualization such as systemd-nspawn does not use and initrd. Might be similar for docker, OpenVZ.
They all however should be honoring /etc/fstab.
Also, I have discovered that, api file systems are still remounted if we modify the fstab. Because when these file systems are mounted the first time, there is no universal way of accessing the fstab file, so the init script just runs and does its thing. If there is a line for an api file system in fstab, they are remounted by systemd-remount-fs.service. So the ultra clear solution would be modifying the init script, which I think is possible for packages to do as well.
There's a dracut module right in security-misc:
usr/lib/dracut/modules.d-disabled
It was working but it was said to be slow. Maybe the slowness issues could be worked on by debugging the script. Tracing, debug output execution times throughout its run. Perhaps parallelization, i.e. running the mount commands into the background.
If you're interested in initramfs-tools or any other init system, you could contribute fstab hardening upstream. That would also get way more scrutiny and probably stability than anything invented and only in Kicksecure.
So for the ultimate solution, I suggest we find a way to solve the api file systems in initramfs.
Also seems we're going in circles here (as the dracut module was previously enabled) and without writing a technical design with all the different approaches, pros and cons of each, solution impartial research this will probably keep going in circles.
Created an overview now: https://www.kicksecure.com/wiki/Dev/remount-secure
Contributions welcome.
There are at least 6 different approaches to mount options hardening, 3 different types of mount points (API, partition, folder) and various different environment.
This is only 1 way to use Debian. That is initramfs-tools, which Kicksecure doesn't use. At time of writing, Kicksecure uses dracut by default. Reference:
Ok very interesting. I have said this is the debian default because debian by default does not use dracut. Because systemd is also just one way to use debian, since they offer other init systems in their repos. But since it is the default, I think it is reasonable to say debian by default uses systemd, and by default uses initramfs-tools. I see the reasoning the porject had to switch to dracut is to support a live system. Is this necessary? I'm pretty sure Ubuntu uses initramfs and they offer a live system. So I don't know why you switched there. Does this make development somehow easier?
If you're interested in initramfs-tools or any other init system, you could contribute fstab hardening upstream. That would also get way more scrutiny and probably stability than anything invented and only in Kicksecure.
I agree. But for this, the place to contribute is the package maintainer of the distribution. Because even this semi-hardening is the choice of debian packagers, and not the upstream devs, far as I know. So instead of bothering with that, we might as well fix it for us ourselves.
Created an overview now: https://www.kicksecure.com/wiki/Dev/remount-secure
Good. This is a good way to keep track of this long discussion. It is more complicated than the our good old usual hardening and there are many ways.
I am getting confused by the many different approaches, tickets, pull requests on this topic.
Wasn't there some git branch you wanted to restore, older approach which you wanted to re-vitalize? Instead https://github.com/Kicksecure/security-misc/pull/195 was done?
found the comment here: https://github.com/Kicksecure/security-misc/pull/165#issuecomment-1878272412
Ok good. Soon when I find the time I will restore the branch, check everything again and reopen the pull.
This is only 1 way to use Debian. That is initramfs-tools, which Kicksecure doesn't use. At time of writing, Kicksecure uses dracut by default. Reference:
Ok very interesting. I have said this is the debian default because debian by default does not use dracut. Because systemd is also just one way to use debian, since they offer other init systems in their repos. But since it is the default, I think it is reasonable to say debian by default uses systemd, and by default uses initramfs-tools. I see the reasoning the porject had to switch to dracut is to support a live system. Is this necessary? I'm pretty sure Ubuntu uses initramfs and they offer a live system. So I don't know why you switched there. Does this make development somehow easier?
Live mode (grub-live) indeed also works with initramfs-tools.
The full story about porting to dracut is here: https://forums.whonix.org/t/replacing-initramfs-tools-with-dracut/4487
Main reasons here: https://forums.whonix.org/t/replacing-initramfs-tools-with-dracut/4487/15
Updated summary why here: https://forums.whonix.org/t/replacing-initramfs-tools-with-dracut/4487/32
The very main reason is: https://www.kicksecure.com/wiki/Ram-wipe
After long consideration of the many different options, I've decided that /etc/fstab file based mount options hardening is the most suitable way to implement secure mount options for Kicksecure. This is the way forward.
Contributions to /usr/share/doc/security-misc/fstab-vm
are welcome.
Will there be a way to control some mount options or control the timing on mounting? If it will use systemd to trigger the fstab read, being able to add an override.d like After=zfs-mount.service
to the unit file would be useful.
With current configuration released to end users, I believe I am having a similar issue to this one: (I have a ZFS dataset for /var/ and /var/log, etc)
Caused quite an interesting scenario where all my data became hidden with a recent patch.
https://old.reddit.com/r/zfs/comments/hpnmiw/systemd_mount_order_between_zfs_and_etcfstab_bind/
After long consideration ofthe many different options, I've decided that /etc/fstab file based mount options hardening is the most suitable way to implement secure mount options for Kicksecure. This is the way forward.
Contributions to
/usr/share/doc/security-misc/fstab-vm
are welcome.
Mounting /tmp
with noexec
would break initramfs-tools
as reported here:
Will there be a way to control some mount options or control the timing on mounting? If it will use systemd to trigger the fstab read, being able to add an override.d like
After=zfs-mount.service
to the unit file would be useful.
No. There's no need for it. Simply a modified /etc/fstab
configuration file by default. So any settings can be easily customized by the user as usual. No extra complexity, extra features, extra changes.
No. There's no need for it. Simply a modified /etc/fstab configuration file by default. So any settings can be easily customized by the user as usual. No extra complexity, extra features, extra changes.
fstab is a user artefact. So the solution there would be to somehow configure the calamares installer or whatever we plan to use to set these secure defaults in the installer. But even then, there won't be binding anything. You can not add bind lines to fstab in the installer. So package would need to do that. And we are back to here.
Will there be a way to control some mount options or control the timing on mounting? If it will use systemd to trigger the fstab read, being able to add an override.d like After=zfs-mount.service to the unit file would be useful.
If you messed a little with these stuff you probably know that this gets really complicated real quickly. Even with /var and /var/log the order is not possible to configure in the fstab file. In a unit file, it is possible, but then it is almost always guaranteed to slow up the boot massively (when we are binding, especially nested binding).
No. There's no need for it. Simply a modified /etc/fstab configuration file by default. So any settings can be easily customized by the user as usual. No extra complexity, extra features, extra changes.
fstab is a user artefact.
So the solution there would be to somehow configure the calamares installer or whatever we plan to use to set these secure defaults in the installer.
Calamares supports fstab configuration already.
Quote https://github.com/calamares/calamares/blob/calamares/src/modules/fstab/fstab.conf
options: "defaults,noatime,mode=1777"
But even then, there won't be binding anything. You can not add bind lines to fstab in the installer.
Why not?
Why not?
Because the installer is a dynamic program. In like the expected scenario the user plugs in a usb and live boots to kicksecure, there is the live environment, he looks around for a minute, ok nice, now he decides to install. Calamares installer, ok nice. When you are at the place where you do the partitions and stuff, the installer does not do everything for you. The installer just makes suggestions basically, but the user still chooses the partitions and whatnot. In an ideal scenarion, we modify the default suggestions made by calamares to use secure defaults. Ok nice, but then there can't be any bind mounts. Which binds are we going to include? Does calamares even have an interface to suggest bind mounts in the installer?
There we need some program to make decisions with elaborate if else statements. Ok if boot is not a separate partition then do this, if it is, do this, if it is like a different file system, then obviously do these stuff because it is different on this. If the user changes the installation or formats one of the partitions or like a billion other things, we still have to at some point decide dynamically. The "let's put the config and forget about it and it works universally" logic only works for api file systems with kernel params and real mounts with real partitions with drop in configs. For binds, there is no such solution that I could find, at least not without creating several layers of abstraction.
The other issue is the fact that bind mounts are already problematic on their own. I remeber I encountered this problem. There is no way to make nested binds work universally. Firstly, having the wrong order with nested binds will result in two valid points of accessing a directory, one of which will easily circumvent the hardened mount options. To remove this possible work around, they have to be mounted in cordination with the order from inner to outer. And that, requires some big after
before
requires
lines in the systemd config file because for some reason I remember it always tried to mount /var
before /var/tmp
. And then, once you find a solution with before after stuff, it just breaks after you install auditd. I remember I encountered this issue with nested bind mounts and I said ok, it is not worth it doing this with mount units. Is fstab going to be any less problematic? Doubt it. Maybe we can just avoid the nested binds all together? Yeah then, maybe fstab will work, but then, at that point the mount units will also work too.
Sorry for the long answer. Think of it as if I'm brainstorming loudly.
When you are at the place where you do the partitions and stuff, the installer does not do everything for you.
Partioning is automatic.
Which binds are we going to include?
Same as https://github.com/Kicksecure/security-misc/issues/157#issuecomment-1938753325
Does calamares even have an interface to suggest bind mounts in the installer?
No, and it's probably not needed either. Seems like https://github.com/calamares/calamares/blob/calamares/src/modules/fstab/fstab.conf can do that.
And if needed, then calamares seems easily extensible with python modules. This is also a functionality which might be welcome if contributed upstream. But even just the fstab.conf (which is a calamares fstab.conf, not a literal /etc/fstab) might be sufficient. Calamares already has some code for custom mount options.
There we need some program to make decisions with elaborate if else statements. Ok if boot is not a separate partition then do this, if it is, do this, if it is like a different file system, then obviously do these stuff because it is different on this. If the user changes the installation or formats one of the partitions or like a billion other things, we still have to at some point decide dynamically.
I am not sure Calamares supports custom partitioning or if that should be supported. But if Calamares can do that, then fstab.conf (speak calamares fstab module) can probably handle that as well.
The "let's put the config and forget about it and it works universally" logic only works for api file systems with kernel params and real mounts with real partitions with drop in configs. For binds, there is no such solution that I could find, at least not without creating several layers of abstraction.
The other issue is the fact that bind mounts are already problematic on their own. I remeber I encountered this problem. There is no way to make nested binds work universally.
If that's true, then this ticket is a cannot implement
anyhow.
Firstly, having the wrong order with nested binds will result in two valid points of accessing a directory, one of which will easily circumvent the hardened mount options. To remove this possible work around, they have to be mounted in cordination with the order from inner to outer. And that, requires some big
after
before
requires
lines in the systemd config file because for some reason I remember it always tried to mount/var
before/var/tmp
. And then, once you find a solution with before after stuff, it just breaks after you install auditd. I remember I encountered this issue with nested bind mounts and I said ok, it is not worth it doing this with mount units. Is fstab going to be any less problematic? Doubt it. Maybe we can just avoid the nested binds all together? Yeah then, maybe fstab will work, but then, at that point the mount units will also work too.
Then upstream needs to be consulted regarding these issues.
Once again. Calamares is a solution, no doubt, at least when installing. If we want a universal solution that ships with a package tho, the remount service is really a good solution. I have done some reading of systemd source code. For api file systems, changing fstab entries just makes systemd remount them with the new options using a service. The initial mounting still takes places with the options of the init script, written by debian mostly. So like, having our own service is not so different than modifying fstab for these file systems at least.
The trickster here is, the remount service systemd has is written in c. Shell scripting a bunch of remounts is slow. Also shell scripts are non universal. Bash or zsh, synthax changes. We should avoid having shell executables whenever possible, especially for services. This is not only for speed, but also for security.
The suid-disabler-permission-hardener deserves a separate repo. So does the remount-secure service.
Having the mount service and permission thing trigger on boot would be enough. Also we would trigger the permission thing with a post apt hook (better than using stat-override for various reasons).
I can personally write these things. I can write makefiles, and dedicated apparmor profiles and like at the end, you would just run make all and you would have everything in a conventional structure under a directory. Or literally like github workflows to build them binaries and put them in a good structure. Packaging that directory then would be the classical good old stuff that you already know. This is just my suggestion. Moving away from shell scripts is a better and cleaner way, and provides modularity. Having several services under a single package is not developer friendly. and if you think this would be complicated than necessary, well, it wouldn't be because the source code is gonna really short and minimalistic.
Either way, moving away from shell scripts or not, I think these services deserve separate repos.
The initial mounting still takes places with the options of the init script, written by debian mostly.
That one I would like to see the source code for.
The trickster here is, the remount service systemd has is written in c. Shell scripting a bunch of remounts is slow.
Why slow? The slowness isn't inherited from being shell scripting. If you run the remount-secure from within the running system it should complete in seconds. Perhaps 1-3 seconds. Please measure that.
But during boot, it might be a lot slower slower. 10, 20 seconds? But that's not because of the shell. There must be other reasons for that.
So first, please compare the speeds:
Bash or zsh, synthax changes.
In 12 years, bash never required a syntax change that I can remember.
We should avoid having shell executables whenever possible, especially for services. This is not only for speed, but also for security.
I am yet to see a threat model where remount-secure being shell versus compiled language results in a significant difference.
The suid-disabler-permission-hardener deserves a separate repo. So does the remount-secure service.
Either way, moving away from shell scripts or not, I think these services deserve separate repos.
Yes, these functionalities are really heavy-weight and ideally had separate repositories.
milestone: "one day"
Having the mount service and permission thing trigger on boot would be enough. Also we would trigger the permission thing with a post apt hook (better than using stat-override for various reasons).
For permission hardener that is already implemented. It has an dpkg trigger that runs wherever any package on the system is installed or updated. dpkg-statoverride is the tool to configure DPKG to prevent restoration of layer permissions when packages that contain files with SUID that are removed are upgraded. If you think there's something missing, please open a separate issue for that.
I can personally write these things. I can write makefiles, and dedicated apparmor profiles and like at the end, you would just run make all and you would have everything in a conventional structure under a directory. Or literally like github workflows to build them binaries and put them in a good structure. Packaging that directory then would be the classical good old stuff that you already know. This is just my suggestion.
Please maintain as independent upstream: source code, binaries, popularity, review, etc.
That would help so I don't have to look into it. If it's independently used and reviewed, then integration into Kicksecure will be much simpler.
@monsieuremre in https://github.com/Kicksecure/security-misc/issues/203#issuecomment-1965219948:
Once again, I want to give my opinion here. I am not a big fan of bind mounting things. My suggested implementation was, how would it be done if we absolutely wanted bind units. It goes without saying that remounting anything is completely unnecessary and does nothing but add extra overhead and complexity. These can be hardened with drop in files, as I suggested previously. Api file systems can be hardened from the kernel command line, again, a drop-in file. Bind mounting stuff is where we can't use drop in files. Actually, if we wanted, we could. But I already am not a fan of bind mounts. We can create bind units under /usr/lib/systemd, but really, should we be the one to have the var.mount unit? Like someone else might want to own that file, I don't know. My suggestion is, let's do drop in's boot parameters. I don't support bind mounts. But if it is desired, it can be done with either
- a service such as this (messy)
- boot parameter, again, fstab lines (preferred)
- systemd units under /usr/lib (less messy, but still is messy)
For API file systems and real partitions, a service or a hook should not be used. This is not only unnecessary, but introduces overhead.
@adrelanos above:
After long consideration of the many different options, I've decided that /etc/fstab file based mount options hardening is the most suitable way to implement secure mount options for Kicksecure. This is the way forward.
Contributions to
/usr/share/doc/security-misc/fstab-vm
are welcome.
Solutions are all messy but the least messy I've settled with is
Otherwise there are too many mount points.
And each requirement a different mechanism to change these would make it even more messy.
let's do drop in's boot parameters
It cannot really cover all mount points A), B), C)?
It cannot really cover all mount points A), B), C)?
It surely can cover everything apart from API file systems, and they can be covered with just drop in kernel parameters, but its just not a good idea.
It cannot really cover all mount points A), B), C)?
It surely can cover everything apart from API file systems, and they can be covered with just drop in kernel parameters, but its just not a good idea.
Anything missing here?
Which software / source code sets the initial insecure mount options anyhow?
Could we fix it there instead of adding a band-aid on top?