Systemd-creds: allow to store host private key in RAM

[ghost]

Hello 👋

Is your feature request related to a problem? Please describe.

• AWS instances does not provide TPM2 virtual chip by default on their EC2 service. In this context, I'm forced to use "host" private key with systemd-creds.
• I want to use systemd-creds to: 1. make sure secrets are in never stored unencrypted on disk, 2. limit secrets availability to system processes 3. Prevent any malicious user to grab a disk image, mount it somewhere else and read the secrets. While 1. and 2. are 👍 , 3. will not be possible because systemd-creds stores its master key on the persistent drive.

(I know the key and secrets would be lost after reboot if the master key is in RAM! It’s a feature: it allows secrets rotation simply by rebooting the machines by grabbing secrets to a vault using AWS authentication system).

Describe the solution you'd like systemd-creds encrypt --with-key-in-ram or something like this.

Then the process looks like this:

1. Instance boots
2. A service fetches secrets in a vault via AWS instance profile credentials
3. The service encrypts secrets via systemd-creds with host key in RAM.
4. Other services securely access secrets handled by systemd-creds.

If someone gets coid access to the root disk: no password in cleartext because systemd-creds key was volatile (so as AWS instance profile creds, they already are)

Describe alternatives you've considered

1. I contacted AWS so they provide a virtual TPM device by default on all of their machines.
2. ln -s /dev/shm/credential.secret /var/lib/systemd/credential.secret: Failed to setup credentials host key: Too many levels of symbolic links

The systemd version you checked that didn't have the feature you are asking for

systemd 250 (v250.3-8.fc36)

Thanks for considering this.

[poettering]

2. A service fetches secrets in a vault via AWS instance profile credentials

Can you elaborate on that? What does this specifically mean? I have no background in awsology. Does this mean network exchange or is this passed in some other way? can this be done from the initrd?

So I was thinking that we could allow passing in a "seed" for that credential secret into the system somehow, but ideally we had that seed available from earliest userspace on.

How do intend the /var/lib/systemd/credential.secret key to be provisioned? i figure you want to pre-provision it centrally? or do you want the systems to generate it locally, but then sync it into the aws storage?

I am simply not at home in AWS, so I don't really know how people use this kind of stuff.

[poettering]

2. ln -s /dev/shm/credential.secret /var/lib/systemd/credential.secret: Failed to setup credentials host key: Too many levels of symbolic links

A bind mount would work. But do note that the credential.secret is checked against '/etc/machine-id, and if it doesn't match it, then we'll delete the file and generate it afresh. This is supposed to be protection against sloppily generated golden images, wherecredential.secretis left in place and replicated over a multitude of systems, but/etc/machine-idis fluhsed out (because older and better known). Thus, it's hard to generate acredential.secret` file outside of the local host.

[ghost]

Hello @poettering.

AWS provides a script used in cloud-init. It initializes short-lived credentials (auto-renewed) to interact with other AWS services or VMs. Roughly, their process looks like this:

• A new virtual machine boots
• That virtual machine is granted access an internal metadata server (only accessible for a specific VM, I guess it’s handled directly by the AWS hypervisor).
• Credentials to access AWS services are fetched (and renewed) by cloud-init.
• The VM can access other AWS services according to previously defined policies (called instance-profiles in this case), like let’s say, a vault.

Fancy thing is, even if these credentials appears on the disk somewhere (they are not supposed to anyway), they will likely not work anymore by the time an attacker read them, because they are renewed every hours or so.

So to get back to what I asked: I want systemd-creds because I believe it’s a very sane way to handle secrets within a running machine. But - because AWS does half the job and does not provide virtual TPMs - systemd-creds uses the hard drive. This "master" password is not short-lived, so any attacker getting access to the disk would be able to read the secrets.

With the process I describe above, systemd-creds is not really the "master" of all secrets anymore, but rather a secure intermediate: changing the systemd-creds master secret at every boot will work well and would be even more secure. I believe then storing the "master" systemd-creds in RAM would be the way to go in virtual machines treated as cattle (only when they miss a TPM of course).

Here is an updated, more detailed process:

1. Machine boots on the cloud.
2. cloud-init fetches/auto-renew "cloud" credentials from its secure local network (all of this is handled by the cloud provider).
3. A system service fetches the secrets needed by my machine on a vault on the network, using the "cloud" credentials.
4. This service encrypts the secrets using systemd-creds, with a host key stored in RAM.

5. Other services securely access secrets handled by systemd-creds.

   2. ln -s /dev/shm/credential.secret /var/lib/systemd/credential.secret: Failed to setup credentials host key: Too many levels of symbolic links

   A bind mount would work. But do note that the credential.secret is checked against '/etc/machine-id, and if it doesn't match it, then we'll delete the file and generate it afresh. This is supposed to be protection against sloppily generated golden images, wherecredential.secretis left in place and replicated over a multitude of systems, but/etc/machine-idis fluhsed out (because older and better known). Thus, it's hard to generate acredential.secret` file outside of the local host.

Okay, I'm gonna try.

[LukeShu]

• That virtual machine is granted access an internal metadata server (only accessible for a specific VM, I guess it’s handled directly by the AWS hypervisor).

• Credentials to access AWS services are fetched (and renewed) by cloud-init.

cloud-init is (unfortunately) a "standard" in that the way a cloud vendor supports it is by PRing specialized support for their system into the cloud-init client, rather than a standard for what the vendor should implement in their system. So how exactly cloud-init fetches data varies from vendor to vendor; I'm not too familiar the AWS' specific implementation, but in general:

cloud-init is a poor choice for secrets because any arbitrary unprivileged process can fetch the secrets. Like all you need is something along the lines of curl http://169.254.169.254/latest/meta-data/.

This can be mitigated a number of ways, such as firewall rules, disabling network access for most services, or (in the case of using cloud-init for the systemd-creds master secret) avoiding SetCredentialEncrypted and instead using .cred files with restricted read access.

[allter]

I came to this issue while trying to understand how to fully comply with the requirement 8.3.2 of PCI DSS 4.0 which will be mandatory beginning from the next year. This rule doesn't allow keys storage in cleartext in any system component.

So some machinery might be needed for complying with this requirement even if TPM is available. For example, using a temporary random keys in a volatile memory, like the issue reporter offered. Also there might be some means of decrypting host private key while systemd boots, possibly at initrd stage.

I've briefly looked at creds.c source. As far as I understood it connects to some decryption process using a unix socket. So systemd-creds looks to be prone to a MITM attack for a person who has permissions to modify the file system contents (such as a compromised root account). So even with kernel lockdown feature set to 'confidentiality', secrets can be accessible to more people than it is needed.

So the real solution is not only to allow to securely initialize host private key at some moment while systemd starts up, but also to allow another way of securely passing credentials to processes mentioned in unit files. Such as passing via stdin of the process being launched. For example, in JSON format {"secretName":"<base64 of a secret>"}