TemporaryFilesystem strategy from SystemD howto doesn't work

[sezaru]

Hello, in your systemd tutorial, you suggest the usage of TemporaryFilesystem to create a temporary tmpfs directory and store the secrets in a file inside it so the executable in ExecStart= can read it.

Example:

[Service]
TemporaryFileSystem=/private
ExecStartPre=vault-cli get mysecret --output=/private/path/to/secret/file

In my case, I want to use this with Postgres, so I can store the TLS certificate and private key inside Vault and save it inside that directory so Postgres can read it from there.

The issue with that strategy is that the private directory created by TemporaryFilesystem will be removed after the ExecStartPre= call and will not be accessible from ExecStart=.

Do you know if there is some way around it? I know this is more a systemd specific issue, but since it is referenced in your howto, I'm asking here so we can improve the howto by specifying what more is needed to be done for this strategy to work or simply remove it from the howto since it will not work in the end.

More about the systemd issue here.

[sezaru]

I think I found a workaround.

We can use systemd's systemd-tmpfiles to create the tmp directory and then remove it.

First you would need to setup a tmpfs directory to store the data inside the RAM.

After that, you need to create a system-tmpfiles config file for your service, for example, let's say I created a script.conf at root directory with the following content:

D /my_ramfs/private 777 root root - -

This will create a directory called private with chmod 777 with user root and group root at ramfs directory /my_ramfs.

Now, the service can be written like this:

[Service]
ExecStartPre=systemd-tmpfiles --create --remove /script.conf
ExecStartPre=vault-cli get mysecret --output=/my_ramfs/private/file
ExecStart=/script.sh
ExecStop=systemd-tmpfiles --remove /script.conf

This will create the directory first, then wun vault-cli, then your main service and when the service is done, it will remove the directory.

It's not as good as creating the ramdisk at runtime when the service starts though.

[ewjoachim]

This looks interesting, but does it make sure that only the relevant process (or root) can access the file in the tmpfs ? I believe this is one of the things we're especially looking for.

[ewjoachim]

By the way, thank you for creating the ticket ! There's no one in active development on this project these days (I believe this could change), but I'm willing to make sure it's usable. I wonder how I missed the original bug, because I'm sure I managed to have this setup work somehow, I may have to investigate. Nevertheless, you're right that we should not advertize it as-is if it mainly doesn't work.

I've seen the following question, it looks very similar to your own (to the point I'm guessing this could be you :D ). Maybe someone will have an idea over there: https://unix.stackexchange.com/questions/632295/how-to-create-a-ramdisk-during-a-systemd-service-initialization-ex-postgresql

[ewjoachim]

I'm thinking of a way to solve this that would be more agnostic: have a way to write an arbitrary number of secrets into files and then exec an arbitrary command. Ideally it would work kind of identically to vault-cli env but writing files instead of preparing env. (maybe, piggy-backing onto vault-cli env would make sense).

This way, vault-cli would be a command wrapper, and not a PreExec.

Would that work for you ?

[sezaru]

but does it make sure that only the relevant process (or root) can access the file in the tmpfs ?

Not with the example I gave, but you can setup the permissions to the desired ones.

For example, this is what I'm using in production: fstab: tmpfs /private/postgresql tmpfs rw,size=1M,nr_inodes=5k,noexec,nodev,nosuid,uid=postgres,gid=postgres,mode=0600 0 0

tmp_dir.conf: D /private/postgresql 700 postgres postgres - -

Now I use User=postgres and Group=postgres to start the service with that user and be able to write to the directory.

Also, I used ExecStop= command to clean the tmp files, but it's better to use ExecStopPost= because it will run even if the service startup fails.

I've seen the following question, it looks very similar to your own (to the point I'm guessing this could be you :D )

Hahaha that's me!

I'm thinking of a way to solve this that would be more agnostic: have a way to write an arbitrary number of secrets into files and then exec an arbitrary command. Ideally it would work kind of identically to vault-cli env but writing files instead of preparing env. (maybe, piggy-backing onto vault-cli env would make sense).

I think it would work, but would be more intrusive since I would need to replace the original ExecStart= command instead of simply adding ExecStartPre and ExecStopPost steps.

[ewjoachim]

I think it would work, but would be more intrusive since I would need to replace the original ExecStart= command instead of simply adding ExecStartPre and ExecStopPost steps.

Yes, that's what we do with vault-cli env (see here). I wish there was a way to reference the previous value of ExecStart when redefining a new ExecStart or something (ExecStartShim). It looks like there's no such thing, sadly. Especially given that, unless we're using OneShot, we're already guaranteed that there is exactly 1 ExecStart command, so it wouldn't even be ambiguous.

(Outside from vault-cli, of course,) we could imagine specifying a script which would add our shim to the previous ExecStart value, dynamically read using systemctl parsing tools. That would be a complicated way of doing it, but it would probably work. Not that I would actually recommend such a thing, of course.

All in all, I'd feel more confident copy-pasting the upstream ExecStart (and if need be, adding some sort of test allowing me to check that upstream haven't changed their ExecStart without me noticing) than I'd be setting up my own tmp with all that might go wrong. But YMMV :)

[sezaru]

Hey @ewjoachim, I found a new way to simplify my solution.

I know you are leaning more toward the ExecStart approach, but I think it is nice that I register it here so if someone is interested they can find it :)

Basically, I found out that you can use RuntimeDirectory option to create a temporary directory inside /run that will be removed when the service stops.

This replaces the systemd-tmpfiles commands and their configuration files and also eliminates the /etc/fstab line to create a tmpfs since /run already is a tmpfs.

So, now the only changes are these:

[Service]
RuntimeDirectory=my_ramfs/private
RuntimeDirectoryMode=700

ExecStartPre=vault-cli get mysecret --output=/run/my_ramfs/private/file

[ewjoachim]

That might be a nice workaround. The fact you're using User/Group + Mode to restrict access forces you to have a dedicated owner, whereas PrivateTmp shares the folder with the process group, which is a bit more strict (and allows you to use any owner). But most critical applications will have a user anyway.

Do you want to adjust the documentation with your findings ?

(In parallel, as you stated, I'm working on adding --file and --template to vault-cli env, which will provide an alternative solution)

[ewjoachim]

Oh BTW, from reading the systemd doc and your line:

ExecStartPre=vault-cli get mysecret --output=/run/my_ramfs/private/file

I think it may be a little bit more generic to use $RUNTIME_DIRECTORY rather than /run/. Both will work, so it's your call.