Using sops with arion-compose

[nviets]

I am trying to set up a container that needs a secret stored with sops-nix. My .sops.yaml and secrets/ are set up, but I'm not sure how to configure my arion-compose.nix file. I have something like:

{ lib, config, pkgs, ... }: let
  mb_key = ''
    export MY_KEY=${config.sops.secrets."my_key".path}
  '';
in {
  project.name = "alma";
  services = {
    gui.service = {
      hostname = "gui";
      image = "almalinux:9";
      command = [
        "${pkgs.writeScript "entrypoint" "${mb_key}"}"
      ];
    };
};

But it's failing on:

error: attribute 'sops' missing

       at /home/nviets/repos/alma-docker/arion-compose.nix:4:21:

            3|   wb_key = ''
            4|     export MY_KEY=${config.sops.secrets."workbench_key".path}
             |                     ^
            5|   '';

Thanks in advance!

[Grantimatter]

I'm working on doing basically the same thing for my setup. From what I've gathered, you can't use the values of the secrets directly in nix configurations in this way, as this will just insert the path to the /run/secrets/ directory of the secret that sops-nix generates.

sops-nix templates seem like they may work for this. I'm hoping to have a working example soon to share.

*Edit: Templates are not the solution here either. This will also just insert the path to a config generated with sops-nix. sops-nix cannot be used to insert secrets directly into nix configurations. See using-secrets-at-evaluation-time in the sops-nix readme. At this point I am thinking about just using sops-nix to create the config, and either use arion to extend the compose file with docker compose secrets, or forego arion and just use a plain docker-compose.yml in my setup that can reference the secrets files that sops-nix outputs.

The only alternative I'm seeing is scalpel which hasn't had a commit in over 2 years, so it's most likely dead.

If there's any other alternatives that anyone knows about, or if you ended up solving this since your post, I'd love to hear your solution!

In case I forget to come back to this, whatever method I end up using will be in my NixOS configuration for my Homelab Setup.

[KiaraGrouwstra]

that docker compose/swarm secret mechanism exposes the secrets as a file to the container. when you only need to access a secret during a docker build step, another approach is to use docker buildx secrets.

(for using sops-nix, also see #247 on passing inputs to arion.)

[thenbe]

The workaround shown in #247 is great, it solved my flake issues. The build passes and the container starts.

When it comes to sops-nix, however, that still doesn't quite work. There are a couple of errors when running arion up:

grafana-1  | could not create symlink /etc/hosts at /nix/store/rg5rf512szdxmnj9qal3wfdnpfsx38qi-setup-etc.pl line 133.
grafana-1  | sops-install-secrets: failed to mount filesystem for secrets: cannot mount: operation not permitted
grafana-1  | Activation script snippet 'setupSecrets' failed (1)

After the container starts, I can verify that /run/secrets directory does not exist.

Here are the extended logs:

$ arion up --remove-orphans
trace: warning: system.stateVersion is not set, defaulting to 24.11. Read why this matters on https://nixos.org/manual/nixos/stable/options.html#opt-system.stateVersion.
trace: Obsolete option `boot.tmpOnTmpfs' is used. It was renamed to `boot.tmp.useTmpfs'.
/nix/store/a2q1w13d55asknig0gfb3rn3vqhb7rxb-docker-compose.yaml
WARN[0000] /home/nbe/projects/sail/.tmp-arion-docker-compose1910732-0.yaml: `version` is obsolete
[+] Running 2/0
 ✔ Container full-nixos-loki-1     Removed                                                                                                                                                                                                                                                                                                                                                                                                                                           0.0s
 ✔ Container full-nixos-grafana-1  Created                                                                                                                                                                                                                                                                                                                                                                                                                                           0.0s
Attaching to grafana-1
grafana-1  |
grafana-1  | <<< NixOS Stage 2 >>>
grafana-1  |
grafana-1  | booting system configuration /nix/store/rhyjydwq95ca9qy7v311n037ll97shh7-nixos-system-unnamed-24.11pre-git
grafana-1  | running activation script...
grafana-1  | setting up /etc...
grafana-1  | could not create symlink /etc/hosts at /nix/store/rg5rf512szdxmnj9qal3wfdnpfsx38qi-setup-etc.pl line 133.
grafana-1  | setting up secrets...
grafana-1  | /nix/store/52a2jyq0y5v93g6276cq5j0jqnid404r-sops-install-secrets-0.0.1/bin/sops-install-secrets: failed to mount filesystem for secrets: cannot mount: operation not permitted
grafana-1  | Activation script snippet 'setupSecrets' failed (1)
grafana-1  | Setting up sops templates...
grafana-1  | Traceback (most recent call last):
grafana-1  |   File "/nix/store/wgzaf45f12da7pq0jri545i9yn248a8i-substitute", line 27, in <module>
grafana-1  |     main()
grafana-1  |   File "/nix/store/wgzaf45f12da7pq0jri545i9yn248a8i-substitute", line 24, in main
grafana-1  |     print(substitute(target, subst))
grafana-1  |           ^^^^^^^^^^^^^^^^^^^^^^^^^
grafana-1  |   File "/nix/store/wgzaf45f12da7pq0jri545i9yn248a8i-substitute", line 15, in substitute
grafana-1  |     with open(path) as f:
grafana-1  |          ^^^^^^^^^^
grafana-1  | FileNotFoundError: [Errno 2] No such file or directory: '/run/secrets/DATABASE_PASSWORD'
grafana-1  | Activation script snippet 'renderSecrets' failed (1)
grafana-1  | starting systemd...
grafana-1  | systemd 255.6 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON +UTMP -SYSVINIT default-hierarchy=unified)
grafana-1  | Detected virtualization docker.
grafana-1  | Detected architecture x86-64.
grafana-1  |
grafana-1  | Welcome to NixOS 24.11 (Vicuna)!
grafana-1  |
grafana-1  | Queued start job for default target Multi-User System.
grafana-1  | [  OK  ] Created slice Slice /system/modprobe.
grafana-1  | [  OK  ] Started Dispatch Password Requests to Console Directory Watch.
grafana-1  | [  OK  ] Started Forward Password Requests to Wall Directory Watch.
grafana-1  | [  OK  ] Reached target Local Encrypted Volumes.
grafana-1  | [  OK  ] Reached target Login Prompts.
grafana-1  | [  OK  ] Reached target Containers.
grafana-1  | [  OK  ] Reached target Path Units.

[KiaraGrouwstra]

@thenbe what about if you give your arion service service.privileged = true;?

[thenbe]

@KiaraGrouwstra Yup that works. Thanks for the hint!

{
  service.privileged = true;
  service.volumes = [ "${toString ./.}/keys.txt:/var/lib/secrets/age" ];
}

---

It also works if we only grant it the SYS_ADMIN capability, which "grants a smaller subset of capabilities to the container, compared to the --privileged switch" (source). See https://docs.docker.com/reference/cli/docker/container/run/#privileged

-service.privileged = true;
+service.capabilities = { SYS_ADMIN = true; };

[thenbe]

I'm still not sure exactly why the container needs to be privileged, the best I have so far is this similar discussion in the nix repo: https://github.com/NixOS/nix/issues/3059

---

After searching a bit more, I found another solution that sidesteps the privilege requirement altogether.

{
  sops.useTmpfs = true;
}

See the issue and PR for more info. This solution also comes with a caveat.

[KiaraGrouwstra]

@thenbe nice! would you maybe have an example of how to expose the secret thru arion?

[thenbe]

Something like this:

{
  project.name = "nixos container";
  services.webserver = { pkgs, lib, ... }: {
    service.volumes = [ "${toString ./.}/keys.txt:/var/lib/secrets/age" ];
    nixos.useSystemd = true;
    service.useHostStore = true;
    nixos.configuration = { config, lib, options, pkgs, ... }: {
      boot.tmp.useTmpfs = true;

      # 1. setup sops-nix
      imports = [
        inputs.sops-nix.nixosModules.sops
      ];
      sops.defaultSopsFile = ../secrets.yaml;
      sops.defaultSopsFormat = "yaml";
      sops.age.keyFile = "/var/lib/secrets/age";
      sops.useTmpfs = true;

      # 2. use sops-nix
      sops.secrets.MYSECRET = { };
      environment.variables = {
        MYSECRET = config.sops.secrets.MYSECRET.path;
      };

    };
  };
}

[KiaraGrouwstra]

thanks! i'm also looking into docker-compose secrets as mentioned by @Grantimatter, tho it looks like the relevants attributes aren't whitelisted in arion yet. (edit: see #52) the above example may well already address the scope of the OP's question tho.