403 when pulling from private registry

[sh54]

If I use an image from a private registry Arion does not seem able to retrieve them. I get a 403 Forbidden.

Error from service

❯ systemctl status arion-arion-example.service
× arion-arion-example.service
     Loaded: loaded (/etc/systemd/system/arion-arion-example.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Sat 2024-07-27 09:25:53 CEST; 1min 24s ago
   Duration: 682ms
    Process: 31983 ExecStart=/nix/store/himnabmjq418nbd0yqkjq6cqiakrqzxj-unit-script-arion-arion-example-start/bin/arion-arion-example-start (code=exited, status=1/FAILURE)
   Main PID: 31983 (code=exited, status=1/FAILURE)
         IP: 0B in, 0B out
        CPU: 112ms

Jul 27 09:25:52 eids-2024 systemd[1]: Started arion-arion-example.service.
Jul 27 09:25:52 eids-2024 arion-arion-example-start[31983]: docker compose file: /nix/store/0c3xkg7nn3pvc7gnj8dlv9nkmdh6myh8-docker-compose.yaml
Jul 27 09:25:53 eids-2024 arion-arion-example-start[32039]: time="2024-07-27T09:25:53+02:00" level=warning msg="/nix/store/0c3xkg7nn3pvc7gnj8dlv9nkmdh6myh8-docker-compose.yaml: the attribute `version` is obsolete, it will be ignored, please rem>
Jul 27 09:25:53 eids-2024 arion-arion-example-start[32039]:  test-website Pulling
Jul 27 09:25:53 eids-2024 arion-arion-example-start[32039]:  test-website Error {"message":"Requesting bearer token: invalid status code from registry 403 (Forbidden)"}
Jul 27 09:25:53 eids-2024 arion-arion-example-start[32039]: Error response from daemon: {"message":"Requesting bearer token: invalid status code from registry 403 (Forbidden)"}
Jul 27 09:25:53 eids-2024 arion-arion-example-start[31984]: arion: FatalError {fatalErrorMessage = "docker-compose failed with ExitFailure 18"}
Jul 27 09:25:53 eids-2024 systemd[1]: arion-arion-example.service: Main process exited, code=exited, status=1/FAILURE
Jul 27 09:25:53 eids-2024 systemd[1]: arion-arion-example.service: Failed with result 'exit-code'.

Example configuration

  virtualisation.arion.backend = "podman-socket";
  virtualisation.arion.projects.arion-example.settings = {
    project.name = "my-services";
    services = {
      test-website = {
        service = {
          image = "my.private.registry/example/website:0.1.10";
        };
      };
    };
  };

More context

I have my own gitlab instance with a private container registry. This has worked fine when using regular docker compose.

I am logged in to my registry:

❯ sudo podman login my.private.registry
[sudo] password for sh54:
Authenticating with existing credentials for my.private.registry
Existing credentials are valid. Already logged in to my.private.registry

Interacting with stuff in my registry via plain podman works just fine. Things fall apart once arion gets involved.

I also have a separate systemd service defined that logs into my container registry.

Workaround

If I manually pull my image then doing a nixos-rebuild or restarting the relevant arion service gets things working as it should do.

e.g.:

sudo podman pull my.private.registry/example/website:0.1.10
sudo systemctl start arion-arion-example.service

brings things online correctly. This is very annoying though.

[roberth]

sudo and a systemd service are quite different in terms of their environment. The latter runs with few environment variables and generally less access to stuff, whereas a command that runs via sudo can use SUDO_USER to figure out where your user config is and use that for authentication. I don't know whether that's exactly what's happening, but either way, you'll have to configure the systemd service to have access to registry credentials, because by default it doesn't know how to authenticate with any private registries. I haven't needed this, but I'd welcome additions to the docs and/or NixOS module.