oauth2_proxy systemd service needs dependency on network-online.target

PhilippWoelfel commented 1 year ago

Describe the bug

When booting my server, oauth2_proxy.service fails with the following error message:

ERROR: Failed to initialise OAuth2 Proxy: error intiailising provider: could not create provider data: error building OIDC ProviderVerifier: could not get verifier builder: error while discovery OIDC configuration: failed to discover OIDC configuration: error performing request: Get "https://<removed_for_privacy>.../.well-known/openid-configuration": dial tcp 10.0.9.12:443: connect: network is unreachable

When starting the service on a running system, the failure does not occur.

The problems seems to be that the systemd service does not require network-online.target. At least adding the following lines to the nixos configuration fixes the problem:

systemd.services."oauth2_proxy" = {
  wants = [ "network-online.target" ];
  after = [ "network-online.target" ];
};

Note that the original systemd service created by the module has after = [ "network.target" ]. But this seems to be the wrong target, according to https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget:

network.target has very little meaning during start-up. [...]

Notify maintainers

@yorickvP @knl @joachifm @zowoq @rimmington @ncfavier @mkaito @kamilchm @jml @dasJ @infinisil @JohnAZoidberg

Metadata

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
- system: `"x86_64-linux"`
- host os: `Linux 5.15.63, NixOS, 22.05 (Quokka)`
- multi-user?: `yes`
- sandbox: `yes`
- version: `nix-env (Nix) 2.8.1`
- nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

dasJ commented 1 year ago

I don't think I maintain oauth2_proxy

PhilippWoelfel commented 1 year ago

I don't think I maintain oauth2_proxy

Apologies. The module lists no maintainers, so I pinged contributors.

ncfavier commented 1 year ago

Me neither. Please be careful when looking at the history for a file: unless a commit message explicitly starts with oauth2_proxy or nixos/oauth2_proxy, it is probably a treewide change that happened to touch the file you're looking at.

knl commented 1 year ago

@PhilippWoelfel I think you're right, it should be network-online.target. Could you create a PR with your changes?

PhilippWoelfel commented 1 year ago

In theory, adding network-online.target to after= and wants= seems to be the right solution. At least that's my understanding from the documentation, and it fixes the network is unreachable error message from my first post.

But it's difficult to test. On my own server, oauth2_proxy now fails most of the time with

ERROR: Failed to initialise OAuth2 Proxy: error intiailising provider: could not create provider data: error building OIDC ProviderVerifier: could not get verifier builder: error while discovery OIDC configuration: failed to discover OIDC configuration: unexpected status "502": <html>

I suspect that's because it's trying to get OIDC information from my Keycloak server, which is not fully available at that time. Unfortunately, the problem even occurs if I add keycloak.service to after= and wants=.

I don't mind creating a PR that adds network-online.target to after= and wants=, but are we sure it's the right solution?

rimmington commented 1 year ago

Moving to network-online.target seems reasonable.

I suspect that's because it's trying to get OIDC information from my Keycloak server, which is not fully available at that time. Unfortunately, the problem even occurs if I add keycloak.service to after= and wants=.

This sounds like a separate issue: the Keycloak service is considered ready before the server is available. This is a fairly common issue; perhaps look into socket activation or the systemd readiness protocol.

benley commented 1 month ago

I wired up the systemd readiness protocol for keycloak a while back, but it hasn't been merged. Needs a reviewer. https://github.com/NixOS/nixpkgs/pull/250638

NixOS / nixpkgs