Systemd Unit Fails to Read Config

Grissess commented 1 month ago

Current Behavior

Using systemctl start openvpn-auth-oauth2 on the provided .deb package on Ubuntu 24.04, the following logs appear in systemctl status openvpn-auth-oauth2:

Jul 11 15:53:42 vpn-host systemd[1]: Started openvpn-auth-oauth2.service - OpenVPN authenticator.
Jul 11 15:53:42 vpn-host openvpn-auth-oauth2[345675]: time=2024-07-11T15:53:42.733Z level=ERROR msg="error loading config: validation error: oauth2.client.id is required"

Starting the service manually as nobody with the appropriate supplemental group works, however:

nobody@vpn-host:/$ openvpn-auth-oauth2 --config /etc/openvpn-auth-oauth2/config.yaml
time=2024-07-11T16:06:37.408Z level=INFO msg="discover oidc auto configuration with provider generic for issuer https://login.microsoftonline.com/[REDACTED]/v2.0"
time=2024-07-11T16:06:37.702Z level=INFO msg="start HTTP server listener on :9000"
time=2024-07-11T16:06:37.702Z level=INFO msg="connect to openvpn management interface unix:///run/openvpn.sock"
time=2024-07-11T16:06:37.703Z level=INFO msg="openvpn-auth-oauth2 started with base url https://vpn-host.[REDACTED]:443"
time=2024-07-11T16:06:37.703Z level=INFO msg="connection to OpenVPN management interface established."
time=2024-07-11T16:06:37.704Z level=INFO msg="OpenVPN Version: OpenVPN 2.6.9 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] [DCO] - Management Version: 5"

Despite the failure to load config, it is opening the file and parsing it correctly, as different errors occur when the file does not exist:

Jul 11 15:53:07 vpn-host openvpn-auth-oauth2[345589]: time=2024-07-11T15:53:07.222Z level=ERROR msg="error loading config: file provider: open /etc/openvpn-auth-oauth2/test.yaml: no such file or directory"

or when the file is not syntactically-valid YAML:

Jul 11 16:01:30 vpn-host openvpn-auth-oauth2[346569]: time=2024-07-11T16:01:30.130Z level=ERROR msg="error loading config: file provider: yaml: did not find expected comment or line break"

Expected Behavior

The systemd unit should start and run successfully.

Steps To Reproduce

The instructions are a Chef cookbook, but to approximate:

1. On AWS, provision a Ubuntu 24.04 host with necessary resources (e.g. a login with `sudo`), and shell in;
2. `sudo apt install openvpn`;
3. `wget https://github.com/jkroepke/openvpn-auth-oauth2/releases/download/v1.21.4/openvpn-auth-oauth2_1.21.4_linux_amd64.deb` and `sudo dpkg -i openvpn-auth-oauth2_1.21.4_linux_amd64.deb`;
4. Configure [as per the wiki](https://github.com/jkroepke/openvpn-auth-oauth2/wiki/Configuration) using the YAML config file in the unit's recommended location of `/etc/openvpn-auth-oauth2/config.yaml`, and set the appropriate group/mode (note some minor variation, such as `/run/openvpn.sock`--this doesn't seem material to this bug, however);
5. `sudo systemctl start openvpn` (and validate a successful config);
6. `sudo systemctl start openvpn-auth-oauth2`.

Environment

openvpn-auth-oauth2 Version:

version: 1.21.4
commit: 8c8a89455ee81c90bd6236124617cc202d05f065
date: 2024-07-01T20:14:23Z
go: go1.22.4

OpenVPN Server Version:

OpenVPN 2.6.9 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] [DCO]
library versions: OpenSSL 3.0.13 30 Jan 2024, LZO 2.10
DCO version: N/A
Originally developed by James Yonan
Copyright (C) 2002-2023 OpenVPN Inc <sales@openvpn.net>
Compile time defines: enable_async_push=no enable_comp_stub=no enable_crypto_ofb_cfb=yes enable_dco=yes enable_dco_arg=yes enable_debug=yes enable_dependency_tracking=no enable_dlopen=unknown enable_dlopen_self=unknown enable_dlopen_self_static=unknown enable_fast_install=needless enable_fragment=yes enable_iproute2=no enable_libtool_lock=yes enable_lz4=yes enable_lzo=yes enable_maintainer_mode=no enable_management=yes enable_option_checking=no enable_pam_dlopen=no enable_pedantic=no enable_pkcs11=yes enable_plugin_auth_pam=yes enable_plugin_down_root=yes enable_plugins=yes enable_port_share=yes enable_selinux=no enable_shared=yes enable_shared_with_static_runtimes=no enable_silent_rules=no enable_small=no enable_static=yes enable_strict=no enable_strict_options=no enable_systemd=yes enable_unit_tests=no enable_werror=no enable_win32_dll=yes enable_wolfssl_options_h=yes enable_x509_alt_username=yes with_aix_soname=aix with_crypto_library=openssl with_gnu_ld=yes with_mem_check=no with_openssl_engine=auto with_sysroot=no

Server OS:

Distributor ID: Ubuntu
Description:    Ubuntu 24.04 LTS
Release:    24.04
Codename:   noble
Linux vpn-host 6.8.0-1009-aws #9-Ubuntu SMP Fri May 17 14:39:23 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

OpenVPN Client (flavor, OS): various
OIDC Provider: Entra ID

openvpn-auth-oauth2 logs

Jul 11 15:53:30 vpn-host systemd[1]: Started openvpn-auth-oauth2.service - OpenVPN authenticator.
Jul 11 15:53:30 vpn-host openvpn-auth-oauth2[345640]: time=2024-07-11T15:53:30.193Z level=ERROR msg="error loading config: validation error: oauth2.client.id is required"
Jul 11 15:53:30 vpn-host systemd[1]: openvpn-auth-oauth2.service: Main process exited, code=exited, status=1/FAILURE
Jul 11 15:53:30 vpn-host systemd[1]: openvpn-auth-oauth2.service: Failed with result 'exit-code'.

openvpn server logs

(For some reason, OpenVPN doesn't appear to be logging to the usual location. Nevertheless, I can confirm a full end-to-end test works with my OpenVPN client when `openvpn-auth-oauth2` is running as root.)

Anything else?

Let me know if you need any further information :)

jkroepke commented 1 month ago

Let me know if you need any further information :)

Any results, if you run

dmesg | grep audit | grep openvpn-auth-oauth2

as root?

Grissess commented 1 month ago

Let me know if you need any further information :)

Any results, if you run

dmesg | grep audit | grep openvpn-auth-oauth2

as root?

Exactly nothing. I forgot to mention, as part of minor config variance, that I did install an apparmor local policy, /etc/apparmor.d/local/usr.bin.openvpn-auth-oauth2:

/etc/openvpn/pki/* r,
/run/openvpn.sock rw,

(For TLS certificates in the testing config and the OpenVPN socket itself. The former is actually no longer required, and I might evict it soon.)

Edit: I should note that I also saw no audit logs for its failure to connect to the socket directly; a random StackOverflow answer induced me to check apparmor, where I found that installed policy. Prior to that, it was occasionally, randomly, racing with an error opening /run/openvpn.sock; now it only fails in the described manner.

jkroepke commented 1 month ago

apparmor local policy

should the /etc/openvpn-auth-oauth2 path not included?

Grissess commented 1 month ago

apparmor local policy

should the /etc/openvpn-auth-oauth2 path not included?

Oh, that's included from the base policy:

root@vpn-host:~# cat /etc/apparmor.d/usr.bin.openvpn-auth-oauth2
# vim:syntax=apparmor

abi <abi/3.0>,

include <tunables/global>

/usr/bin/openvpn-auth-oauth2 flags=(attach_disconnected) {
  include <abstractions/base>
  include <abstractions/nameservice>

  @{etc_ro}/openvpn-auth-oauth2/** r,

  @{run}/openvpn{,-server}/* rw,

  @{sys}/kernel/mm/{hugepages,transparent_hugepage}/{,**} r,

  # Site-specific additions and overrides. See local/README for details.
  include if exists <local/usr.bin.openvpn-auth-oauth2>
}

I'm assuming that was installed by the package, and does include @{etc_ro}/openvpn-auth-oauth2/** r :)

That's indeed necessary--per recommendation, I broke out some of the secrets into restrictively-moded files in /etc/openvpn-auth-oauth2, and refer to them using file:// URLs (with absolute paths).

jkroepke commented 1 month ago

I guess the issue must be in some of the over-engineered systemd properties that I had enabled. systemd uses namespaces and secomp to restrict file access.

Maybe something in 24.04 works differently.

Grissess commented 1 month ago

I'll happily await some tests; if you need this system to test some theories/fixes, I proffer it :)

Good luck!

jkroepke commented 1 month ago

I'll happily await some tests; if you need this system to test some theories/fixes, I proffer it :)

Good luck!

While I have access to free resources at Azure (e.g. VMs), I'm in vacation at least for one week until next saturday.

Grissess commented 1 month ago

no worries, then, take your time :)

In the meantime, these servers are merely running the daemon as an unprivileged user in a tmux session--it'll hold out for a little while.

jkroepke commented 1 month ago

Guess comment this lines out

https://github.com/jkroepke/openvpn-auth-oauth2/blob/8c8a89455ee81c90bd6236124617cc202d05f065/packaging/usr/lib/systemd/system/openvpn-auth-oauth2.service#L34-L62

should also resolve the issue for now

Grissess commented 1 month ago

I actually tracked this down to an issue unrelated to permissions at all: it's the EnvironmentFile shipped with the project :)

# This file is sourced by the openvpn-auth-oauth2.service

# CONFIG_FILE is the path to the configuration file and used in the systemd service file only.
CONFIG_FILE=/etc/openvpn-auth-oauth2/config.yaml

# https://github.com/jkroepke/openvpn-auth-oauth2/wiki/Configuration

CONFIG_OPENVPN_ADDR=unix:///run/openvpn/server.sock
CONFIG_OAUTH2_ISSUER=https://login.microsoftonline.com/<tenant-id>/v2.0
CONFIG_OAUTH2_CLIENT_ID=
CONFIG_OAUTH2_CLIENT_SECRET=
CONFIG_HTTP_LISTEN=:9000
# Define a random value with 16 or 24 characters
CONFIG_HTTP_SECRET=
# Define the public http endpoint here.
CONFIG_HTTP_BASEURL=http://localhost:9000

Note that env vars are loaded later than config, meaning the former overrides the latter. That means this example skeleton config, with invalid values, unexpectedly overrides the user's config.yaml in the default configuration. (As I wasn't about to use env vars, I didn't even bother to check this file.)

The easy workaround, of course, is to just comment out the lines--and it might be preferable if upstream did so, both so they're there as examples that most users will use, and to avoid this precise footgun :)

After doing so, I tested the systemd unit without any systemd-level overrides, and it seems fine.

jkroepke commented 1 month ago

The easy workaround, of course, is to just comment out the lines

You are right, this should be comment out by default to avoid this scenario. Are you interested in a PR?

jkroepke / openvpn-auth-oauth2