nixos/gns3-server: ubridge doesn't work

dani0854 commented 6 months ago

Describe the bug

When I try to start docker appliences I get the following error:

uBridge requires root access or the capability to interact with network adapters

I viewed module source, and noticied that ubridge_path is set to /nix/store/... package, rather than /run/wrappers/bin/ubridge which gives those capabilities.

I have created PR (#292095) to fix this. But then run into another problem, it finds the wrapper, but fails when it tries to verify version with an error.

Could not determine uBridge version for /run/wrappers/bin/ubridge

After few patches to gns3-server code, I got a more informative error:

Could not determine uBridge version, error: failed to inherit capabilities: Operation not permitted

I have searched for this error, and it originates from nixos/modules/security/wrappers/wrapper.c

Unfortunatly at this point of diagnostic my knowledge is lacking with respect to capabilites, and C.

Also gns3-server package works fine when started as a user with ubridge group (with the wrapper in ubridge_path). It looks like it has something to do with systemd, systemd service configuration or hardening. I tried commenting out the later, but it didn't change much.

Steps To Reproduce

Steps to reproduce the behavior:

Enable services.gns3-server and services.gns3-server.ubridge
Create any Docker applience
Try to start Docker applience

Expected behavior

Docker applience start without errors

Notify maintainers

@anthonyroussel

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.1.79, NixOS, 24.05 (Uakari), 24.05.20240226.13aff9b`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.18.1`
 - channels(dsuetin): `""`
 - nixpkgs: `/nix/store/1gpgihsi21bni03hy6i5ccw9lfrhnbqi-source`

Add a :+1: reaction to issues you find important.

dani0854 commented 6 months ago

Looks like it has something to do with systemd dynamic users, created gns3 user normally, and the wrapper seems to be working.

However, there is another error now:

Could not start auxiliary console process: [Errno 2] No such file or directory: 'script'

Not sure if related to dynamic user removal, or unrelated to above yet.

cutemagi commented 5 months ago

Any update on this?

dani0854 commented 5 months ago

Any update on this?

I haven't done anything further on this issue. I was hopping a maintainer or someone else with more knowledge in this area would step in.

If I have spare time, I might look into it deeper.

anthonyroussel commented 5 months ago

Hello, and sorry for the late reply.

I started investigating the issue, looks like it is related to the use of DynamicUser in the gns3-server SystemD service which produces this error:

error: failed to inherit capabilities: Operation not permitted

DynamicUser is a rather complex setting and its use with SUID wrappers (security.wrappers) seems to be incompatible.

Indeed, the SystemD documentation states that DynamicUser enabled implies NoNewPrivileges enabled:

DynamicUser: [...] NoNewPrivileges= and RestrictSUIDSGID= are implicitly enabled (and cannot be disabled), to ensure that processes invoked cannot take benefit or create SUID/SGID files or directories

And the NoNewPrivileges argument cannot be disabled (in the case DynamicUser is used), and it clearly forbids the service process to elevate privileges, and so to run ubridge SUID wrapper:

ensures that the service process and all its children can never gain new privileges through execve() (e.g. via setuid or setgid bits, or filesystem capabilities). This is the simplest and most effective way to ensure that a process and its children can never elevate privileges again

As a result, I think we should remove this DynamicUser setting in the NixOS module. Because after thinking about it, we should not use DynamicUser when we have a complex setup and even in the case of stateful services like GNS3 Server.

anthonyroussel commented 5 months ago

I did the PR to remove SystemD hardenings: https://github.com/NixOS/nixpkgs/pull/303442

I have tested it on my workstation and it fixes the problems with ubridge on my workstation:

base_node.py:701 Starting new uBridge hypervisor 0.0.0.0:36177
hypervisor.py:169 starting ubridge: ['/run/wrappers/bin/ubridge', '-H', '0.0.0.0:36177']
hypervisor.py:179 ubridge started PID=292152
base_node.py:704 Hypervisor 0.0.0.0:36177 has successfully started
ubridge_hypervisor.py:83 Connected to uBridge hypervisor on 127.0.0.1:36177 after 0.1015 seconds

I would need help from someone to review the PR and also test the patch, please :pray:

NixOS / nixpkgs