search

NixOS / nixpkgs

Nix Packages collection & NixOS
MIT License
17.71k stars 13.85k forks source link

services.gitlab-runner: systemd-networkd tries to manage network links #242062

Open datafoo opened 1 year ago

datafoo commented 1 year ago

Describe the bug

When using services.gitlab-runner and a job is processed, many warnings of the style "veth***: Failed to manage SR-IOV PF and VF ports, ignoring: Invalid argument" are logged.

Steps To Reproduce

Steps to reproduce the behavior:

  1. Use services.gitlab-runner and systemd-networkd.
  2. Notice the logs: 
    U Fri Jul  7 13:08:08 2023 p4 systemd-networkd.service systemd-networkd[1741]: vethec40d2a: Failed to manage SR-IOV PF and VF ports, ignoring: Invalid argument

Expected behavior

No warning logs should appear.

Additional context

A similar issue was logged in https://github.com/systemd/systemd/issues/27369 and in https://github.com/moby/moby/issues/45480.

More logs:

[root@mymachine.example.com:~]# journalctl -u systemd-networkd.service -f
Jul 07 13:05:30 mymachine systemd-networkd[1741]: docker0: Link UP
Jul 07 13:07:51 mymachine systemd-networkd[1741]: veth1eb988c: Link UP
Jul 07 13:07:51 mymachine systemd-networkd[1741]: vethdaeb49b: Link UP
Jul 07 13:07:51 mymachine systemd-networkd[1741]: veth7d1cd6c: Link UP
Jul 07 13:07:51 mymachine systemd-networkd[1741]: docker0: Gained carrier
Jul 07 13:07:51 mymachine systemd-networkd[1741]: veth21dcfe7: Link UP
Jul 07 13:07:51 mymachine systemd-networkd[1741]: veth1eb988c: Gained carrier
Jul 07 13:07:51 mymachine systemd-networkd[1741]: veth6d76cb1: Link UP
Jul 07 13:07:51 mymachine systemd-networkd[1741]: veth1eb988c: Lost carrier
Jul 07 13:07:51 mymachine systemd-networkd[1741]: veth1eb988c: Link DOWN
Jul 07 13:07:51 mymachine systemd-networkd[1741]: vethdaeb49b: Gained carrier
Jul 07 13:07:51 mymachine systemd-networkd[1741]: veth21dcfe7: Gained carrier
Jul 07 13:07:51 mymachine systemd-networkd[1741]: veth7d1cd6c: Gained carrier
Jul 07 13:07:51 mymachine systemd-networkd[1741]: veth6d76cb1: Gained carrier
Jul 07 13:07:52 mymachine systemd-networkd[1741]: veth21dcfe7: Lost carrier
Jul 07 13:07:52 mymachine systemd-networkd[1741]: vethdaeb49b: Lost carrier
Jul 07 13:07:52 mymachine systemd-networkd[1741]: veth21dcfe7: Link DOWN
Jul 07 13:07:52 mymachine systemd-networkd[1741]: veth7d1cd6c: Lost carrier
Jul 07 13:07:52 mymachine systemd-networkd[1741]: vethdaeb49b: Link DOWN
Jul 07 13:07:52 mymachine systemd-networkd[1741]: veth6d76cb1: Lost carrier
Jul 07 13:07:52 mymachine systemd-networkd[1741]: veth7d1cd6c: Link DOWN
Jul 07 13:07:52 mymachine systemd-networkd[1741]: veth6d76cb1: Link DOWN
Jul 07 13:07:52 mymachine systemd-networkd[1741]: docker0: Lost carrier
Jul 07 13:07:53 mymachine systemd-networkd[1741]: docker0: Gained IPv6LL
Jul 07 13:07:53 mymachine systemd-networkd[1741]: vethe10ed01: Link UP
Jul 07 13:07:53 mymachine systemd-networkd[1741]: veth43fd027: Link UP
Jul 07 13:07:53 mymachine systemd-networkd[1741]: vethe40fa77: Link UP
Jul 07 13:07:53 mymachine systemd-networkd[1741]: docker0: Gained carrier
Jul 07 13:07:53 mymachine systemd-networkd[1741]: vethf0282a6: Link UP
Jul 07 13:07:53 mymachine systemd-networkd[1741]: vethe10ed01: Gained carrier
Jul 07 13:07:53 mymachine systemd-networkd[1741]: veth35a3023: Link UP
Jul 07 13:07:53 mymachine systemd-networkd[1741]: veth43fd027: Gained carrier
Jul 07 13:07:53 mymachine systemd-networkd[1741]: vethe10ed01: Lost carrier
Jul 07 13:07:53 mymachine systemd-networkd[1741]: veth43fd027: Lost carrier
Jul 07 13:07:54 mymachine systemd-networkd[1741]: vethe10ed01: Link DOWN
Jul 07 13:07:54 mymachine systemd-networkd[1741]: veth43fd027: Link DOWN
Jul 07 13:07:54 mymachine systemd-networkd[1741]: vethf0282a6: Gained carrier
Jul 07 13:07:54 mymachine systemd-networkd[1741]: vethe40fa77: Gained carrier
Jul 07 13:07:54 mymachine systemd-networkd[1741]: veth35a3023: Gained carrier
Jul 07 13:07:54 mymachine systemd-networkd[1741]: vethf0282a6: Lost carrier
Jul 07 13:07:54 mymachine systemd-networkd[1741]: vethe40fa77: Lost carrier
Jul 07 13:07:54 mymachine systemd-networkd[1741]: veth35a3023: Lost carrier
Jul 07 13:07:54 mymachine systemd-networkd[1741]: vethf0282a6: Link DOWN
Jul 07 13:07:54 mymachine systemd-networkd[1741]: veth35a3023: Link DOWN
Jul 07 13:07:54 mymachine systemd-networkd[1741]: vethe40fa77: Link DOWN
Jul 07 13:07:54 mymachine systemd-networkd[1741]: veth519ce91: Link UP
Jul 07 13:07:54 mymachine systemd-networkd[1741]: vethf7e8c80: Link UP
Jul 07 13:07:54 mymachine systemd-networkd[1741]: vetha0a0e30: Link UP
Jul 07 13:07:54 mymachine systemd-networkd[1741]: veth7979366: Link UP
Jul 07 13:07:54 mymachine systemd-networkd[1741]: veth519ce91: Gained carrier
Jul 07 13:07:54 mymachine systemd-networkd[1741]: veth98fe183: Link UP
Jul 07 13:07:54 mymachine systemd-networkd[1741]: vethf7e8c80: Gained carrier
Jul 07 13:07:54 mymachine systemd-networkd[1741]: vetha0a0e30: Gained carrier
Jul 07 13:07:54 mymachine systemd-networkd[1741]: veth98fe183: Gained carrier
Jul 07 13:07:55 mymachine systemd-networkd[1741]: veth7979366: Gained carrier
Jul 07 13:07:55 mymachine systemd-networkd[1741]: veth519ce91: Gained IPv6LL
Jul 07 13:07:56 mymachine systemd-networkd[1741]: veth7979366: Gained IPv6LL
Jul 07 13:07:56 mymachine systemd-networkd[1741]: veth98fe183: Gained IPv6LL
Jul 07 13:07:56 mymachine systemd-networkd[1741]: vethf7e8c80: Gained IPv6LL
Jul 07 13:07:56 mymachine systemd-networkd[1741]: vetha0a0e30: Gained IPv6LL
Jul 07 13:08:00 mymachine systemd-networkd[1741]: vetha0a0e30: Lost carrier
Jul 07 13:08:00 mymachine systemd-networkd[1741]: vetha0a0e30: Link DOWN
Jul 07 13:08:00 mymachine systemd-networkd[1741]: veth4607ce1: Link UP
Jul 07 13:08:00 mymachine systemd-networkd[1741]: veth4607ce1: Gained carrier
Jul 07 13:08:00 mymachine systemd-networkd[1741]: veth4607ce1: Lost carrier
Jul 07 13:08:00 mymachine systemd-networkd[1741]: vethf7e8c80: Lost carrier
Jul 07 13:08:00 mymachine systemd-networkd[1741]: veth4607ce1: Link DOWN
Jul 07 13:08:00 mymachine systemd-networkd[1741]: vethf7e8c80: Link DOWN
Jul 07 13:08:00 mymachine systemd-networkd[1741]: veth519ce91: Lost carrier
Jul 07 13:08:00 mymachine systemd-networkd[1741]: veth519ce91: Link DOWN
Jul 07 13:08:00 mymachine systemd-networkd[1741]: vethd01dc37: Link UP
Jul 07 13:08:00 mymachine systemd-networkd[1741]: veth7979366: Lost carrier
Jul 07 13:08:00 mymachine systemd-networkd[1741]: veth7979366: Link DOWN
Jul 07 13:08:00 mymachine systemd-networkd[1741]: veth98fe183: Lost carrier
Jul 07 13:08:00 mymachine systemd-networkd[1741]: veth98fe183: Link DOWN
Jul 07 13:08:00 mymachine systemd-networkd[1741]: vethc8dd8f4: Link UP
Jul 07 13:08:00 mymachine systemd-networkd[1741]: vethc442ec7: Link UP
Jul 07 13:08:00 mymachine systemd-networkd[1741]: vethd01dc37: Gained carrier
Jul 07 13:08:00 mymachine systemd-networkd[1741]: vethd01dc37: Lost carrier
Jul 07 13:08:01 mymachine systemd-networkd[1741]: docker0: Lost carrier
Jul 07 13:08:01 mymachine systemd-networkd[1741]: vethd01dc37: Link DOWN
Jul 07 13:08:01 mymachine systemd-networkd[1741]: vethb160df9: Link UP
Jul 07 13:08:01 mymachine systemd-networkd[1741]: veth230982a: Link UP
Jul 07 13:08:01 mymachine systemd-networkd[1741]: docker0: Gained carrier
Jul 07 13:08:01 mymachine systemd-networkd[1741]: vethc8dd8f4: Gained carrier
Jul 07 13:08:01 mymachine systemd-networkd[1741]: vethc442ec7: Gained carrier
Jul 07 13:08:01 mymachine systemd-networkd[1741]: vethb160df9: Gained carrier
Jul 07 13:08:01 mymachine systemd-networkd[1741]: veth230982a: Gained carrier
Jul 07 13:08:01 mymachine systemd-networkd[1741]: vethc442ec7: Lost carrier
Jul 07 13:08:01 mymachine systemd-networkd[1741]: vethc442ec7: Link DOWN
Jul 07 13:08:01 mymachine systemd-networkd[1741]: veth6eaba2f: Link UP
Jul 07 13:08:01 mymachine systemd-networkd[1741]: veth6eaba2f: Gained carrier
Jul 07 13:08:02 mymachine systemd-networkd[1741]: vethc8dd8f4: Gained IPv6LL
Jul 07 13:08:02 mymachine systemd-networkd[1741]: veth6eaba2f: Gained IPv6LL
Jul 07 13:08:03 mymachine systemd-networkd[1741]: vethb160df9: Gained IPv6LL
Jul 07 13:08:03 mymachine systemd-networkd[1741]: veth230982a: Gained IPv6LL
Jul 07 13:08:08 mymachine systemd-networkd[1741]: vethb160df9: Lost carrier
Jul 07 13:08:08 mymachine systemd-networkd[1741]: vethb160df9: Link DOWN
Jul 07 13:08:08 mymachine systemd-networkd[1741]: veth380bc77: Link UP
Jul 07 13:08:08 mymachine systemd-networkd[1741]: vetha3749a6: Link UP
Jul 07 13:08:08 mymachine systemd-networkd[1741]: veth380bc77: Gained carrier
Jul 07 13:08:08 mymachine systemd-networkd[1741]: vetha3749a6: Gained carrier
Jul 07 13:08:08 mymachine systemd-networkd[1741]: vetha3749a6: Lost carrier
Jul 07 13:08:08 mymachine systemd-networkd[1741]: vetha3749a6: Link DOWN
Jul 07 13:08:08 mymachine systemd-networkd[1741]: vethec40d2a: Failed to manage SR-IOV PF and VF ports, ignoring: Invalid argument
Jul 07 13:08:10 mymachine systemd-networkd[1741]: veth380bc77: Gained IPv6LL
^C

[root@mymachine.example.com:~]# networkctl 
IDX LINK        TYPE      OPERATIONAL SETUP     
  1 lo          loopback  carrier     unmanaged
  2 eno2        ether     routable    configured
  3 wg0         wireguard routable    configured
  4 wg1         wireguard routable    configured
  5 wlo1        wlan      off         unmanaged
  6 docker0     bridge    routable    unmanaged
 42 vethc8dd8f4 ether     enslaved    unmanaged
 48 veth230982a ether     enslaved    unmanaged
 50 veth6eaba2f ether     enslaved    unmanaged
 52 veth380bc77 ether     enslaved    unmanaged

10 links listed.

Notify maintainers

@bachp

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.1.37, NixOS, 23.05 (Stoat)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.13.3`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`
bachp commented 1 year ago

@datafoo I assume you are using the docker executor.

I would assume this warning is also produced when you start a container from the docker cli.

Could you verify this.

datafoo commented 1 year ago

I forgot to indicate that the issue is not appearing all the time I run my CI pipelines. It comes randomly.

I tried starting the a container from the cli but everything seems fine. The logs starting from "Jul 12 07:15:30" are for the manually started container.

[root@mymachine.example.com:~]# docker run alpine

[root@mymachine.example.com:~]# journalctl -u systemd-networkd.service -f
Jul 12 07:10:44 mymachine systemd-networkd[1739]: veth36bbb91: Link DOWN
Jul 12 07:10:44 mymachine systemd-networkd[1739]: veth20f9be7: Lost carrier
Jul 12 07:10:44 mymachine systemd-networkd[1739]: veth20f9be7: Link DOWN
Jul 12 07:10:45 mymachine systemd-networkd[1739]: docker0: Lost carrier
Jul 12 07:15:30 mymachine systemd-networkd[1739]: veth26607e9: Link UP
Jul 12 07:15:31 mymachine systemd-networkd[1739]: veth26607e9: Gained carrier
Jul 12 07:15:31 mymachine systemd-networkd[1739]: docker0: Gained carrier
Jul 12 07:15:31 mymachine systemd-networkd[1739]: veth26607e9: Lost carrier
Jul 12 07:15:31 mymachine systemd-networkd[1739]: veth26607e9: Link DOWN
Jul 12 07:15:31 mymachine systemd-networkd[1739]: docker0: Lost carrier