search

flatcar / Flatcar

Flatcar project repository for issue tracking, project documentation, etc.
https://www.flatcar.org/
Apache License 2.0
774 stars 32 forks source link

Flatcar with container openvpn-client not wokring properly #1515

Open lmq1999 opened 3 months ago

lmq1999 commented 3 months ago

Description

Flatcar with openvpn client container not wokring

Impact

Unable to use flatcar with openvpn

Environment and steps to reproduce

  1. Set-up:
    Flatcar image: flatcar_production_openstack_image.img
pool-g4dzrku5-sj3dtqihuu6cjof6-node-hshs3fbx ~ # cat /etc/os-release
NAME="Flatcar Container Linux by Kinvolk"
ID=flatcar
ID_LIKE=coreos
VERSION=3975.2.0
VERSION_ID=3975.2.0
BUILD_ID=2024-08-05-2103
SYSEXT_LEVEL=1.0
PRETTY_NAME="Flatcar Container Linux by Kinvolk 3975.2.0 (Oklo)"
ANSI_COLOR="38;5;75"
HOME_URL="https://flatcar.org/"
BUG_REPORT_URL="https://issues.flatcar.org"
FLATCAR_BOARD="amd64-usr"
CPE_NAME="cpe:2.3:o:flatcar-linux:flatcar_linux:3975.2.0:*:*:*:*:*:*:*"
  1. Task: Running Openvpn-client container (I have tried a lot docker openvpn-client on github and build one myself)

2.1: Install docker-compose

curl -SL https://github.com/docker/compose/releases/download/v2.29.1/docker-compose-linux-x86_64 -o /opt/bin/docker-compose
chmod +x /opt/bin/docker-compose

2.2: Write docker-compose VPN file

version: "3.3"
services:
  vpn:
    image: cr-hn-1.bizflycloud.vn/31ff9581861a4d0ea4df5e7dda0f665d/openvpn-client@sha256:2defe3062e65ad0ecf43bc747d60d1274ebbf7a45100c5cf5448fd7114caac80
    volumes:
      - /etc/openvpn/kengine.conf:/vpn/kengine.conf
      - /dev/net/tun:/dev/net/tun
    cap_add:
    - NET_ADMIN
    restart: always
    network_mode: "host"

2.3: Write openvpn-client file

client
dev kengine
dev-type tap
reneg-sec 0
proto tcp-client
remote xxx.xx.xx.xx xxxxx
resolv-retry infinite
nobind
<ca>
-----BEGIN CERTIFICATE-----
.........
-----END CERTIFICATE-----

</ca>
<key>
-----BEGIN PRIVATE KEY-----
..........
-----END PRIVATE KEY-----
</key>
<cert>
-----BEGIN CERTIFICATE-----
......
-----END CERTIFICATE-----

</cert>
<tls-auth>
#
# 2048 bit OpenVPN static key
#
-----BEGIN OpenVPN Static key V1-----
...........
-----END OpenVPN Static key V1-----

</tls-auth>
remote-cert-tls server
key-direction 1
script-security 3
keepalive 10 60
persist-key
persist-tun
comp-lzo
verb 3
  1. Action(s): a. Run the docker-compose 
    pool-g4dzrku5-sj3dtqihuu6cjof6-node-hshs3fbx ~ # docker-compose -f docker-compose-2.yaml up
WARN[0000] /root/docker-compose-2.yaml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion 
WARN[0000] Found orphan containers ([openvpn-client root-openvpn-1]) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up. 
[+] Running 1/0
✔ Container root-vpn-1  Created                                                                                                                                                                                                                                                                                                                                                       0.0s 
Attaching to vpn-1
vpn-1  | + exec sg vpn -c 'openvpn --cd /vpn --config /vpn/kengine.conf --script-security 2 --redirect-gateway def1                 '
vpn-1  | Mon Aug 12 08:30:58 2024 OpenVPN 2.4.9 x86_64-alpine-linux-musl [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD] built on Apr 20 2020
vpn-1  | Mon Aug 12 08:30:58 2024 library versions: OpenSSL 1.1.1g  21 Apr 2020, LZO 2.10
vpn-1  | Mon Aug 12 08:30:58 2024 Outgoing Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
vpn-1  | Mon Aug 12 08:30:58 2024 Incoming Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
vpn-1  | Mon Aug 12 08:30:58 2024 TCP/UDP: Preserving recently used remote address: [AF_INET]123.31.11.151:10001
vpn-1  | Mon Aug 12 08:30:58 2024 Socket Buffers: R=[131072->131072] S=[16384->16384]
vpn-1  | Mon Aug 12 08:30:58 2024 Attempting to establish TCP connection with [AF_INET]123.31.11.151:10001 [nonblock]
vpn-1  | Mon Aug 12 08:30:59 2024 TCP connection established with [AF_INET]123.31.11.151:10001
vpn-1  | Mon Aug 12 08:30:59 2024 TCP_CLIENT link local: (not bound)
vpn-1  | Mon Aug 12 08:30:59 2024 TCP_CLIENT link remote: [AF_INET]123.31.11.151:10001
vpn-1  | Mon Aug 12 08:30:59 2024 TLS: Initial packet from [AF_INET]123.31.11.151:10001, sid=b30c806f 6b1db9ce
vpn-1  | Mon Aug 12 08:30:59 2024 VERIFY OK: depth=1, CN=bke-vpn
vpn-1  | Mon Aug 12 08:30:59 2024 VERIFY KU OK
vpn-1  | Mon Aug 12 08:30:59 2024 Validating certificate extended key usage
vpn-1  | Mon Aug 12 08:30:59 2024 ++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication
vpn-1  | Mon Aug 12 08:30:59 2024 VERIFY EKU OK
vpn-1  | Mon Aug 12 08:30:59 2024 VERIFY OK: depth=0, CN=bke-vpn
vpn-1  | Mon Aug 12 08:30:59 2024 Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, 2048 bit RSA
vpn-1  | Mon Aug 12 08:30:59 2024 [bke-vpn] Peer Connection Initiated with [AF_INET]123.31.11.151:10001
vpn-1  | Mon Aug 12 08:31:00 2024 SENT CONTROL [bke-vpn]: 'PUSH_REQUEST' (status=1)
vpn-1  | Mon Aug 12 08:31:00 2024 PUSH: Received control message: 'PUSH_REPLY,route-gateway 10.99.0.1,ping 20,ping-restart 60,ifconfig 10.99.0.4 255.255.240.0,peer-id 0,cipher AES-256-GCM'
vpn-1  | Mon Aug 12 08:31:00 2024 OPTIONS IMPORT: timers and/or timeouts modified
vpn-1  | Mon Aug 12 08:31:00 2024 OPTIONS IMPORT: --ifconfig/up options modified
vpn-1  | Mon Aug 12 08:31:00 2024 OPTIONS IMPORT: route-related options modified
vpn-1  | Mon Aug 12 08:31:00 2024 OPTIONS IMPORT: peer-id set
vpn-1  | Mon Aug 12 08:31:00 2024 OPTIONS IMPORT: adjusting link_mtu to 1659
vpn-1  | Mon Aug 12 08:31:00 2024 OPTIONS IMPORT: data channel crypto options modified
vpn-1  | Mon Aug 12 08:31:00 2024 Data Channel: using negotiated cipher 'AES-256-GCM'
vpn-1  | Mon Aug 12 08:31:00 2024 Outgoing Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
vpn-1  | Mon Aug 12 08:31:00 2024 Incoming Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
vpn-1  | Mon Aug 12 08:31:00 2024 ROUTE_GATEWAY 103.107.182.1/255.255.255.0 IFACE=eth0 HWADDR=fa:16:3e:c1:ee:2b
vpn-1  | Mon Aug 12 08:31:00 2024 TUN/TAP device kengine opened
vpn-1  | Mon Aug 12 08:31:00 2024 TUN/TAP TX queue length set to 100
vpn-1  | Mon Aug 12 08:31:00 2024 /sbin/ip link set dev kengine up mtu 1500
vpn-1  | Mon Aug 12 08:31:00 2024 /sbin/ip addr add dev kengine 10.99.0.4/20 broadcast 10.99.15.255
vpn-1  | Mon Aug 12 08:31:00 2024 /sbin/ip route add 123.31.11.151/32 via 103.107.182.1
vpn-1  | Mon Aug 12 08:31:00 2024 /sbin/ip route add 0.0.0.0/1 via 10.99.0.1
vpn-1  | Mon Aug 12 08:31:00 2024 /sbin/ip route add 128.0.0.0/1 via 10.99.0.1
vpn-1  | Mon Aug 12 08:31:00 2024 WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
vpn-1  | Mon Aug 12 08:31:00 2024 Initialization Sequence Completed

    This seem working but actually not, the ip addr add dev kengine 10.99.0.4/20 broadcast 10.99.15.255 is not working so there is no ip on the kengine 

112: kengine: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 100
    link/ether 52:a3:e8:10:4c:96 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::ac5b:61ff:fe54:98f6/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

b. I have to manualy add myself and the VPN finally work

112: kengine: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 100
    link/ether 52:a3:e8:10:4c:96 brd ff:ff:ff:ff:ff:ff
    inet 10.99.0.4/20 brd 10.99.15.255 scope global kengine
       valid_lft forever preferred_lft forever
    inet6 fe80::ac5b:61ff:fe54:98f6/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

Testing

pool-g4dzrku5-sj3dtqihuu6cjof6-node-hshs3fbx ~ # ping 10.99.0.1
PING 10.99.0.1 (10.99.0.1) 56(84) bytes of data.
64 bytes from 10.99.0.1: icmp_seq=1 ttl=64 time=2.50 ms
64 bytes from 10.99.0.1: icmp_seq=2 ttl=64 time=1.42 ms
64 bytes from 10.99.0.1: icmp_seq=3 ttl=64 time=1.45 ms
^C
--- 10.99.0.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 1.416/1.786/2.496/0.502 ms
pool-g4dzrku5-sj3dtqihuu6cjof6-node-hshs3fbx ~ # ping 10.99.0.2
PING 10.99.0.2 (10.99.0.2) 56(84) bytes of data.
64 bytes from 10.99.0.2: icmp_seq=1 ttl=64 time=2.11 ms
64 bytes from 10.99.0.2: icmp_seq=2 ttl=64 time=2.75 ms
64 bytes from 10.99.0.2: icmp_seq=3 ttl=64 time=2.14 ms
^C
--- 10.99.0.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 2.109/2.334/2.752/0.295 ms

but since i want automaticly run VPN and I cant add it manual all the time so this problem need look up to

  1. Error:

Not add IP properly on VPN interface

Expected behavior

The IP on interface automaticlly added

Additional information

None

jepio commented 3 months ago

Can you share the output of networkctl status from the host? Does the device show up as unmanaged?

Are you able to create a minimal reproducer for this?

lmq1999 commented 3 months ago

Can you share the output of networkctl status from the host? Does the device show up as unmanaged?

Are you able to create a minimal reproducer for this?


pool-g4dzrku5-sj3dtqihuu6cjof6-node-hshs3fbx ~ # networkctl status
● Interfaces: 1, 2, 3, 7, 8, 12, 14, 16, 24, 20, 22, 18, 30, 28, 26, 33, 34, 4, 113
State: routable                                    
Online state: online                                      
Address: 103.107.182.222 on eth0
10.20.4.127 on eth1
172.17.0.1 on docker0
10.200.0.159 on cilium_host
172.18.0.1 on br-098937d94946
fe80::f816:3eff:fec1:ee2b on eth0
fe80::f816:3eff:fefd:b77d on eth1
fe80::42:96ff:fe5b:9fc4 on docker0
fe80::40d7:33ff:fe7a:910b on cilium_net
fe80::48a5:55ff:fe9d:3bb2 on cilium_host
fe80::e037:abff:fe5c:6eb3 on lxc99eb9c4d3eb4
fe80::6c06:47ff:fe44:2265 on lxcd81121c9fde0
fe80::90dc:a0ff:fe41:bea0 on lxce0fba8614383
fe80::b842:75ff:fee7:cec3 on lxc6c022ee53b5c
fe80::40f3:58ff:fe97:219a on lxc21e65491adf6
fe80::a448:14ff:fe58:3189 on lxc7df0526efc64
fe80::1070:18ff:fe05:e8c0 on lxc59b2200808e0
fe80::6c7e:31ff:fe1f:316d on lxcc625a36d34a6
fe80::5c8e:d2ff:fe18:106 on lxc6316b6423586
fe80::8463:c4ff:fe74:5fb1 on lxcb04e3a7afb05
fe80::2cc2:e2ff:fe77:6d04 on lxc_health
fe80::42:97ff:fe94:1333 on br-098937d94946
fe80::d489:f4ff:fe2a:18e4 on kengine
Gateway: 103.107.182.1 on eth0
DNS: 8.8.8.8
45.124.93.62
Search Domains: openstacklocal

Aug 12 08:31:00 pool-g4dzrku5-sj3dtqihuu6cjof6-node-hshs3fbx systemd-networkd[1169]: kengine: Gained carrier Aug 12 08:31:00 pool-g4dzrku5-sj3dtqihuu6cjof6-node-hshs3fbx systemd-networkd[1169]: kengine: Configuring with /usr/lib/systemd/network/zz-default.network. Aug 12 08:31:02 pool-g4dzrku5-sj3dtqihuu6cjof6-node-hshs3fbx systemd-networkd[1169]: kengine: Gained IPv6LL Aug 12 09:46:24 pool-g4dzrku5-sj3dtqihuu6cjof6-node-hshs3fbx systemd-networkd[1169]: kengine: Link DOWN Aug 12 09:46:24 pool-g4dzrku5-sj3dtqihuu6cjof6-node-hshs3fbx systemd-networkd[1169]: kengine: Lost carrier Aug 12 09:46:24 pool-g4dzrku5-sj3dtqihuu6cjof6-node-hshs3fbx systemd-networkd[1169]: kengine: DHCPv6 lease lost Aug 12 09:46:30 pool-g4dzrku5-sj3dtqihuu6cjof6-node-hshs3fbx systemd-networkd[1169]: kengine: Link UP Aug 12 09:46:30 pool-g4dzrku5-sj3dtqihuu6cjof6-node-hshs3fbx systemd-networkd[1169]: kengine: Gained carrier Aug 12 09:46:30 pool-g4dzrku5-sj3dtqihuu6cjof6-node-hshs3fbx systemd-networkd[1169]: kengine: Configuring with /usr/lib/systemd/network/zz-default.network. Aug 12 09:46:31 pool-g4dzrku5-sj3dtqihuu6cjof6-node-hshs3fbx systemd-networkd[1169]: kengine: Gained IPv6LL



here
I can give you the client full config if you want
I have tried other openvpn-client and build 1 myself but seem they have common error not adding ip on itnerface
jepio commented 3 months ago

sorry, i meant networkctl list

jepio commented 3 months ago

but this already confirms my suspicion that networkd is trying to manage your device:

systemd-networkd[1169]: kengine: Configuring with /usr/lib/systemd/network/zz-default.network.
lmq1999 commented 3 months ago

but this already confirms my suspicion that networkd is trying to manage your device:

systemd-networkd[1169]: kengine: Configuring with /usr/lib/systemd/network/zz-default.network.

IDX LINK            TYPE     OPERATIONAL SETUP      
1 lo              loopback carrier     unmanaged
2 eth0            ether    routable    configured 
3 eth1            ether    routable    configured 
4 docker0         bridge   no-carrier  unmanaged
7 cilium_net      ether    degraded    unmanaged
8 cilium_host     ether    routable    unmanaged
12 lxc99eb9c4d3eb4 ether    degraded    unmanaged
14 lxcd81121c9fde0 ether    degraded    unmanaged
16 lxce0fba8614383 ether    degraded    unmanaged
18 lxc6c022ee53b5c ether    degraded    unmanaged
20 lxc21e65491adf6 ether    degraded    unmanaged
22 lxc7df0526efc64 ether    degraded    unmanaged
24 lxc59b2200808e0 ether    degraded    unmanaged
26 lxcc625a36d34a6 ether    degraded    unmanaged
28 lxc6316b6423586 ether    degraded    unmanaged
30 lxcb04e3a7afb05 ether    degraded    unmanaged
33 lxc_health      ether    degraded    unmanaged
34 br-098937d94946 bridge   no-carrier  unmanaged
113 kengine         ether    degraded    configuring

19 links listed.


look like is not working properly, so how to fix this problem
jepio commented 3 months ago

for a quick fix create /etc/systemd/network/kengine.network with these contents:

[Match]
Name=kengine

[Link]
Unmanaged=yes

and then systemctl restart systemd-networkd.

lmq1999 commented 3 months ago

for a quick fix create /etc/systemd/network/kengine.network with these contents:

[Match]
Name=kengine

[Link]
Unmanaged=yes

and then systemctl restart systemd-networkd.

wow ok it worked for now, tks for your support

jepio commented 3 months ago

I'll leave this issue open as we should exclude tun/tap interfaces from networkd management by default.

krishjainx commented 3 months ago

@jepio Hey Jeremi! How are you doing? I'm thinking of adding a new configuration file with the following content:

[Match]
Kind=tun tap

[Link]
Unmanaged=yes

This configuration will match all tun and tap interfaces and set them as unmanaged. I can make a PR if this looks good to you :)

jepio commented 3 months ago

Sure, try it out. The PR needs to be against flatcar/init and flatcar/bootengine (for consistency).

krishjainx commented 3 months ago

Sure, try it out. The PR needs to be against flatcar/init and flatcar/bootengine (for consistency).

Cool, I started a discussion here: link