Gefyra could not successfully establish the connection to '192.168.5.2'

[chrisegner]

What happened?

I'm new to colima and gefyra, so I apologize if I'm missing something obvious. I've attempted to follow the instructions on my M1 mac for connecting to a local k8s cluster running on colima without success. I've tried both with a k8s cluster created via colima's --with-kubernetes flag and with kind on colima.

Ideally I'd like to use gefyra with a kind-created cluster on colima, all on my mac. It failed with the same error, so I retried using just colima to create the cluster. That failed in the same way.

What did you expect to happen?

'gefyra up' should complete without error.

Please provide the output of gefyra check.

$  gefyra check
Usage: gefyra [OPTIONS] COMMAND [ARGS]...
Try 'gefyra --help' for help.

Error: No such command 'check'.

How can we reproduce it (as minimally and precisely as possible)?

Below is the step by step output for the colima --with-kubernetes attempt following the instructions at https://gefyra.dev/docs/local-environments/colima/:

$   colima start --with-kubernetes --network-address ktest1                                                                                                                          
INFO[0000] starting colima [profile=ktest1]
INFO[0000] runtime: docker+k3s
INFO[0001] starting ...                                  context=vm
INFO[0012] provisioning ...                              context=docker
INFO[0012] starting ...                                  context=docker
INFO[0013] provisioning ...                              context=kubernetes
INFO[0014] starting ...                                  context=kubernetes
INFO[0019] updating config ...                           context=kubernetes
INFO[0020] Switched to context "colima-ktest1".          context=kubernetes
INFO[0020] done
$   colima list
PROFILE    STATUS     ARCH       CPUS    MEMORY    DISK     RUNTIME       ADDRESS
default    Stopped    aarch64    2       2GiB      60GiB
dev        Stopped    aarch64    2       2GiB      60GiB
ktest1     Running    aarch64    2       2GiB      60GiB    docker+k3s    192.168.106.4
test2      Running    aarch64    2       2GiB      60GiB    docker+k3s    192.168.106.3
$   kubectx -c
colima-ktest1
$   gefyra up
Connecting local network 'gefyra-network-default' to the cluster (up to 10 min) |██████████████████████████████⚠︎         | (!) 3/4 [75%] in 6:29.5
Error: Gefyra could not successfully establish the connection to '192.168.5.2'.
If you have run 'gefyra up' with a remote cluster, a newly created route may not be working immediately.
Try running 'gefyra up' again after some time. Error: Gefyra could not successfully confirm the connection working.

FWIW, the manifest in step 3 applies just fine and the resources get created. But a curl to http://hello.127.0.0.1.nip.io:8080/ does not work. Not sure if that's related. Maybe I'm not understanding where I should be running that curl from?

What Kubernetes setup are you working with?

I'm working on an M1 mac using colima to run a k8s cluster created by colima.

  kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.1", GitCommit:"17b7accf8", GitTreeState:"clean", BuildDate:"2023-08-11T12:30:33Z", GoVersion:"go1.19.10", Compiler:"gc", Platform:"darwin/arm64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"28", GitVersion:"v1.28.3+k3s2", GitCommit:"bbafb86e91ae3682a1811119d136203957df9061", GitTreeState:"clean", BuildDate:"2023-11-08T01:45:45Z", GoVersion:"go1.20.10", Compiler:"gc", Platform:"linux/arm64"}
WARNING: version difference between client (1.26) and server (1.28) exceeds the supported minor version skew of +/-1

OS version

# Version info
$   gefyra version
Gefyra client version: 2.0.2
$   colima version
colima version 0.6.8
git commit: 9b0809d0ed9ad3ff1e57c405f27324e6298ca04f
$   sw_vers
ProductName:            macOS
ProductVersion:         14.2.1
BuildVersion:           23C71

Anything else we need to know?

No response

[chrisegner]

Checking the debug output, it appears something may be up with the wireguard networking. It doesn't appear any of the 80 connection probe attempts succeeds, either in the first or second rounds. Is that expected?

  gefyra -d up
DEBUG gefyra gefyra.configuration:242 - Docker Context: colima-ktest1
on 0: DEBUG gefyra gefyra.configuration:242 - Docker Context: colima-ktest1
on 0: DEBUG gefyra.api.status gefyra.api.status:93 - Reading API resources from Kubernetes
on 0: DEBUG gefyra.api.status gefyra.api.status:100 - Reading gefyra namespace
on 0: DEBUG gefyra.api.status gefyra.api.status:109 - Checking operator deployment
on 0: DEBUG gefyra.api.status gefyra.api.status:126 - Checking Stowaway endpoint
on 0: DEBUG gefyra.api.status gefyra.api.status:35 - Checking cargo container running
on 0: DEBUG gefyra.api.status gefyra.api.status:49 - Checking gefyra network available
on 0: DEBUG gefyra.api.status gefyra.api.status:63 - Probing wireguard connection
on 0: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 0/40))
on 0: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 1/40))
on 0: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 2/40))
on 0: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 3/40))
on 0: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 4/40))
on 0: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 5/40))
on 0: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 6/40))
on 0: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 7/40))
on 0: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 8/40))
on 0: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 9/40))
on 0: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 10/40))
on 0: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 11/40))
on 0: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 12/40))
on 0: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 13/40))
on 0: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 14/40))
on 0: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 15/40))
on 0: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 16/40))
on 0: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 17/40))
on 0: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 18/40))
on 0: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 19/40))
on 0: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 20/40))
on 0: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 21/40))
on 0: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 22/40))
on 0: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 23/40))
on 0: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 24/40))
on 0: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 25/40))
on 0: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 26/40))
on 0: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 27/40))
on 0: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 28/40))
on 0: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 29/40))
on 0: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 30/40))
on 0: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 31/40))
on 0: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 32/40))
on 0: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 33/40))
on 0: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 34/40))
on 0: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 35/40))
on 0: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 36/40))
on 0: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 37/40))
on 0: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 38/40))
on 0: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 39/40))
on 0: DEBUG gefyra.api.utils gefyra.api.utils:69 - Operation time for 'status(...)' was 41812.3852ms
on 0: WARNING gefyra gefyra.cli.updown:31 - Gefyra is not installed, but operating properly. Aborting.
on 0: DEBUG gefyra gefyra.configuration:242 - Docker Context: colima-ktest1
on 0: DEBUG gefyra.api.utils gefyra.api.utils:69 - Operation time for 'get_client(...)' was 23.1976ms
on 2: DEBUG gefyra gefyra.configuration:242 - Docker Context: colima-ktest1
on 2: DEBUG gefyra gefyra.configuration:242 - Docker Context: colima-ktest1
on 2: DEBUG gefyra.api.utils gefyra.api.utils:69 - Operation time for 'get_client(...)' was 22.2391ms
on 2: DEBUG gefyra.api.clients gefyra.api.clients:111 - gefyra_server: 192.168.5.2:31820
on 2: DEBUG gefyra.api.utils gefyra.api.utils:69 - Operation time for 'write_client_file(...)' was 39.0100ms
on 2: DEBUG gefyra gefyra.configuration:242 - Docker Context: colima-ktest1
on 3: DEBUG gefyra gefyra.cli.updown:152 - Minikube profile None
on 3: DEBUG gefyra gefyra.configuration:242 - Docker Context: colima-ktest1
on 3: DEBUG gefyra gefyra.configuration:242 - Docker Context: colima-ktest1
on 3: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 0/2))
on 3: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 1/2))
on 3: DEBUG gefyra.api.utils gefyra.api.utils:69 - Operation time for 'list_connections(...)' was 2117.7643ms
on 3: DEBUG gefyra.api.connect gefyra.api.connect:48 - Restoring exinsting connection default
on 3: DEBUG gefyra gefyra.configuration:242 - Docker Context: colima-ktest1
on 3: DEBUG gefyra gefyra.configuration:242 - Docker Context: colima-ktest1
on 3: DEBUG gefyra.api.utils gefyra.api.utils:69 - Operation time for 'get_client(...)' was 26.1328ms
on 3: INFO gefyra.local.networking gefyra.local.networking:29 - Gefyra network already exists
on 3: DEBUG gefyra.local.networking gefyra.local.networking:17 - Network {'Name': 'gefyra-network-default', 'Id': 'd55349d6b0890e1f575893f3894c3d56be7fa69271372b91b51d4857790d8081', 'Created': '2024-02-12T12:24:09.940403904-05:00', 'Scope': 'local', 'Driver': 'bridge', 'EnableIPv6': False, 'IPAM': {'Driver': 'default', 'Options': None, 'Config': [{'Subnet': '172.18.0.0/16'}]}, 'Internal': False, 'Attachable': False, 'Ingress': False, 'ConfigFrom': {'Network': ''}, 'ConfigOnly': False, 'Containers': {'6e4ba2fe94e50b6d4112f3b5d775f07e66bd7e1cad50188ac43d5f149950a035': {'Name': 'gefyra-cargo-default', 'EndpointID': '6d416cae6f86cb97a70b493d891ba352f834b97196bc802c81e9d82d38f12cf5', 'MacAddress': '02:42:ac:12:00:95', 'IPv4Address': '172.18.0.149/16', 'IPv6Address': ''}}, 'Options': {'com.docker.network.driver.mtu': '1340'}, 'Labels': {'created_by.gefyra.dev': 'true'}}
on 3: DEBUG gefyra.types gefyra.types:156 - Fetching object GefyraClient default
on 3: DEBUG gefyra.types gefyra.types:156 - Fetching object GefyraClient default
on 3: DEBUG gefyra.types gefyra.types:156 - Fetching object GefyraClient default
on 3: DEBUG gefyra gefyra.configuration:242 - Docker Context: colima-ktest1
on 3: DEBUG gefyra.api.connect gefyra.api.connect:136 - 192.168.5.2:31820
on 3: DEBUG gefyra.api.connect gefyra.api.connect:202 - Checking wireguard connection
on 3: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 0/40))
on 3: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 1/40))
on 3: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 2/40))
on 3: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 3/40))
on 3: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 4/40))
on 3: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 5/40))
on 3: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 6/40))
on 3: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 7/40))
on 3: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 8/40))
on 3: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 9/40))
on 3: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 10/40))
on 3: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 11/40))
on 3: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 12/40))
on 3: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 13/40))
on 3: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 14/40))
on 3: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 15/40))
on 3: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 16/40))
on 3: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 17/40))
on 3: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 18/40))
on 3: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 19/40))
on 3: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 20/40))
on 3: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 21/40))
on 3: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 22/40))
on 3: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 23/40))
on 3: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 24/40))
on 3: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 25/40))
on 3: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 26/40))
on 3: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 27/40))
on 3: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 28/40))
on 3: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 29/40))
on 3: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 30/40))
on 3: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 31/40))
on 3: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 32/40))
on 3: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 33/40))
on 3: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 34/40))
on 3: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 35/40))
on 3: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 36/40))
on 3: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 37/40))
on 3: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 38/40))
on 3: DEBUG gefyra.cargo gefyra.local.cargo:18 - Probing connection to 192.168.99.1 (attempt 39/40))
Connecting local network 'gefyra-network-default' to the cluster (up to 10 min) |██████████████████████████████⚠︎         | (!) 3/4 [75%] in 1:27.0
Error: Gefyra could not successfully establish the connection to '192.168.5.2'.
If you have run 'gefyra up' with a remote cluster, a newly created route may not be working immediately.
Try running 'gefyra up' again after some time. Error: Gefyra could not successfully confirm the connection working.

[chrisegner]

FWIW, the manifest in step 3 applies just fine and the resources get created. But a curl to http://hello.127.0.0.1.nip.io:8080/ does not work. Not sure if that's related. Maybe I'm not understanding where I should be running that curl from?

This is a docs-out-of-sync issue. I filed gefyrahq/gefyrahq.github.io#95.

[chrisegner]

On the assumption that the connection attempt is from inside the colima VM, I poked around the routing table and tested connecting to the host (mac). Worked fine.

$  colima ssh -p ktest2 -- routel
Dst             Gateway         Prefsrc         Protocol Scope   Dev              Table
default         192.168.5.2     192.168.5.6     dhcp             eth0
default         192.168.106.1   192.168.106.5   dhcp             col0
10.42.0.0/24                    10.42.0.1       kernel   link    cni0
172.17.0.0/16                   172.17.0.1      kernel   link    docker0
172.18.0.0/16                   172.18.0.1      kernel   link    br-6beb8a93d027
192.168.5.0/24                  192.168.5.6     kernel   link    eth0
192.168.5.2                     192.168.5.6     dhcp     link    eth0
192.168.106.0/24                 192.168.106.5   kernel   link    col0
192.168.106.1                   192.168.106.5   dhcp     link    col0
10.42.0.0                       10.42.0.0       kernel   host    flannel.1        local
10.42.0.1                       10.42.0.1       kernel   host    cni0             local
10.42.0.255                     10.42.0.1       kernel   link    cni0             local
127.0.0.0/8                     127.0.0.1       kernel   host    lo               local
127.0.0.1                       127.0.0.1       kernel   host    lo               local
127.255.255.255                 127.0.0.1       kernel   link    lo               local
172.17.0.1                      172.17.0.1      kernel   host    docker0          local
172.17.255.255                  172.17.0.1      kernel   link    docker0          local
172.18.0.1                      172.18.0.1      kernel   host    br-6beb8a93d027  local
172.18.255.255                  172.18.0.1      kernel   link    br-6beb8a93d027  local
192.168.5.6                     192.168.5.6     kernel   host    eth0             local
192.168.5.255                   192.168.5.6     kernel   link    eth0             local
192.168.106.5                   192.168.106.5   kernel   host    col0             local
192.168.106.255                 192.168.106.5   kernel   link    col0             local

# Demonstrates connecting from VM to host using the mac's sshd.
$  colima ssh -p ktest2 -- socat - TCP:192.168.5.2:22
SSH-2.0-OpenSSH_9.4

(ktest2 is the same as ktest1. Something got trashed and I started fresh.)