search

gefyrahq / gefyra

Blazingly-fast :rocket:, rock-solid, local application development :arrow_right: with Kubernetes.
https://gefyra.dev
Apache License 2.0
692 stars 28 forks source link

gefyra up: secrets "gefyra-cargo-connection" not found #112

Closed ilovechai closed 1 year ago

ilovechai commented 2 years ago

Tried setting up gefyra, but it errored out with the below error:

➜  ~ gefyra version
[INFO] Gefyra client version: 0.8.1

Logs:

➜  ~ gefyra -d up
[INFO] There was no --endpoint argument provided. Connecting to a local Kubernetes node.
[INFO] Installing Gefyra Operator
[DEBUG] Creating Docker network
[INFO] Created network 'gefyra' (63ea1b4a3c)
[DEBUG] Network {'Name': 'gefyra', 'Id': '63ea1b4a3c7db6343d701f981c2ecef650db3800911de5c8d61517c51bac5', 'Created': '2022-07-13T20:05:14.75968771Z', 'Scope': 'local', 'Driver': 'bridge', 'EnableIPv6': False, 'IPAM': {'Driver': 'default', 'Options': None, 'Config': [{'Subnet': '172.22.0.0/16'}]}, 'Internal': False, 'Attachable': False, 'Ingress': False, 'ConfigFrom': {'Network': ''}, 'ConfigOnly': False, 'Containers': {}, 'Options': {}, 'Labels': {}}
[INFO] Container image "quay.io/gefyra/operator:0.8.1" already present on machine
[INFO] Operator became ready in 190.4024 seconds
[ERROR] Not Found: {'kind': 'Status', 'apiVersion': 'v1', 'metadata': {}, 'status': 'Failure', 'message': 'secrets "gefyra-cargo-connection" not found', 'reason': 'NotFound', 'details': {'name': 'gefyra-cargo-connection', 'kind': 'secrets'}, 'code': 404}
 ~ oc get all
NAME                                   READY   STATUS    RESTARTS   AGE
pod/gefyra-operator-579fb7d567-s6qrp   1/1     Running   0          3m15s

NAME                                TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)           AGE
service/gefyra-stowaway-rsync       ClusterIP   None            <none>        10873/TCP         3m13s
service/gefyra-stowaway-wireguard   NodePort    172.30.126.77   <none>        51820:31820/UDP   3m13s

NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/gefyra-operator   1/1     1            1           3m16s
deployment.apps/gefyra-stowaway   0/1     0            0           3m13s

NAME                                         DESIRED   CURRENT   READY   AGE
replicaset.apps/gefyra-operator-579fb7d567   1         1         1       3m17s
replicaset.apps/gefyra-stowaway-68886d4c9c   1         0         0       3m14s
➜  ~ oc get secrets
NAME                              TYPE                                  DATA   AGE
builder-dockercfg-hj9sl           kubernetes.io/dockercfg               1      87s
builder-token-6jznc               kubernetes.io/service-account-token   4      87s
builder-token-sjd5s               kubernetes.io/service-account-token   4      87s
default-dockercfg-trn5j           kubernetes.io/dockercfg               1      87s
default-token-8wqtw               kubernetes.io/service-account-token   4      87s
default-token-hk2ww               kubernetes.io/service-account-token   4      87s
deployer-dockercfg-zsmqk          kubernetes.io/dockercfg               1      87s
deployer-token-kndfn              kubernetes.io/service-account-token   4      87s
deployer-token-nwk2q              kubernetes.io/service-account-token   4      87s
gefyra-operator-dockercfg-f9vv9   kubernetes.io/dockercfg               1      87s
gefyra-operator-token-r5ptn       kubernetes.io/service-account-token   4      87s
gefyra-operator-token-sl6g8       kubernetes.io/service-account-token   4      87s

oc version:

Client Version: v4.2.0-alpha.0-1420-gf1f09a3
Server Version: 4.8.43
Kubernetes Version: v1.21.11+6b3cbdd
SteinRobert commented 2 years ago

It looks like the stowaway deployment is not ready. Could you please provide us with the logs of the stowaway pod?

ilovechai commented 2 years ago

@SteinRobert I don't see a stowaway pod. 

oc logs -f pod/gefyra-operator-6789fd6ddd-x5ksw
[2022-07-14 08:13:56,283] gefyra               [INFO    ] Gefyra Operator startup
[2022-07-14 08:13:56,284] gefyra               [INFO    ] Loaded in-cluster config
[2022-07-14 08:13:56,324] kopf.activities.star [INFO    ] Ensuring Gefyra components with the following configuration: {'NAMESPACE': 'gefyra', 'STOWAWAY_IMAGE': 'quay.io/gefyra/stowaway', 'STOWAWAY_IMAGE_PULLPOLICY': 'Always', 'STOWAWAY_TAG': '0.8.1', 'WIREGUARD_EXT_PORT': 31820, 'STOWAWAY_PGID': '1000', 'STOWAWAY_PUID': '1000', 'STOWAWAY_STARTUP_TIMEOUT': 180, 'STOWAWAY_PEER_DNS': 'auto', 'STOWAWAY_PEER_CONFIG_PATH': '/config/peer1/peer1.conf', 'STOWAWAY_INTERNAL_SUBNET': '192.168.99.0', 'GEFYRA_PEER_SUBNET': '172.23.0.0/16', 'STOWAWAY_PROXYROUTE_CONFIGMAPNAME': 'gefyra-stowaway-proxyroutes', 'CARRIER_IMAGE': 'quay.io/gefyra/carrier', 'CARRIER_IMAGE_TAG': 'latest', 'CARRIER_STARTUP_TIMEOUT': 60}
[2022-07-14 08:13:56,349] kopf.activities.star [INFO    ] Gefyra CRD InterceptRequest created
[2022-07-14 08:13:56,365] kopf.activities.star [INFO    ] Stowaway proxy route configmap created
[2022-07-14 08:13:56,384] kopf.activities.star [INFO    ] Stowaway deployment created
[2022-07-14 08:13:56,420] kopf.activities.star [INFO    ] Stowaway nodeport service created
[2022-07-14 08:13:56,434] kopf.activities.star [INFO    ] Stowaway rsync service created
[2022-07-14 08:13:56,451] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:13:57,460] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:13:58,470] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:13:59,492] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:00,502] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:01,514] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:02,525] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:03,533] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:04,545] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:05,554] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:06,567] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:07,578] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:08,588] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:09,597] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:10,607] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:11,619] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:12,631] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:13,642] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:14,654] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:15,664] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:16,676] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:17,686] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:18,710] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:19,729] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:20,739] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:21,748] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:22,759] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:23,770] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:24,780] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:25,791] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:26,805] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:27,816] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:28,828] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:29,838] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:30,848] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:31,858] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:32,871] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:33,882] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:34,892] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:35,902] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:36,940] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:37,948] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:38,957] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:39,969] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:40,980] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:42,000] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:43,012] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:44,021] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:45,033] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:46,044] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:47,055] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:48,065] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:49,076] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:50,085] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:51,095] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:52,107] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:53,119] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:54,130] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:55,140] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:56,149] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:57,159] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:58,169] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:14:59,179] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:00,191] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:01,202] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:02,212] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:03,225] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:04,236] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:05,247] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:06,256] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:07,268] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:08,279] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:09,290] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:10,302] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:11,313] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:12,323] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:13,333] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:14,343] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:15,355] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:16,365] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:17,379] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:18,389] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:19,400] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:20,410] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:21,421] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:22,432] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:23,443] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:24,452] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:25,462] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:26,473] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:27,483] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:28,496] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:29,510] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:30,521] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:31,534] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:32,550] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:33,561] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:34,572] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:35,581] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:36,590] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:37,601] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:38,612] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:39,659] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:40,669] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:41,680] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:42,692] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:43,709] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:44,720] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:45,731] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:46,742] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:47,753] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:48,773] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:49,797] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:50,807] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:51,817] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:52,827] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:53,838] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:54,853] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:55,863] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:56,872] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:57,883] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:58,893] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:15:59,903] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:00,914] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:01,925] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:02,935] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:03,947] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:04,958] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:05,969] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:06,979] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:07,987] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:08,997] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:10,009] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:11,027] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:12,040] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:13,050] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:14,060] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:15,070] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:16,080] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:17,089] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:18,099] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:19,109] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:20,119] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:21,129] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:22,138] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:23,150] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:24,161] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:25,173] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:26,182] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:27,192] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:28,202] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:29,211] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:30,226] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:31,236] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:32,245] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:33,255] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:34,267] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:35,276] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:36,286] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:37,295] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:38,307] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:39,317] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:40,329] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:41,339] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:42,348] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:43,359] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:44,369] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:45,379] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:46,388] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:47,397] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:48,409] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:49,419] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:50,430] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:51,440] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:52,457] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:53,469] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:54,484] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:55,496] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:56,510] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:57,521] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:58,534] gefyra.stowaway      [INFO    ] Waiting for Stowaway to become ready
[2022-07-14 08:16:59,546] gefyra.stowaway      [ERROR   ] Stowaway error: Stowaway did not become ready
[2022-07-14 08:16:59,570] kopf.activities.star [INFO    ] Gefyra components installed/patched
[2022-07-14 08:16:59,570] kopf.activities.star [INFO    ] Activity 'check_gefyra_components' succeeded.
[2022-07-14 08:16:59,572] kopf.activities.star [INFO    ] Activity 'configure' succeeded.
[2022-07-14 08:16:59,573] kopf._core.engines.a [INFO    ] Initial authentication has been initiated.
[2022-07-14 08:16:59,575] kopf.activities.auth [INFO    ] Activity 'login_via_client' succeeded.
[2022-07-14 08:16:59,576] kopf._core.engines.a [INFO    ] Initial authentication has finished.
oc describe deploy/gefyra-stowaway
Name:                   gefyra-stowaway
Namespace:              gefyra
CreationTimestamp:      Thu, 14 Jul 2022 13:43:56 +0530
Labels:                 <none>
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               app=stowaway
Replicas:               1 desired | 0 updated | 0 total | 0 available | 1 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  app=stowaway
  Containers:
   stowaway:
    Image:      quay.io/gefyra/stowaway:0.8.1
    Port:       51820/UDP
    Host Port:  0/UDP
    Limits:
      cpu:     750m
      memory:  500Mi
    Requests:
      cpu:     100m
      memory:  100Mi
    Environment:
      PEERS:                     1
      SERVERPORT:                31820
      PUID:                      1000
      PGID:                      1000
      PEERDNS:                   auto
      INTERNAL_SUBNET:           192.168.99.0
      SERVER_ALLOWEDIPS_PEER_1:  172.23.0.0/16
    Mounts:
      /lib/modules from host-libs (rw)
      /stowaway/proxyroutes from proxyroutes (rw)
  Volumes:
   proxyroutes:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      gefyra-stowaway-proxyroutes
    Optional:  false
   host-libs:
    Type:          HostPath (bare host directory volume)
    Path:          /lib/modules
    HostPathType:  Directory
Conditions:
  Type             Status  Reason
  ----             ------  ------
  Available        False   MinimumReplicasUnavailable
  ReplicaFailure   True    FailedCreate
  Progressing      False   ProgressDeadlineExceeded
OldReplicaSets:    <none>
NewReplicaSet:     gefyra-stowaway-78f4c945d (0/1 replicas created)
Events:            <none>
SteinRobert commented 2 years ago

Thank you! Could you describe the stowaway replicaset?

Schille commented 2 years ago

I would suspect OpenShift does not allow to run Pods with privileged mode (see: https://kubernetes.io/docs/concepts/security/pod-security-policy/#:~:text=Privileged%20%2D%20determines%20if%20any%20container,processes%20running%20on%20the%20host) per default. The Stowaway is currently running with privileged: true and capabilities ["NET_ADMIN", "SYS_MODULE"]

I am not very familiar with OpenShift, but as of a short research I assume we need some additional policies to enable this for OpenShift.

Schille commented 2 years ago

Alternatively, we could try to remove these requirements as they seem to come from a time prior to using wireguard-go for the VPN. Although I am a bit pessimistic about this.

ilovechai commented 2 years ago

This is https://github.com/gefyrahq/gefyra/issues/112#issuecomment-1185248439 exactly why the pod is not running. 

oc describe rs gefyra-stowaway-5965df8947
Name:           gefyra-stowaway-5965df8947
Namespace:      gefyra
Selector:       app=stowaway,pod-template-hash=5965df8947
Labels:         app=stowaway
                pod-template-hash=5965df8947
Annotations:    deployment.kubernetes.io/desired-replicas: 1
                deployment.kubernetes.io/max-replicas: 2
                deployment.kubernetes.io/revision: 1
Controlled By:  Deployment/gefyra-stowaway
Replicas:       0 current / 1 desired
Pods Status:    0 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:  app=stowaway
           pod-template-hash=5965df8947
  Containers:
   stowaway:
    Image:      quay.io/gefyra/stowaway:0.8.1
    Port:       51820/UDP
    Host Port:  0/UDP
    Limits:
      cpu:     750m
      memory:  500Mi
    Requests:
      cpu:     100m
      memory:  100Mi
    Environment:
      PEERS:                     1
      SERVERPORT:                31820
      PUID:                      1000
      PGID:                      1000
      PEERDNS:                   auto
      INTERNAL_SUBNET:           192.168.99.0
      SERVER_ALLOWEDIPS_PEER_1:  172.24.0.0/16
    Mounts:
      /lib/modules from host-libs (rw)
      /stowaway/proxyroutes from proxyroutes (rw)
  Volumes:
   proxyroutes:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      gefyra-stowaway-proxyroutes
    Optional:  false
   host-libs:
    Type:          HostPath (bare host directory volume)
    Path:          /lib/modules
    HostPathType:  Directory
Conditions:
  Type             Status  Reason
  ----             ------  ------
  ReplicaFailure   True    FailedCreate
Events:
  Type     Reason        Age                   From                   Message
  ----     ------        ----                  ----                   -------
  Warning  FailedCreate  3m18s (x19 over 14m)  replicaset-controller  Error creating: pods "gefyra-stowaway-5965df8947-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider "*****": Forbidden: not usable by user or serviceaccount, provider "******": Forbidden: not usable by user or serviceaccount, provider "******": Forbidden: not usable by user or serviceaccount, spec.volumes[1]: Invalid value: "hostPath": hostPath volumes are not allowed to be used, spec.containers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed, spec.containers[0].securityContext.capabilities.add: Invalid value: "NET_ADMIN": capability may not be added, spec.containers[0].securityContext.capabilities.add: Invalid value: "SYS_MODULE": capability may not be added, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "******": Forbidden: not usable by user or serviceaccount, provider "*****": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount]
Schille commented 2 years ago

Wonderful. Thanks for your response and willingness to support us with your data.

In the meantime, I have successfully tested an unprivileged version of Stowaway with my k3d setup. If you can confirm it working, too, this will be the default with the next release. Please try running it like so: gefyra up --operator quay.io/gefyra/operator:unprivileged

Edit: If this command terminated successfully (exit code: 0) it means the connection is working. There is an integrated mechanism to probe the connection upon setting everything up.

ilovechai commented 2 years ago

@Schille I executed gefyra up --operator quay.io/gefyra/operator:unprivileged, but it seems that stowaway is still trying to run as privileged 

gefyra up --operator quay.io/gefyra/operator:unprivileged
[INFO] There was no --endpoint argument provided. Connecting to a local Kubernetes node.
[INFO] Installing Gefyra Operator
[INFO] Created network 'gefyra' (9a156dc3e1)
[INFO] Container image "quay.io/gefyra/operator:unprivileged" already present on machine
[INFO] Operator became ready in 190.8331 seconds
[ERROR] Not Found: {'kind': 'Status', 'apiVersion': 'v1', 'metadata': {}, 'status': 'Failure', 'message': 'secrets "gefyra-cargo-connection" not found', 'reason': 'NotFound', 'details': {'name': 'gefyra-cargo-connection', 'kind': 'secrets'}, 'code': 404}

I see the same error as described here https://github.com/gefyrahq/gefyra/issues/112#issuecomment-1185308019

Schille commented 2 years ago

I was able to break down the issues here. 1) in OpenShift there is a so called SCC (see https://docs.openshift.com/container-platform/4.10/authentication/managing-security-context-constraints.html) preventing Stowaway to start with the required capabilities 2) Stowaway does currently not employ a proper service account to assign privileges 3) there is another issue with running wireguard-go in CRC (at least with OpenShift Local)

In my current development state I could remedy point 1 and 2. After assigning Stowaway a service account (which will be created automatically), I am able to releax the SCC on it with: oc adm policy add-scc-to-user privileged -z gefyra-stowaway -n gefyra. Then, the Stowaway Pod was successfully scheduled and started with an error message concerning wireguard-go. That is where I am currently stuck. I am going to investigate this one in the coming days.

ilovechai commented 2 years ago

@Schille Let me know if need any other logs. 1 and 2 in https://github.com/gefyrahq/gefyra/issues/112#issuecomment-1186874193 works for me, but still stuck at 3.

Schille commented 2 years ago

Thank you for your help. Do you run CRC locally? - on my commodity development machine it runs quite sluggish.

I wonder if it is possible to contact the makers of OpenShift to find out how to run a Pod with wireguard-go in it. :smile: Anyway, I am at it, however it will take some time unfortunately.

ilovechai commented 2 years ago

@Schille I do not run CRC, I have an openshift environment setup. What I meant when I was stuck was, after executing oc adm policy add-scc-to-user privileged -z gefyra-stowaway -n gefyra I do not see the stowaway pod running. I get the following error:

$ oc describe rs gefyra-stowaway-6b565ffc7d
Name:           gefyra-stowaway-6b565ffc7d
Namespace:      gefyra
Selector:       app=stowaway,pod-template-hash=6b565ffc7d
Labels:         app=stowaway
                pod-template-hash=6b565ffc7d
Annotations:    deployment.kubernetes.io/desired-replicas: 1
                deployment.kubernetes.io/max-replicas: 2
                deployment.kubernetes.io/revision: 1
Controlled By:  Deployment/gefyra-stowaway
Replicas:       0 current / 1 desired
Pods Status:    0 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:  app=stowaway
           pod-template-hash=6b565ffc7d
  Containers:
   stowaway:
    Image:      quay.io/gefyra/stowaway:0.8.1
    Port:       51820/UDP
    Host Port:  0/UDP
    Limits:
      cpu:     750m
      memory:  500Mi
    Requests:
      cpu:     100m
      memory:  100Mi
    Environment:
      PEERS:                     1
      SERVERPORT:                31820
      PUID:                      1000
      PGID:                      1000
      PEERDNS:                   auto
      INTERNAL_SUBNET:           192.168.99.0
      SERVER_ALLOWEDIPS_PEER_1:  172.19.0.0/16
    Mounts:
      /lib/modules from host-libs (rw)
      /stowaway/proxyroutes from proxyroutes (rw)
  Volumes:
   proxyroutes:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      gefyra-stowaway-proxyroutes
    Optional:  false
   host-libs:
    Type:          HostPath (bare host directory volume)
    Path:          /lib/modules
    HostPathType:  Directory
Conditions:
  Type             Status  Reason
  ----             ------  ------
  ReplicaFailure   True    FailedCreate
Events:
  Type     Reason        Age                 From                   Message
  ----     ------        ----                ----                   -------
  Warning  FailedCreate  11s (x15 over 93s)  replicaset-controller  Error creating: pods "gefyra-stowaway-6b565ffc7d-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider "<some-secret-1>": Forbidden: not usable by user or serviceaccount, provider "<some-secret-2>": Forbidden: not usable by user or serviceaccount, provider "<some-secret-3>": Forbidden: not usable by user or serviceaccount, spec.volumes[1]: Invalid value: "hostPath": hostPath volumes are not allowed to be used, spec.containers[0].securityContext.capabilities.add: Invalid value: "NET_ADMIN": capability may not be added, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "machine-api-termination-handler": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount]
Schille commented 2 years ago

With version 0.8.4 I was finally able to connect Gefyra with Openshift 4 local (crc). Here is what I did: 1) crc start At some point the output told me the IP address of this local cluster INFO CRC instance is running with IP 192.168.130.11 2) gefyra up --endpoint 192.168.130.11:31820 This initially installs Gefyra's cluster side components with Stowaway and its service account 3) oc adm policy add-scc-to-user privileged -z gefyra-stowaway -n gefyra With cluster admin privileges, I was able to grant the service account "gefyra-stowaway" all privileges. I executed that command when gefyra up ... was at [INFO] Successfully pulled image "quay.io/gefyra/stowaway:0.8.4" (afterwards should be fine, too).

The last log line of gefyra up ... was [INFO] Deploying Cargo (network sidecar) with IP <myip>which indicated that the connection has been established successfully. Awesome. I assume the rest should be working fine (I did not fully test the "getting started" as the performance is very poor on my machine running crc).

Anyway. This is not really a satisfying developer experience at this point. One solution I see to get Gefyra working a bit less hacky would be to split up the installation of the cluster side components (the operator, and the VPN endpoint) and the connection from a developer's machine. It would look like this: 1) install Gefyra in the cluster (requires "admin" role privileges in the cluster) 2) connect to the cluster using Gefyra's client (requires only "developer" role privileges) 3) gefyra down ... would not eliminate the cluster parts The seconds step can be performed as often as needed; Gefyra's operator remains in the cluster.

@ilovechai I would like to learn more about your development setup and how you would like to use Gefyra. Based on your input I can imagine to build a solution which works best for you and other people, supporting remote development scenarios with Gefyra.

ilovechai commented 2 years ago

@Schille In https://github.com/gefyrahq/gefyra/issues/112#issuecomment-1197723321, I gave the appropriate permission to the gefyra, but it still gave the erorr. Looking at the error, it seems that privileged pod cannot be created in the namespace.

By default, openshift does not allow creating privilege pod.

I think we would have to follow these steps(will verify):

to allow the gefyra pod to run.

I would like to learn more about your development setup and how you would like to use Gefyra. Based on your input I can imagine to build a solution which works best for you and other people, supporting remote development scenarios with Gefyra.

We have openshift development environments setup and the clear usecase is to intercept environment variables and volume mounts of a pod. Local dev setup should also be able to communicate with other pods/services inside the cluster.

Schille commented 2 years ago

@ilovechai it would be wonderful if you can get it running creating the service account with appropriate permissions upfront. From looking into the code, the service account should be picked up in case it already exists as long as it is named "gefyra-stowaway". If there is something preventing that from working, I will eagerly remove that obstacle so that you can get a working setup as fast as possible.

With crc I could not observe this error. Maybe this is because of a difference between CRC and the OpenShift you are using.

We have openshift development environments setup and the clear usecase is to intercept environment variables and volume mounts of a pod. Local dev setup should also be able to communicate with other pods/services inside the cluster.

Thank you very much. That is exactly what we're trying to achieve with Gefyra.

Schille commented 2 years ago

Hi @ilovechai Do you have any updates on this? Is there something I can help you with?

Schille commented 2 years ago

Hello @ilovechai is this still on your mind?

In the meantime, I was trying to get an Openshift cluster through Redhat running on Google Cloud. But the web service was broken somehow and I was not able to spin it up. :( However, I would be very interested in a working implementation of Gefyra with Openshift.

ilovechai commented 2 years ago

Hello @Schille , Apologies for the late reply. I'm yet to test this as I described in my earlier comment. I will let you know the results soon.

SteinRobert commented 2 years ago

@ilovechai any news on this? Have you had the chance to take a look?

SteinRobert commented 1 year ago

Closed due to inactivity.