gefyrahq / gefyra

Blazingly-fast :rocket:, rock-solid, local application development :arrow_right: with Kubernetes.
https://gefyra.dev
Apache License 2.0
695 stars 28 forks source link

`gefyra run` to set common Kubernetes env variables and ServiceAccount data (K8s cert and token) #319

Open Schille opened 1 year ago

Schille commented 1 year ago

What is the new feature about?

At the least, those two are required in order to connect a locally running container with the remote K8s API through an internal path: KUBERNETES_SERVICE_HOST - ip KUBERNETES_SERVICE_PORT - port

In addition, it would be a nice to have to get/assign a service account for the local container.

Why would such a feature be important to you?

When writing applications that communicate with K8s API server, it would be important to make the address available to the locally running container.

Anything else we need to know?

No response

Schille commented 1 year ago

@schwobaseggl Here's my PoC with k3d. 1) Create cluster k3d cluster create mycluster --agents 1 -p 8080:80@agent:0 -p 31820:31820/UDP@agent:0 2) Run a Pod with Ubuntu in mycluster

apiVersion: v1
kind: Pod
metadata:
  name: ubuntu
spec:
  containers:
    - name: ubuntu
      image: ubuntu
      command:
        - sleep
        - infinity

3) Retrieve Kubernetes certificate and SA token from ubuntu

4) Setup a Gefyra container

5) Copy the Kubernetes certificate and SA token to Gefyra container from the current directory on the host

docker cp sa.tar.gz ubuntu:var/sa.tar.gz

6) Prepare Gefyra container with curl and the copied data

In both shells (K3d running Pod and local Gefyra container), this should work:

APISERVER=https://kubernetes

# Path to ServiceAccount token
SERVICEACCOUNT=/var/run/secrets/kubernetes.io/serviceaccount

# Read this Pod's namespace
NAMESPACE=$(cat ${SERVICEACCOUNT}/namespace)

# Read the ServiceAccount bearer token
TOKEN=$(cat ${SERVICEACCOUNT}/token)

# Reference the internal certificate authority (CA)
CACERT=${SERVICEACCOUNT}/ca.crt

# Explore the API with TOKEN     
curl --cacert ${CACERT} --header "Authorization: Bearer ${TOKEN}" -X GET ${APISERVER}/api

{
  "kind": "APIVersions",
  "versions": [
    "v1"
  ],
  "serverAddressByClientCIDRs": [
    {
      "clientCIDR": "0.0.0.0/0",
      "serverAddress": "172.20.0.2:6443"
    }
  ]
}
Schille commented 1 year ago

Here's the PoC for a specific ServiceAccount (when given with gefyra run --sa mysa) after setting everything up from above.

1) Create a service account, ClusterRole, and ClusterRoleBinding

apiVersion: v1
kind: ServiceAccount
metadata:
  name: mysa
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: pod-manager
rules:
- apiGroups: [""] 
  resources: ["pods"]
  verbs: ["*"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: manage-pods
subjects: 
- kind: ServiceAccount
  name: mysa
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: pod-manager

2) Retrieve ServiceAccount token

SECRET=$(kubectl get serviceaccount mysa -o json | jq -Mr '.secrets[].name | select(contains("token"))')
TOKEN=$(kubectl get secret ${SECRET} -o json | jq -Mr '.data.token' | base64 -d)

3) Put the token to the local Gefyra container

docker exec ubuntu bash -c "echo '`echo $TOKEN`' > /var/run/secrets/kubernetes.io/serviceaccount/token"

4) curl the API server from local Gefyra container

{ "kind": "PodList", "apiVersion": "v1", "metadata": { "resourceVersion": "2498" }, "items": [ { "metadata": { "name": "ubuntu", "namespace": "default", "uid": "4cd343e6-7854-4d63-a21c-c4efe479341b", "resourceVersion": "877", "creationTimestamp": "2023-03-17T16:26:40Z", "annotations": { [...] "startTime": "2023-03-17T16:26:40Z", "containerStatuses": [ { "name": "ubuntu", "state": { "running": { "startedAt": "2023-03-17T16:27:09Z" } }, "lastState": {

        },
        "ready": true,
        "restartCount": 0,
        "image": "docker.io/library/ubuntu:latest",
        "imageID": "docker.io/library/ubuntu@sha256:67211c14fa74f070d27cc59d69a7fa9aeff8e28ea118ef3babc295a0428a6d21",
        "containerID": "containerd://5c7bae9481439bd5e35f08caf7446e47b0f30a963ec06cc5d66f9c445a5b919a",
        "started": true
      }
    ],
    "qosClass": "BestEffort"
  }
}

] }

Schille commented 1 year ago

I'd say for getting the env from a Gefyra phantom Pod in the cluster when gefyra running, we should create a busybox Pod:

apiVersion: v1
kind: Pod
metadata:
  name: my-phantom-run-1
spec:
  containers:
    - name: busybox
      image: busybox
      command:
        - sleep
        - infinity

And once the Pod is running:

kubectl exec my-phantom-run-1 -- env
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=my-phantom-run-1
KUBERNETES_SERVICE_HOST=10.43.0.1
KUBERNETES_SERVICE_PORT=443
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_PORT=tcp://10.43.0.1:443
KUBERNETES_PORT_443_TCP=tcp://10.43.0.1:443
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_ADDR=10.43.0.1
HOME=/root

While copying the environment to the local Gefyra container, we should modify the HOSTNAME value to match the local container's name.

schwobaseggl commented 1 year ago

@Schille Reproduction with exact steps above. ubuntu 20.04LTS amd gefyra 1.0.4 Docker engine 23.0.1 api version 1.42 Go 1.19.5 containerd 1.6.18 runc 1.1.4 docker-init 0.19.0 k3d 5.4.3 k3s 1.23.6 kubectl client 1.24.1

  1. Create cluster

    k3d cluster create mycluster --agents 1 -p 8080:80@agent:0 -p 31820:31820/UDP@agent:0

    Cluster spins up w/o issues

  2. Run pod with exact config from your comment

    $ kubectl apply -f po.yaml
    $ kubectl get po
    NAME     READY   STATUS    RESTARTS   AGE
    ubuntu   1/1     Running   0          53s

    All good

  3. .... all steps to 5. with no issues

  4. API curl:

    Output in k3d pod

    {
    "kind": "APIVersions",
    "versions": [
    "v1"
    ],
    "serverAddressByClientCIDRs": [
    {
      "clientCIDR": "0.0.0.0/0",
      "serverAddress": "172.21.0.2:6443"
    }
    ]
    }

    Output in gefyra docker container

    
    curl: (60) SSL certificate problem: self-signed certificate in certificate chain
    More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above.


7. API curl with dedicated service account
Same symptoms: in k3d pod, I get the PodList, in gefyra docker container,  I get
the error from above

8. One notable difference seems to be that the api server sends different certificates to the pod and container respectively:
In k3d pod:

root@ubuntu:/# echo | openssl s_client -showcerts -servername kubernetes -connect 10.43.0.1:443 CONNECTED(00000003) depth=0 O = k3s, CN = k3s verify error:num=20:unable to get local issuer certificate verify return:1 depth=0 O = k3s, CN = k3s verify error:num=21:unable to verify the first certificate verify return:1 depth=0 O = k3s, CN = k3s verify return:1

Certificate chain 0 s:O = k3s, CN = k3s i:CN = k3s-server-ca@1679651976 a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA256 v:NotBefore: Mar 24 09:59:36 2023 GMT; NotAfter: Mar 23 10:23:11 2024 GMT -----BEGIN CERTIFICATE----- MIICRDCCAeugAwIBAgIIfsJFLH1/1bcwCgYIKoZIzj0EAwIwIzEhMB8GA1UEAwwY azNzLXNlcnZlci1jYUAxNjc5NjUxOTc2MB4XDTIzMDMyNDA5NTkzNloXDTI0MDMy MzEwMjMxMVowHDEMMAoGA1UEChMDazNzMQwwCgYDVQQDEwNrM3MwWTATBgcqhkjO PQIBBggqhkjOPQMBBwNCAAQHM/gpMUVyBlWd5lhQ+qPZFAIP6Ee1kMPt+FTP5UIT GtUc7chqtFey/nnV3OMreDC8u+ys5D5CDXTCUep1XnyGo4IBDjCCAQowDgYDVR0P AQH/BAQDAgWgMBMGA1UdJQQMMAoGCCsGAQUFBwMBMB8GA1UdIwQYMBaAFObja+hv hjOakT2BwP4n67draHAEMIHBBgNVHREEgbkwgbaCCWdudXBnLm9yZ4IWazNkLW15 Y2x1c3Rlci1zZXJ2ZXItMIIKa3ViZXJuZXRlc4ISa3ViZXJuZXRlcy5kZWZhdWx0 ghZrdWJlcm5ldGVzLmRlZmF1bHQuc3ZjgiRrdWJlcm5ldGVzLmRlZmF1bHQuc3Zj LmNsdXN0ZXIubG9jYWyCCWxvY2FsaG9zdIcEAAAAAIcECisAAYcEfwAAAYcErBUA AocQAAAAAAAAAAAAAAAAAAAAATAKBggqhkjOPQQDAgNHADBEAiANBup7pWDiuFwj tfeeOB5aBWDJ0Qpk7pFCnGQXNRZPawIgc2s/LR/x2n5PkErMsC4/WUBRHkQ0ENIb JX3YQ3zt4fo= -----END CERTIFICATE-----

Server certificate subject=O = k3s, CN = k3s issuer=CN = k3s-server-ca@1679651976

No client certificate CA names sent Requested Signature Algorithms: RSA-PSS+SHA256:ECDSA+SHA256:Ed25519:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA384:ECDSA+SHA512:RSA+SHA1:ECDSA+SHA1 Shared Requested Signature Algorithms: RSA-PSS+SHA256:ECDSA+SHA256:Ed25519:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA384:ECDSA+SHA512 Peer signing digest: SHA256 Peer signature type: ECDSA Server Temp Key: X25519, 253 bits

SSL handshake has read 1007 bytes and written 406 bytes Verification error: unable to verify the first certificate

New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256 Server public key is 256 bit Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE No ALPN negotiated Early data was not sent Verify return code: 21 (unable to verify the first certificate)


Post-Handshake New Session Ticket arrived: SSL-Session: Protocol : TLSv1.3 Cipher : TLS_AES_128_GCM_SHA256 Session-ID: 7BADB03C1BA4B4A861429A86A1BA5191CFDD3F8AE91EBB96F2C50F37905F3AE0 Session-ID-ctx: Resumption PSK: FB38CB1E6181712B29AB55A159283B5E4AFA06F8011C4E2321424A1D5BCC7DD0 PSK identity: None PSK identity hint: None SRP username: None TLS session ticket lifetime hint: 604800 (seconds) TLS session ticket: 0000 - fc a6 ec 25 45 77 5f 32-62 b8 92 45 7a 7c 6e 36 ...%Ew2b..Ez|n6 0010 - ae 1c 1f 27 d9 52 ea ab-be 6a 9f 92 90 59 3b 69 ...'.R...j...Y;i 0020 - 9a 41 5b 87 38 c4 aa 75-55 fa ea 47 2c 27 64 9b .A[.8..uU..G,'d. 0030 - 49 ba e9 03 1a ce 0d 74-fe b4 48 d8 50 2e 33 0d I......t..H.P.3. 0040 - 6f 2f 0e cc a8 0d 33 c4-a6 fb c1 ec 58 63 5b 5e o/....3.....Xc[^ 0050 - 3c f4 12 57 de 61 e9 d6-46 3c 5f f2 fc d7 63 f9 <..W.a..F<...c. 0060 - ae 6d 7c de 86 ac 76 69-09 a1 40 2e fc 15 4b 62 .m|...vi..@...Kb 0070 - 1f .

Start Time: 1679653722
Timeout   : 7200 (sec)
Verify return code: 21 (unable to verify the first certificate)
Extended master secret: no
Max Early Data: 0

read R BLOCK DONE


In gefyra docker container:

root@a37c034df362:/# echo | openssl s_client -showcerts -servername kubernetes -connect 10.43.0.1:443 CONNECTED(00000003) depth=1 CN = k3s-server-ca@1656681228 verify error:num=19:self-signed certificate in certificate chain verify return:1 depth=1 CN = k3s-server-ca@1656681228 verify return:1 depth=0 O = k3s, CN = k3s verify return:1

Certificate chain 0 s:O = k3s, CN = k3s i:CN = k3s-server-ca@1656681228 a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA256 v:NotBefore: Jul 1 13:13:48 2022 GMT; NotAfter: Mar 23 07:10:48 2024 GMT -----BEGIN CERTIFICATE----- MIICWDCCAf+gAwIBAgIIZ+uGCbjCVdAwCgYIKoZIzj0EAwIwIzEhMB8GA1UEAwwY azNzLXNlcnZlci1jYUAxNjU2NjgxMjI4MB4XDTIyMDcwMTEzMTM0OFoXDTI0MDMy MzA3MTA0OFowHDEMMAoGA1UEChMDazNzMQwwCgYDVQQDEwNrM3MwWTATBgcqhkjO PQIBBggqhkjOPQMBBwNCAARIwNcU7SL2kS+L5wlp43bsXfJiHzNEXkvUgQfX4D94 o1eCUZ6QqxG/kqbkjKrPPJjARdGeSp9WPdrLs/c+UWpro4IBIjCCAR4wDgYDVR0P AQH/BAQDAgWgMBMGA1UdJQQMMAoGCCsGAQUFBwMBMB8GA1UdIwQYMBaAFGBhC1Id CwJZCKfa14iF8rtp+IN2MIHVBgNVHREEgc0wgcqCCWdudXBnLm9yZ4IKa3ViZXJu ZXRlc4ISa3ViZXJuZXRlcy5kZWZhdWx0ghZrdWJlcm5ldGVzLmRlZmF1bHQuc3Zj giRrdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWyCCWxvY2FsaG9z dIISdmVpdC10aGlua3BhZC10NDgwhwQKKwABhwR/AAABhwTAqLJDhwTAqAJshwTA qAJuhxAgAwDIHz94wroA5kPnkApshxAAAAAAAAAAAAAAAAAAAAABMAoGCCqGSM49 BAMCA0cAMEQCIDb1UoOkv5pw/u+c7T+2dc6WJgVU1g32QeltwrvY1OSHAiBdjohf 1dO0HUtyxfIIrYxKF9h/7RizAYoL+2Xg8v+UPg== -----END CERTIFICATE----- 1 s:CN = k3s-server-ca@1656681228 i:CN = k3s-server-ca@1656681228 a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA256 v:NotBefore: Jul 1 13:13:48 2022 GMT; NotAfter: Jun 28 13:13:48 2032 GMT -----BEGIN CERTIFICATE----- MIIBdzCCAR2gAwIBAgIBADAKBggqhkjOPQQDAjAjMSEwHwYDVQQDDBhrM3Mtc2Vy dmVyLWNhQDE2NTY2ODEyMjgwHhcNMjIwNzAxMTMxMzQ4WhcNMzIwNjI4MTMxMzQ4 WjAjMSEwHwYDVQQDDBhrM3Mtc2VydmVyLWNhQDE2NTY2ODEyMjgwWTATBgcqhkjO PQIBBggqhkjOPQMBBwNCAATYNXGbR6z8kGvyybn4w2eSP+VvU3bRJHKdOp2InSiT dIhwD0h3+js1M6QpQujCiLOQujmEpkXLNqOGnmyz74TYo0IwQDAOBgNVHQ8BAf8E BAMCAqQwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUYGELUh0LAlkIp9rXiIXy u2n4g3YwCgYIKoZIzj0EAwIDSAAwRQIhAOzNwzyqk2VGvhL7sOpz7eKT2qeWwI/l 6o0copjOS+kqAiB3z2hKB7GwtE/R8BJ7gLzlNAMLjO3Bjju5Xl/6dzr2cg== -----END CERTIFICATE-----

Server certificate subject=O = k3s, CN = k3s issuer=CN = k3s-server-ca@1656681228

No client certificate CA names sent Requested Signature Algorithms: RSA-PSS+SHA256:ECDSA+SHA256:Ed25519:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA384:ECDSA+SHA512:RSA+SHA1:ECDSA+SHA1 Shared Requested Signature Algorithms: RSA-PSS+SHA256:ECDSA+SHA256:Ed25519:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA384:ECDSA+SHA512 Peer signing digest: SHA256 Peer signature type: ECDSA Server Temp Key: X25519, 253 bits

SSL handshake has read 1410 bytes and written 406 bytes Verification error: self-signed certificate in certificate chain

New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256 Server public key is 256 bit Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE No ALPN negotiated Early data was not sent Verify return code: 19 (self-signed certificate in certificate chain)


Post-Handshake New Session Ticket arrived: SSL-Session: Protocol : TLSv1.3 Cipher : TLS_AES_128_GCM_SHA256 Session-ID: CA378D595E4D76C414C28123694911AF76E0070CC228DAE3AA60F64F70E05FE9 Session-ID-ctx: Resumption PSK: 4CA4356C57C0A29F9367CAED2067A61EA16ACD211A827EA6978F04F3AB39FB44 PSK identity: None PSK identity hint: None SRP username: None TLS session ticket lifetime hint: 604800 (seconds) TLS session ticket: 0000 - a5 31 e1 7c 09 37 6a 8b-92 3c 3c 7c 27 94 8b d6 .1.|.7j..<<|'... 0010 - 5e 51 96 4d 67 6b b3 c7-36 f1 da c5 83 fe 2c c5 ^Q.Mgk..6.....,. 0020 - bd c9 a9 93 b7 e6 cb 2c-aa fa 38 19 59 d8 66 57 .......,..8.Y.fW 0030 - 4f da 79 85 3c 3e 0b fc-7e cb 35 ba 5a a9 92 4a O.y.<>..~.5.Z..J 0040 - 54 f2 23 e0 ab e1 f5 a7-42 93 25 d9 41 3f 06 70 T.#.....B.%.A?.p 0050 - c6 10 7f 53 cc 23 f5 b1-f2 13 f9 91 84 bf 01 8f ...S.#.......... 0060 - 9f 83 c8 9a 90 fb d0 61-20 bf 6c f7 20 23 7a e2 .......a .l. #z. 0070 - e8 .

Start Time: 1679653803
Timeout   : 7200 (sec)
Verify return code: 19 (self-signed certificate in certificate chain)
Extended master secret: no
Max Early Data: 0

read R BLOCK DONE

schwobaseggl commented 1 year ago

@Schille With the new Laptop and a clean environment, it now works for me, too. Shall we keep this shelved or move forward with implementing it along this way?