search

airshipit / treasuremap

Reference Airship manifests, CICD, and reference architecture.
http://openstack.org
Apache License 2.0
52 stars 39 forks source link

Deliver Dex helm charts (Helm Operator) #19

Closed lb4368 closed 3 years ago

lb4368 commented 4 years ago

To support software stack deployment of an OIDC provider, create a manifest function to allow delivery of Dex from treasuremap.

Proposed change Complete the patchset here (https://review.opendev.org/#/c/733647/) to deliver the charts for Dex. Then utilize the flux helm operator in airshipctl (airshipit/airshipctl#177) to create a manifest function to deliver Dex to a cluster. Implement proper gating to validate delivery.

lb4368 commented 4 years ago

Blocked by #26

lb4368 commented 4 years ago

Dependent issue #26 is closed so removing blocked label.

sshiba commented 3 years ago

I am willing to work on this issue. Please, assign it to me. Thanks.

drewwalters96 commented 3 years ago

@sshiba all yours. Thanks!

sshiba commented 3 years ago

I was able to run the PS https://review.opendev.org/c/airship/charts/+/770677 on my PC. Once the credentials retrieved using the client app, tested the authorization aspects (role/cluster role/role binding) with success.

Tried to deploy Dex with connectors for Github and Google IdPs for testing purposes. o Github IdP: got it working once but failed during the callback. It might be the callback url is not correctly configured o Google IdP: failed to get dex pod running (CrashLoopBack) and might be due to “connector” configuration.

Also, searched for a way to login using command line for script base validation. Found a go program that might work but didn't test it yet. The go code can be found here: https://github.com/argoproj/argo-cd/blob/master/cmd/argocd/commands/login.go.

Meanwhile, will now work on deploying manually Dex on a CAPZ target cluster.

sshiba commented 3 years ago

Started the work on deploying Dex on a Target cluster deploy on Azure Cloud using airshipctl phase plan.

Todo: determine how to configure the template to “load” the Dex certificate that will be used by API server, e.g., oidc-ca-file: /etc/Kubernetes/pki/ca.crt.

File: manifests/function/k8scontrol-capz/v0.4.9/controlplane.yaml

apiVersion: controlplane.cluster.x-k8s.io/v1alpha3
kind: KubeadmControlPlane
metadata:
  name: target-cluster-control-plane
  namespace: default
spec:
  infrastructureTemplate:
    apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
    kind: AzureMachineTemplate
    name: target-cluster-control-plane
  kubeadmConfigSpec:
    clusterConfiguration:
      apiServer:
        extraArgs:
          cloud-config: /etc/kubernetes/azure.json
          cloud-provider: azure
          oidc-issuer-url: https://dex.jarvis.local:5556/dex
          oidc-client-id: jarvis-kubernetes        
          oidc-username-claim: email        
          oidc-username-prefix: "oidc:"        
          oidc-groups-claim: groups        
          oidc-ca-file: /etc/kubernetes/pki/apiserver.crt        
        extraVolumes:
        - hostPath: /etc/kubernetes/azure.json
          mountPath: /etc/kubernetes/azure.json

Result: Describe API Server Pod below:

esidshi@Airship2:~/projects/765629/airshipctl$ kubectl --kubeconfig /tmp/edge-5g-cluster.kubeconfig describe pod kube-apiserver-edge-5g-cluster-control-plane-hflb4 -n kube-system
Name:                 kube-apiserver-edge-5g-cluster-control-plane-hflb4
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 edge-5g-cluster-control-plane-hflb4/10.0.0.4
Start Time:           Thu, 28 Jan 2021 18:35:20 -0600
Labels:               component=kube-apiserver
                      tier=control-plane
Annotations:          kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 10.0.0.4:6443
                      kubernetes.io/config.hash: d4c661669c4d0f5691ee4f476423a9d1
                      kubernetes.io/config.mirror: d4c661669c4d0f5691ee4f476423a9d1
                      kubernetes.io/config.seen: 2021-01-29T00:35:19.228234595Z
                      kubernetes.io/config.source: file
Status:               Running
IP:                   10.0.0.4
IPs:
  IP:           10.0.0.4
Controlled By:  Node/edge-5g-cluster-control-plane-hflb4
Containers:
  kube-apiserver:
    Container ID:  containerd://b93cd0472700eb684609938cfbbc5357e8e3f15557668f860c15937efe21c296
    Image:         k8s.gcr.io/kube-apiserver:v1.18.10
    Image ID:      k8s.gcr.io/kube-apiserver@sha256:ff1e3568d00c3f6ed45e9d9032a1099ca44a3919afdfe6e709978b980d6de43f
    Port:          <none>
    Host Port:     <none>
    Command:
      kube-apiserver
      --advertise-address=10.0.0.4
      --allow-privileged=true
      --authorization-mode=Node,RBAC
      --client-ca-file=/etc/kubernetes/pki/ca.crt
      --cloud-config=/etc/kubernetes/azure.json
      --cloud-provider=azure
      --enable-admission-plugins=NodeRestriction
      --enable-bootstrap-token-auth=true
      --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
      --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
      --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
      --etcd-servers=https://127.0.0.1:2379
      --insecure-port=0
      --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
      --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
      --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
      --oidc-ca-file=/etc/kubernetes/pki/apiserver.crt
      --oidc-client-id=jarvis-kubernetes
      --oidc-groups-claim=groups
      --oidc-issuer-url=https://dex.jarvis.local:5556/dex
      --oidc-username-claim=email
      --oidc-username-prefix=oidc:
      --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
      --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
      --requestheader-allowed-names=front-proxy-client
      --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
      --requestheader-extra-headers-prefix=X-Remote-Extra-
      --requestheader-group-headers=X-Remote-Group
      --requestheader-username-headers=X-Remote-User
      --secure-port=6443
      --service-account-key-file=/etc/kubernetes/pki/sa.pub
      --service-cluster-ip-range=10.96.0.0/12
      --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
      --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    State:          Running
      Started:      Thu, 28 Jan 2021 18:34:57 -0600
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:        250m
    Liveness:     http-get https://10.0.0.4:6443/healthz delay=15s timeout=15s period=10s #success=1 #failure=8
    Environment:  <none>
    Mounts:
      /etc/ca-certificates from etc-ca-certificates (ro)
      /etc/kubernetes/azure.json from cloud-config (ro)
      /etc/kubernetes/pki from k8s-certs (ro)
      /etc/ssl/certs from ca-certs (ro)
      /usr/local/share/ca-certificates from usr-local-share-ca-certificates (ro)
      /usr/share/ca-certificates from usr-share-ca-certificates (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  ca-certs:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/ssl/certs
    HostPathType:  DirectoryOrCreate
  cloud-config:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes/azure.json
    HostPathType:  
  etc-ca-certificates:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/ca-certificates
    HostPathType:  DirectoryOrCreate
  k8s-certs:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes/pki
    HostPathType:  DirectoryOrCreate
  usr-local-share-ca-certificates:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/local/share/ca-certificates
    HostPathType:  DirectoryOrCreate
  usr-share-ca-certificates:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/share/ca-certificates
    HostPathType:  DirectoryOrCreate
QoS Class:         Burstable
Node-Selectors:    <none>
Tolerations:       :NoExecute
Events:
  Type     Reason     Age    From                                          Message
  ----     ------     ----   ----                                          -------
  Normal   Pulled     5m11s  kubelet, edge-5g-cluster-control-plane-hflb4  Container image "k8s.gcr.io/kube-apiserver:v1.18.10" already present on machine
  Warning  Unhealthy  3m38s  kubelet, edge-5g-cluster-control-plane-hflb4  Liveness probe failed: HTTP probe failed with statuscode: 500
sshiba commented 3 years ago

Issue: When trying the authenticator web app on a browser, the request to load it does not reach the Dex service. I think the issue is that the port 30556 (NodePort) is not opened in the Control Plane node. Will continue troubleshooting Monday.

I have documented the integration process here.

sshiba commented 3 years ago

Troubleshooting with Airship Spend the day trying to access the NodePortservice on a Target cluster deployed on Azure cloud using airshipctl. Tried many things such as creating a simple NGINX application without success. Engaged the Azure team on CAPZ slack community but it didn’t help. Tried also creating an NGINX application with LoadBalancerand no success either.

Troubleshooting with Cluster API Switched to clusterctlto create the Target cluster as troubleshooting strategy. Negative results with NodePortbut it worked with LoadBalancer.

Reconfigured the CAPI manifest to include the API server flags required for supporting OIDC and redeployed the target cluster with “clusterctl”. When creating the secrets for Dex (tools/gate/dex/200-deploy-certmanger-target.sh), it failed when creating the ClusterIssuerObject. Not sure why and will continue investigating it tomorrow.

Alternatives: If things don’t move with Azure, will try on GCP (May be)

sshiba commented 3 years ago

Still working on getting Dex installed in a Target cluster deployed on Azure Cloud using “clusterctl”. The ClusterIssuer issue from yesterday was due to using the clusterctl cert-manager. Instead, it required the jetstack cert-manager as issue identified during the Airship Voyager DSU technical discussion.

Got a new error (see below) while deploying the certificates for Dex and will continue troubleshooting it tomorrow.

Error from server (InternalError): error when creating "/tmp/ca-issuers.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s: context deadline exceeded

sshiba commented 3 years ago

Started the work for deploying Dex using the Helm Operator, as follow:

TODO:

sshiba commented 3 years ago

Today’s activity

TODO: move the dex-chart-collator, dex-helm-operator and API server configurations (kustomization for KubeadmControlPlane object) to Treasuremap/function.

sshiba commented 3 years ago

Dex implementation requires three patchsets: two of them are supporting patchsets and one for Dex itself:

TODO:

eak13 commented 3 years ago

Please ensure that this is included in the airship-core type deployment

michaelfix commented 3 years ago

@sshiba - Sidney, any update to this issue? Are we close(r) to finishing?

sshiba commented 3 years ago

After many re-structuring of implementation, the patchset (https://review.opendev.org/c/airship/treasuremap/+/780123) implementing issue #19 has been reviewed and being merged.

lb4368 commented 3 years ago

@sshiba I was just looking at your patchset above. You have added Dex to the multi-tenant type. I believe we want Dex as part of the airship-core type which would be inherited by the multi-tenant type. Is there a plan to update this?

sshiba commented 3 years ago

I recall being asked to add for multi-tenant while didn't see the comment requesting to add it to airship-core. Will need to create a new patchset to address this change.

sshiba commented 3 years ago

Created the patchset https://review.opendev.org/c/airship/treasuremap/+/781172 moving the OIDC API server kustomization to airship-core.

sshiba commented 3 years ago

https://review.opendev.org/c/airship/treasuremap/+/781172 has been merged. But a new PS was created to address the issue of "illegal based64 data input" (https://review.opendev.org/c/airship/treasuremap/+/783376)

sshiba commented 3 years ago

Integration of API server/Dex/LDAP has been demonstrated using the following patchsets:

These patchsets do not support the encryption of LDAP password and CA certificate for API server has not been signed by a known authority and secure it.

Note: in order to validate the deployment of dex-aio microservice through flux/helm operator + helm collator, the charts patchset needs to be merged.

sshiba commented 3 years ago

The following patchset are ready for review:

This document (https://hackmd.io/nvUupD15RlabSuahSwDOKA) describes how to validate these patchset.

sshiba commented 3 years ago

Found a workaround for supporting catalogues and PS https://review.opendev.org/c/airship/treasuremap/+/791835 was created but still work-in-progress and not tested yet.

lb4368 commented 3 years ago

Closing per merge of below patchsets...