search

redhat-cop / cert-utils-operator

Set of functionalities around certificates packaged in a Kubernetes operator
Apache License 2.0
94 stars 35 forks source link

Operator logs access token #153

Closed lukeelten closed 8 months ago

lukeelten commented 9 months ago

I installed the cert-utils operator from OperatorHub on OpenShift. In default configuration it logs valid access token to the container log output.

OpenShift Version: 4.12.40 Operator Version: 1.3.11 Source: OperatorHub

This is problematic because a) it bypasses the security of the service account secrets and b) we store logs in a central log store which now contains valid secrets.

Expected behavior: The operator should not output any secret information, including access token, to the container log stream.

Example log output:

I1124 08:34:29.524403       1 round_trippers.go:443] POST https://<KUBERNETES API>:443/apis/authentication.k8s.io/v1/tokenreviews 201 Created in 2 milliseconds
I1124 08:34:29.524460       1 round_trippers.go:449] Response Headers:
I1124 08:34:29.524465       1 round_trippers.go:452]     Date: Fri, 24 Nov 2023 08:34:29 GMT
I1124 08:34:29.524469       1 round_trippers.go:452]     Audit-Id: 6be7eb43-f0cb-4fff-b870-af4a18f2edb7
I1124 08:34:29.524472       1 round_trippers.go:452]     Cache-Control: no-cache, private
I1124 08:34:29.524475       1 round_trippers.go:452]     Content-Type: application/json
I1124 08:34:29.524479       1 round_trippers.go:452]     Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
I1124 08:34:29.524482       1 round_trippers.go:452]     X-Kubernetes-Pf-Flowschema-Uid: ccae85e0-c2c6-4c72-93e5-edb0879cf529
I1124 08:34:29.524486       1 round_trippers.go:452]     X-Kubernetes-Pf-Prioritylevel-Uid: 93718d61-0ceb-4eac-9ce5-4dacad0cded8
I1124 08:34:29.524489       1 round_trippers.go:452]     Content-Length: 2144
I1124 08:34:29.524542       1 request.go:1097] Response Body: {"kind":"TokenReview","apiVersion":"authentication.k8s.io/v1","metadata":{"creationTimestamp":null,"managedFields":[{"manager":"kube-rbac-proxy","operation":"Update","apiVersion":"authentication.k8s.io/v1","time":"2023-11-24T08:34:29Z","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:token":{}}}}]},"spec":{"token":"eyJhbGciOiJSUzI1NiIsImtpZCI6InNiVFlkT2pqNF9PZ3pLZ0J5Um0tejRhWDl3SG1BUjNkQUlTc0NaNDFxaWMifQ<REST OF TOKEN>"},"status":{"authenticated":true,"user":{"username":"system:serviceaccount:openshift-monitoring:prometheus-k8s","uid":"743aa7b0-6d8e-47f7-ba0e-517041d56f2b","groups":["system:serviceaccounts","system:serviceaccounts:openshift-monitoring","system:authenticated"],"extra":{"authentication.kubernetes.io/pod-name":["prometheus-k8s-1"],"authentication.kubernetes.io/pod-uid":["f9519d8e-5f3f-461d-b96e-cc2107d2dfdc"]}},"audiences":["https://kubernetes.default.svc"]}}
I1124 08:35:53.240019       1 request.go:1097] Request Body: {"kind":"TokenReview","apiVersion":"authentication.k8s.io/v1","metadata":{"creationTimestamp":null},"spec":{"token":"eyJhbGciOiJSUzI1NiIsImtpZCI6InNiVFlkT2pqNF9PZ3pLZ0J5Um0tejRhWDl3SG1BUjNkQUlTc0NaNDFxaWMifQ.<REST OF TOKEN>"},"status":{"user":{}}}
I1124 08:35:53.240208       1 round_trippers.go:423] curl -k -v -XPOST  -H "Accept: application/json, */*" -H "Content-Type: application/json" -H "User-Agent: kube-rbac-proxy/v0.0.0 (linux/amd64) kubernetes/$Format" -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6InNiVFlkT2pqNF9PZ3pLZ0J5Um0tejRhWDl3SG1BUjNkQUlTc0NaNDFxaWMifQ.<REST OF TOKEN>" 'https://192.168.0.1:443/apis/authentication.k8s.io/v1/tokenreviews'
I1124 08:35:53.243705       1 round_trippers.go:443] POST https://<KUBERNETES API>:443/apis/authentication.k8s.io/v1/tokenreviews 201 Created in 3 milliseconds

Test Login with access token

oc login --token="eyJhbGciOiJSUzI1NiIsImtpZCI6InNiVFlkT2pqNF9PZ3pLZ0J5Um0tejRhWDl3SG1BUjNkQUlTc0NaNDFxaWMifQ.<REST OF TOKEN>" --server=https://api.<CLUSTERNAME>:6443

Logged into "https://api.<CLUSTERNAME>:6443" as "system:serviceaccount:cert-utils-operator:controller-manager" using the token provided.

You have access to XXX projects, the list has been suppressed. You can list all projects with 'oc projects'

Using project "".
mathianasj commented 8 months ago

Hi, thank you for reporting this I will be working on this today and hope to get a fix shortly.

mathianasj commented 8 months ago

Hi @lukeelten , can you provide any details on what features you are using, route certificate, truststore creation, etc? I am trying to duplicate the issue so that I can confirm a fix.

lukeelten commented 8 months ago

We use the operator mainly to copy certificates from secrets to routes. I think some users use it to create java keystores or to inject CA bundles.

lukeelten commented 8 months ago

I guess I missed an important detail: The log output comes from the kube-rbac-proxy sidecar container.

So on a default installation of the operator the sidecar is deployed with -v=10 verbose mode, which logs all HTTP requests including the access token.

mathianasj commented 8 months ago

Hi @lukeelten,

I am still struggling to duplicate the issue, could you provide the image of the sidecar proxy? Mine is currently using the following to try to duplicate

quay.io/redhat-cop/cert-utils-operator@sha256:57567c570c6d2c3b0f5cf2eef7526e302396f313503735fee2c5eb1c3a21ba8d

quay.io/redhat-cop/kube-rbac-proxy@sha256:c68135620167c41e3d9f6c1d2ca1eb8fa24312b86186d09b8010656b9d25fb47

lukeelten commented 8 months ago

This is the deployment a brand new installation of the operator creates. The images are:

The problematic line appears to be --v=10 on the kube-rbac-proxy sidecar which automatically logs all incoming HTTP requests including all headers.

kind: Deployment
apiVersion: apps/v1
metadata:
  name: cert-utils-operator-controller-manager
  namespace: cert-utils-operator
  ownerReferences:
    - apiVersion: operators.coreos.com/v1alpha1
      kind: ClusterServiceVersion
      name: cert-utils-operator.v1.3.11
      uid: dc59b5e8-6ce4-406b-b9e4-2bd97784d783
      controller: false
      blockOwnerDeletion: false
  labels:
    olm.deployment-spec-hash: 7cbf8f97bf
    olm.owner: cert-utils-operator.v1.3.11
    olm.owner.kind: ClusterServiceVersion
    olm.owner.namespace: cert-utils-operator
    operators.coreos.com/cert-utils-operator.cert-utils-operator: ''
spec:
  replicas: 1
  selector:
    matchLabels:
      control-plane: cert-utils-operator
  template:
    metadata:
      creationTimestamp: null
      labels:
        control-plane: cert-utils-operator
      annotations:
        operators.operatorframework.io/builder: operator-sdk-v1.8.0+git
        operators.operatorframework.io/project_layout: go.kubebuilder.io/v3
        certified: 'false'
        olm.targetNamespaces: ''
        operatorframework.io/properties: >-
          {"properties":[{"type":"olm.package","value":{"packageName":"cert-utils-operator","version":"1.3.11"}}]}
        repository: 'https://github.com/redhat-cop/cert-utils-operator'
        support: Best Effort
        operatorframework.io/cluster-monitoring: 'true'
        operators.openshift.io/infrastructure-features: '["Disconnected"]'
        alm-examples: '[]'
        capabilities: Deep Insights
        olm.operatorNamespace: cert-utils-operator
        containerImage: >-
          quay.io/redhat-cop/cert-utils-operator@sha256:57567c570c6d2c3b0f5cf2eef7526e302396f313503735fee2c5eb1c3a21ba8d
        createdAt: '2023-04-26T16:55:05Z'
        categories: Security
        operatorframework.io/suggested-namespace: cert-utils-operator
        description: Set of utilities for TLS certificates
        olm.operatorGroup: cert-utils-operator
    spec:
      restartPolicy: Always
      serviceAccountName: controller-manager
      schedulerName: default-scheduler
      terminationGracePeriodSeconds: 10
      securityContext: {}
      containers:
        - resources: {}
          terminationMessagePath: /dev/termination-log
          name: kube-rbac-proxy
          env:
            - name: OPERATOR_CONDITION_NAME
              value: cert-utils-operator.v1.3.11
          ports:
            - name: https
              containerPort: 8443
              protocol: TCP
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - name: tls-cert
              mountPath: /etc/certs/tls
          terminationMessagePolicy: File
          image: >-
            quay.io/redhat-cop/kube-rbac-proxy@sha256:c68135620167c41e3d9f6c1d2ca1eb8fa24312b86186d09b8010656b9d25fb47
          args:
            - '--secure-listen-address=0.0.0.0:8443'
            - '--upstream=http://127.0.0.1:8080/'
            - '--logtostderr=true'
            - '--v=10'
            - '--tls-cert-file=/etc/certs/tls/tls.crt'
            - '--tls-private-key-file=/etc/certs/tls/tls.key'
        - resources:
            requests:
              cpu: 100m
              memory: 20Mi
          readinessProbe:
            httpGet:
              path: /readyz
              port: 8081
              scheme: HTTP
            initialDelaySeconds: 5
            timeoutSeconds: 1
            periodSeconds: 10
            successThreshold: 1
            failureThreshold: 3
          terminationMessagePath: /dev/termination-log
          name: manager
          command:
            - /manager
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8081
              scheme: HTTP
            initialDelaySeconds: 15
            timeoutSeconds: 1
            periodSeconds: 20
            successThreshold: 1
            failureThreshold: 3
          env:
            - name: OPERATOR_CONDITION_NAME
              value: cert-utils-operator.v1.3.11
          securityContext:
            allowPrivilegeEscalation: false
          imagePullPolicy: IfNotPresent
          terminationMessagePolicy: File
          image: >-
            quay.io/redhat-cop/cert-utils-operator@sha256:57567c570c6d2c3b0f5cf2eef7526e302396f313503735fee2c5eb1c3a21ba8d
          args:
            - '--health-probe-bind-address=:8081'
            - '--metrics-bind-address=127.0.0.1:8080'
            - '--leader-elect'
      serviceAccount: controller-manager
      volumes:
        - name: tls-cert
          secret:
            secretName: cert-utils-operator-certs
            defaultMode: 420
      dnsPolicy: ClusterFirst
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 25%
      maxSurge: 25%
  revisionHistoryLimit: 1
  progressDeadlineSeconds: 600