lwolf / kube-cleanup-operator

Kubernetes Operator to automatically delete completed Jobs and their Pods
MIT License
503 stars 109 forks source link

make client-go log to console instead of file #41

Closed thomasdupas closed 4 years ago

thomasdupas commented 4 years ago

The Kubernetes client-go package was logging to file instead of console (klog defaults), which is less than ideal. Noticed this by "accident" when applying a stricter PSP which required the rootFs to be read-only.

/ $ ls -lah /tmp/
total 8
drwxrwsrwx    2 root     10000        294 May  9 18:33 .
drwxr-xr-x    1 root     root          17 May  9 18:33 ..
lrwxrwxrwx    1 10000    10000         99 May  9 18:33 kube-cleanup-operator.INFO -> kube-cleanup-operator.kube-cleanup-operator-64f6f69bdc-qw8xz.unknownuser.log.INFO.20200509-183302.1
lrwxrwxrwx    1 10000    10000        102 May  9 18:33 kube-cleanup-operator.WARNING -> kube-cleanup-operator.kube-cleanup-operator-64f6f69bdc-qw8xz.unknownuser.log.WARNING.20200509-183302.1
-rw-r--r--    1 10000    10000        364 May  9 18:33 kube-cleanup-operator.kube-cleanup-operator-64f6f69bdc-qw8xz.unknownuser.log.INFO.20200509-183302.1
-rw-r--r--    1 10000    10000        364 May  9 18:33 kube-cleanup-operator.kube-cleanup-operator-64f6f69bdc-qw8xz.unknownuser.log.WARNING.20200509-183302.1

/ $ cat /tmp/kube-cleanup-operator.INFO 
Log file created at: 2020/05/09 18:33:02
Running on machine: kube-cleanup-operator-64f6f69bdc-qw8xz
Binary: Built with gc go1.13 for linux/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
W0509 18:33:02.090551       1 client_config.go:541] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.

/ $ cat /tmp/kube-cleanup-operator.WARNING 
Log file created at: 2020/05/09 18:33:02
Running on machine: kube-cleanup-operator-64f6f69bdc-qw8xz
Binary: Built with gc go1.13 for linux/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
W0509 18:33:02.090551       1 client_config.go:541] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.

by setting the klog flags ourselves we can steer this behaviour, getting those logs in the same stream as the app logging (and being able to mount the rootFs read-only, which for us at least was a plus, without creating an emptyDir just for this)

lwolf commented 4 years ago

Hi, thanks for the PR. could you please attach the PSP and deployment you're using so I can reproduce and test

thomasdupas commented 4 years ago

kube-cleanup-operator-0.0.3.zip Hi, see attached tgz for the helm chart I'm using, repackaged as zip since GitHub doesn't allow tgz (I could add it to this PR, or make another PR, if you think it could be of use for others). the default restricted PSP we use, although it can be tested with the Deployment as such since it's also specified in the securityContext of the pod.

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: '0'
spec:
  readOnlyRootFilesystem: true
  privileged: false
  hostIPC: false
  hostPID: false
  hostNetwork: false
  hostPorts: []
  allowedHostPaths: []
  allowPrivilegeEscalation: false
  allowedCapabilities: []
  requiredDropCapabilities:
    - all
  runAsUser:
    rule: 'MustRunAs'
    ranges:
      - min: 10000
        max: 65536
  supplementalGroups:
    rule: 'MustRunAs'
    ranges:
      - min: 10000
        max: 65536
  fsGroup:
    rule: 'MustRunAs'
    ranges:
      - min: 10000
        max: 65536
  seLinux:
    rule: 'RunAsAny'
  volumes:
    - configMap
    - downwardAPI
    - emptyDir
    - persistentVolumeClaim
    - secret
    - projected
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: psp.default
rules:
  - apiGroups:
      - policy
    resourceNames:
      - '0'
    resources:
      - podsecuritypolicies
    verbs:
      - use
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: psp.default:authenticated
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: psp.default
subjects:
  - kind: Group
    apiGroup: rbac.authorization.k8s.io
    name: system:authenticated
thomasdupas commented 4 years ago

I did a go mod vendor, but that didn't produce any changes (the k8s.io/klog was already in go.mod, as dependency for the other k8s.io packages)