search

ressu / kube-plex

Scalable Plex Media Server on Kubernetes -- dispatch transcode jobs as pods on your cluster!
Apache License 2.0
107 stars 23 forks source link

Collect kubernetes events to Plex log #44

Open ressu opened 1 year ago

ressu commented 1 year ago

There is an issue with visibility when something goes wrong. One option to solve this issue would be to collect the events from kubernetes API to Plex log. Since logging framework already exists, this should be as simple as just watching for the relevant events.

The most important collection point would be when an error occurs, but I don't think it hurts to collect everything. We can filter the messages later if things get too noisy.

lknite commented 1 week ago

Maybe use something like fluentbit and send to a log aggregator such as elastic or similar?

ressu commented 1 week ago

Fluentbit will definitely solve the issue of having the logs available somewhere. What I really wanted to make happen is have all the logs in the same place. That way you could check a single log for all possible issues that might be affecting the transcoding pipeline.

lknite commented 1 week ago

I see the plex api has an end point for logging a single message. We could add a long lasting side car (initContainer) that only watches for events and logs those. We'd just need that along with the rbac needed.

As a side car, its something that could be optionally enabled/disabled via the helm chart when debugging.

I have an initContainer that uses the kubeapi to wait for a secret to appear... should be able to put something together pretty easy, maybe initially just use curl and pipe the output to the api.

I suppose, it doesn't need to be an initContainer, we could just add something to plex itself.

ressu commented 1 week ago

As long as the container starts, we can collect the logs. That's actually what the launcher does today. As soon as it starts it connects back to the main container and starts logging. It currently doesn't collect the stdout/stderr from the transcode process, but the transcode process already does enough logging. I consider that portion solved.

And if there was an issue with the startup that the transcode process doesn't report back, we could collect the stdout/stderr and pass them to the logger here

https://github.com/ressu/kube-plex/blob/472229321407052afb2b26f24aa8b63347ad1a44/cmd/transcode-launcher/main.go#L79-L82

What hasn't been solved is the kubernetes job handling. Let's say we configure hardware transcoding but end up starting more jobs than we have hardware for. Those jobs will now block until resources come available. The information about the startup blockers is in the Kubernetes events, but that data isn't logged anywhere. In fact, even with something like fluentd that log isn't immediately obvious and you need to look at the scheduler logs to understand why something isn't scheduling. The sidecar wouldn't really help here either since the sidecar would be blocked along with the rest of the job.

lknite commented 1 week ago

Here's an example of an initContainer that uses the kubeapi:

  initContainers:
  - name: wait-for-secret
    image: radial/busyboxplus
    imagePullPolicy: Always
    command: ['sh', '-c']
    args:
    - |
      # Uncomment to cause local generation of keys, rather than wait for keys from vault
      #return 0

      # Point to the internal API server hostname
      APISERVER=https://kubernetes.default.svc

      # Path to ServiceAccount token
      SERVICEACCOUNT=/var/run/secrets/kubernetes.io/serviceaccount

      # Read this Pod's namespace
      NAMESPACE=$(cat ${SERVICEACCOUNT}/namespace)

      # Read the ServiceAccount bearer token
      TOKEN=$(cat ${SERVICEACCOUNT}/token)

      # Reference the internal certificate authority (CA)
      CACERT=${SERVICEACCOUNT}/ca.crt

      while [ 200 -ne $(curl --cacert ${CACERT} --header "Authorization: Bearer ${TOKEN}" \
        -X GET ${APISERVER}/api/v1/namespaces/sealed-secrets/secrets/sealed-secrets-shared-key \
        --head \
        --silent \
        --output /dev/null \
        --write-out '%{http_code}') ]
      do
        echo "waiting for secret: sealed-secrets-shared-key"
        sleep 5
      done
      echo "waiting for secret: sealed-secrets-shared-key"
      echo "  detected"

along with the required rbac rules needed to give the required access:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-secrets
subjects:
# You can specify more than one "subject"
- kind: ServiceAccount
  name: default
roleRef:
  kind: Role
  name: secret-get
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: secret-get
rules:
- apiGroups: [""] # "" indicates the core API group
  resources: ["secrets"]
  verbs: ["get"]

I originally got the strategy from here: https://kubernetes.io/docs/tasks/run-application/access-api-from-pod/

And here's a starter guide to the kubeapi api: https://kubernetes.io/docs/reference/using-api/api-concepts/

lknite commented 1 week ago

If that were an initContainer on the main plex image, and we adjusted the RBAC to watch for events instead of secrets, then we should be able to watch for events and doing something with them ... just write to console at first to see its working, then log via the plex api

I'll create a deployment to try it out ...

lknite commented 1 week ago

This works to watch events:

Example output:

{"type":"ADDED","object":{"kind":"Event","apiVersion":"v1","metadata":{"name":"testjob.18065a7f6b33194e","namespace":"plex","uid":"a50e4e5d-b0b3-476e-9a15-d649d1f95c18","resourceVersion":"6314819","creationTimestamp":"2024-11-09T16:43:30Z","managedFields":[{"manager":"kube-controller-manager","operation":"Update","apiVersion":"v1","time":"2024-11-09T16:43:30Z","fieldsType":"FieldsV1","fieldsV1":{"f:count":{},"f:firstTimestamp":{},"f:involvedObject":{},"f:lastTimestamp":{},"f:message":{},"f:reason":{},"f:reportingComponent":{},"f:source":{"f:component":{}},"f:type":{}}}]},"involvedObject":{"kind":"Job","namespace":"plex","name":"testjob","uid":"e8b4c58c-3162-47cb-8ab3-af44a8bcfe13","apiVersion":"batch/v1","resourceVersion":"6314815"},"reason":"SuccessfulCreate","message":"Created pod: testjob-hm6dp","source":{"component":"job-controller"},"firstTimestamp":"2024-11-09T16:43:30Z","lastTimestamp":"2024-11-09T16:43:30Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"job-controller","reportingInstance":""}}
{"type":"ADDED","object":{"kind":"Event","apiVersion":"v1","metadata":{"name":"testjob-hm6dp.18065a7f6baf8ddc","namespace":"plex","uid":"1f0fb287-cc52-4855-b7dc-b2739b1d0e9c","resourceVersion":"6314820","creationTimestamp":"2024-11-09T16:43:30Z","managedFields":[{"manager":"kube-scheduler","operation":"Update","apiVersion":"events.k8s.io/v1","time":"2024-11-09T16:43:30Z","fieldsType":"FieldsV1","fieldsV1":{"f:action":{},"f:eventTime":{},"f:note":{},"f:reason":{},"f:regarding":{},"f:reportingController":{},"f:reportingInstance":{},"f:type":{}}}]},"involvedObject":{"kind":"Pod","namespace":"plex","name":"testjob-hm6dp","uid":"78249ea2-ae9a-4e6f-940e-c32bf0144a09","apiVersion":"v1","resourceVersion":"6314816"},"reason":"Scheduled","message":"Successfully assigned plex/testjob-hm6dp to k-prod-w3","source":{},"firstTimestamp":null,"lastTimestamp":null,"type":"Normal","eventTime":"2024-11-09T16:43:30.084542Z","action":"Binding","reportingComponent":"default-scheduler","reportingInstance":"default-scheduler-k-prod-c3"}}
{"type":"ADDED","object":{"kind":"Event","apiVersion":"v1","metadata":{"name":"testjob-hm6dp.18065a7f89cad2c9","namespace":"plex","uid":"9495ae3b-dbb7-48c8-b1ab-8a60e8c64c53","resourceVersion":"6314825","creationTimestamp":"2024-11-09T16:43:30Z","managedFields":[{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2024-11-09T16:43:30Z","fieldsType":"FieldsV1","fieldsV1":{"f:count":{},"f:firstTimestamp":{},"f:involvedObject":{},"f:lastTimestamp":{},"f:message":{},"f:reason":{},"f:reportingComponent":{},"f:reportingInstance":{},"f:source":{"f:component":{},"f:host":{}},"f:type":{}}}]},"involvedObject":{"kind":"Pod","namespace":"plex","name":"testjob-hm6dp","uid":"78249ea2-ae9a-4e6f-940e-c32bf0144a09","apiVersion":"v1","resourceVersion":"6314817","fieldPath":"spec.containers{testjob}"},"reason":"Pulling","message":"Pulling image \"alpine/curl\"","source":{"component":"kubelet","host":"k-prod-w3"},"firstTimestamp":"2024-11-09T16:43:30Z","lastTimestamp":"2024-11-09T16:43:30Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"kubelet","reportingInstance":"k-prod-w3"}}
{"type":"ADDED","object":{"kind":"Event","apiVersion":"v1","metadata":{"name":"testjob-hm6dp.18065a7faf03c008","namespace":"plex","uid":"f1236c00-f607-43c1-83c7-5ef01b344a45","resourceVersion":"6314826","creationTimestamp":"2024-11-09T16:43:31Z","managedFields":[{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2024-11-09T16:43:31Z","fieldsType":"FieldsV1","fieldsV1":{"f:count":{},"f:firstTimestamp":{},"f:involvedObject":{},"f:lastTimestamp":{},"f:message":{},"f:reason":{},"f:reportingComponent":{},"f:reportingInstance":{},"f:source":{"f:component":{},"f:host":{}},"f:type":{}}}]},"involvedObject":{"kind":"Pod","namespace":"plex","name":"testjob-hm6dp","uid":"78249ea2-ae9a-4e6f-940e-c32bf0144a09","apiVersion":"v1","resourceVersion":"6314817","fieldPath":"spec.containers{testjob}"},"reason":"Pulled","message":"Successfully pulled image \"alpine/curl\" in 624ms (624ms including waiting). Image size: 6188514 bytes.","source":{"component":"kubelet","host":"k-prod-w3"},"firstTimestamp":"2024-11-09T16:43:31Z","lastTimestamp":"2024-11-09T16:43:31Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"kubelet","reportingInstance":"k-prod-w3"}}
{"type":"ADDED","object":{"kind":"Event","apiVersion":"v1","metadata":{"name":"testjob-hm6dp.18065a7fb167dff7","namespace":"plex","uid":"0aacb99d-614e-413b-8674-d936adb6eadc","resourceVersion":"6314829","creationTimestamp":"2024-11-09T16:43:31Z","managedFields":[{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2024-11-09T16:43:31Z","fieldsType":"FieldsV1","fieldsV1":{"f:count":{},"f:firstTimestamp":{},"f:involvedObject":{},"f:lastTimestamp":{},"f:message":{},"f:reason":{},"f:reportingComponent":{},"f:reportingInstance":{},"f:source":{"f:component":{},"f:host":{}},"f:type":{}}}]},"involvedObject":{"kind":"Pod","namespace":"plex","name":"testjob-hm6dp","uid":"78249ea2-ae9a-4e6f-940e-c32bf0144a09","apiVersion":"v1","resourceVersion":"6314817","fieldPath":"spec.containers{testjob}"},"reason":"Created","message":"Created container testjob","source":{"component":"kubelet","host":"k-prod-w3"},"firstTimestamp":"2024-11-09T16:43:31Z","lastTimestamp":"2024-11-09T16:43:31Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"kubelet","reportingInstance":"k-prod-w3"}}
{"type":"ADDED","object":{"kind":"Event","apiVersion":"v1","metadata":{"name":"testjob-hm6dp.18065a7fb3de5952","namespace":"plex","uid":"4d85f8f3-8c05-436a-ae45-0a77357d1198","resourceVersion":"6314830","creationTimestamp":"2024-11-09T16:43:31Z","managedFields":[{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2024-11-09T16:43:31Z","fieldsType":"FieldsV1","fieldsV1":{"f:count":{},"f:firstTimestamp":{},"f:involvedObject":{},"f:lastTimestamp":{},"f:message":{},"f:reason":{},"f:reportingComponent":{},"f:reportingInstance":{},"f:source":{"f:component":{},"f:host":{}},"f:type":{}}}]},"involvedObject":{"kind":"Pod","namespace":"plex","name":"testjob-hm6dp","uid":"78249ea2-ae9a-4e6f-940e-c32bf0144a09","apiVersion":"v1","resourceVersion":"6314817","fieldPath":"spec.containers{testjob}"},"reason":"Started","message":"Started container testjob","source":{"component":"kubelet","host":"k-prod-w3"},"firstTimestamp":"2024-11-09T16:43:31Z","lastTimestamp":"2024-11-09T16:43:31Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"kubelet","reportingInstance":"k-prod-w3"}}
{"type":"ADDED","object":{"kind":"Event","apiVersion":"v1","metadata":{"name":"tesjob-mk2p4.18065a80e4c81f39","namespace":"plex","uid":"d5c315dc-0825-4262-afbe-2eccaf121b7d","resourceVersion":"6314870","creationTimestamp":"2024-11-09T16:43:36Z","managedFields":[{"manager":"kube-scheduler","operation":"Update","apiVersion":"events.k8s.io/v1","time":"2024-11-09T16:43:36Z","fieldsType":"FieldsV1","fieldsV1":{"f:action":{},"f:eventTime":{},"f:note":{},"f:reason":{},"f:regarding":{},"f:reportingController":{},"f:reportingInstance":{},"f:type":{}}}]},"involvedObject":{"kind":"Pod","namespace":"plex","name":"tesjob-mk2p4","uid":"f15306ee-c3dc-4a13-aa33-45859488208c","apiVersion":"v1","resourceVersion":"6314860"},"reason":"Scheduled","message":"Successfully assigned plex/tesjob-mk2p4 to k-prod-w3","source":{},"firstTimestamp":null,"lastTimestamp":null,"type":"Normal","eventTime":"2024-11-09T16:43:36.411157Z","action":"Binding","reportingComponent":"default-scheduler","reportingInstance":"default-scheduler-k-prod-c3"}}
{"type":"ADDED","object":{"kind":"Event","apiVersion":"v1","metadata":{"name":"tesjob.18065a80e33abedf","namespace":"plex","uid":"82299e50-c2da-42c4-afd9-7e460776757c","resourceVersion":"6314877","creationTimestamp":"2024-11-09T16:43:36Z","managedFields":[{"manager":"kube-controller-manager","operation":"Update","apiVersion":"v1","time":"2024-11-09T16:43:36Z","fieldsType":"FieldsV1","fieldsV1":{"f:count":{},"f:firstTimestamp":{},"f:involvedObject":{},"f:lastTimestamp":{},"f:message":{},"f:reason":{},"f:reportingComponent":{},"f:source":{"f:component":{}},"f:type":{}}}]},"involvedObject":{"kind":"Job","namespace":"plex","name":"tesjob","uid":"fb86fa3b-484f-46c7-9789-c88756f4718e","apiVersion":"batch/v1","resourceVersion":"6314804"},"reason":"SuccessfulCreate","message":"Created pod: tesjob-mk2p4","source":{"component":"job-controller"},"firstTimestamp":"2024-11-09T16:43:36Z","lastTimestamp":"2024-11-09T16:43:36Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"job-controller","reportingInstance":""}}
{"type":"ADDED","object":{"kind":"Event","apiVersion":"v1","metadata":{"name":"tesjob-mk2p4.18065a815d2979f3","namespace":"plex","uid":"f8b8a710-83a9-446f-8726-b9c2c03a645a","resourceVersion":"6314889","creationTimestamp":"2024-11-09T16:43:38Z","managedFields":[{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2024-11-09T16:43:38Z","fieldsType":"FieldsV1","fieldsV1":{"f:count":{},"f:firstTimestamp":{},"f:involvedObject":{},"f:lastTimestamp":{},"f:message":{},"f:reason":{},"f:reportingComponent":{},"f:reportingInstance":{},"f:source":{"f:component":{},"f:host":{}},"f:type":{}}}]},"involvedObject":{"kind":"Pod","namespace":"plex","name":"tesjob-mk2p4","uid":"f15306ee-c3dc-4a13-aa33-45859488208c","apiVersion":"v1","resourceVersion":"6314867","fieldPath":"spec.containers{tesjob}"},"reason":"Pulling","message":"Pulling image \"alpine/curl\"","source":{"component":"kubelet","host":"k-prod-w3"},"firstTimestamp":"2024-11-09T16:43:38Z","lastTimestamp":"2024-11-09T16:43:38Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"kubelet","reportingInstance":"k-prod-w3"}}
{"type":"ADDED","object":{"kind":"Event","apiVersion":"v1","metadata":{"name":"tesjob-mk2p4.18065a8185be545c","namespace":"plex","uid":"19dee78f-6613-4111-bbc2-24ed8f8bcf6c","resourceVersion":"6314892","creationTimestamp":"2024-11-09T16:43:39Z","managedFields":[{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2024-11-09T16:43:39Z","fieldsType":"FieldsV1","fieldsV1":{"f:count":{},"f:firstTimestamp":{},"f:involvedObject":{},"f:lastTimestamp":{},"f:message":{},"f:reason":{},"f:reportingComponent":{},"f:reportingInstance":{},"f:source":{"f:component":{},"f:host":{}},"f:type":{}}}]},"involvedObject":{"kind":"Pod","namespace":"plex","name":"tesjob-mk2p4","uid":"f15306ee-c3dc-4a13-aa33-45859488208c","apiVersion":"v1","resourceVersion":"6314867","fieldPath":"spec.containers{tesjob}"},"reason":"Pulled","message":"Successfully pulled image \"alpine/curl\" in 680ms (680ms including waiting). Image size: 6188514 bytes.","source":{"component":"kubelet","host":"k-prod-w3"},"firstTimestamp":"2024-11-09T16:43:39Z","lastTimestamp":"2024-11-09T16:43:39Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"kubelet","reportingInstance":"k-prod-w3"}}
{"type":"ADDED","object":{"kind":"Event","apiVersion":"v1","metadata":{"name":"tesjob-mk2p4.18065a81886a18e9","namespace":"plex","uid":"f193602c-d631-4571-8d78-14c993a49aa1","resourceVersion":"6314893","creationTimestamp":"2024-11-09T16:43:39Z","managedFields":[{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2024-11-09T16:43:39Z","fieldsType":"FieldsV1","fieldsV1":{"f:count":{},"f:firstTimestamp":{},"f:involvedObject":{},"f:lastTimestamp":{},"f:message":{},"f:reason":{},"f:reportingComponent":{},"f:reportingInstance":{},"f:source":{"f:component":{},"f:host":{}},"f:type":{}}}]},"involvedObject":{"kind":"Pod","namespace":"plex","name":"tesjob-mk2p4","uid":"f15306ee-c3dc-4a13-aa33-45859488208c","apiVersion":"v1","resourceVersion":"6314867","fieldPath":"spec.containers{tesjob}"},"reason":"Created","message":"Created container tesjob","source":{"component":"kubelet","host":"k-prod-w3"},"firstTimestamp":"2024-11-09T16:43:39Z","lastTimestamp":"2024-11-09T16:43:39Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"kubelet","reportingInstance":"k-prod-w3"}}
{"type":"ADDED","object":{"kind":"Event","apiVersion":"v1","metadata":{"name":"tesjob-mk2p4.18065a818b3d98a8","namespace":"plex","uid":"7598f7d3-c23a-4ce9-86fb-6060ddfc5dd0","resourceVersion":"6314895","creationTimestamp":"2024-11-09T16:43:39Z","managedFields":[{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2024-11-09T16:43:39Z","fieldsType":"FieldsV1","fieldsV1":{"f:count":{},"f:firstTimestamp":{},"f:involvedObject":{},"f:lastTimestamp":{},"f:message":{},"f:reason":{},"f:reportingComponent":{},"f:reportingInstance":{},"f:source":{"f:component":{},"f:host":{}},"f:type":{}}}]},"involvedObject":{"kind":"Pod","namespace":"plex","name":"tesjob-mk2p4","uid":"f15306ee-c3dc-4a13-aa33-45859488208c","apiVersion":"v1","resourceVersion":"6314867","fieldPath":"spec.containers{tesjob}"},"reason":"Started","message":"Started container tesjob","source":{"component":"kubelet","host":"k-prod-w3"},"firstTimestamp":"2024-11-09T16:43:39Z","lastTimestamp":"2024-11-09T16:43:39Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"kubelet","reportingInstance":"k-prod-w3"}}
{"type":"ADDED","object":{"kind":"Event","apiVersion":"v1","metadata":{"name":"testjob.18065a81f6eb7880","namespace":"plex","uid":"6c5ecc8d-c6dc-4a89-a9ca-7e7d6ac807d3","resourceVersion":"6314915","creationTimestamp":"2024-11-09T16:43:41Z","managedFields":[{"manager":"kube-controller-manager","operation":"Update","apiVersion":"v1","time":"2024-11-09T16:43:41Z","fieldsType":"FieldsV1","fieldsV1":{"f:count":{},"f:firstTimestamp":{},"f:involvedObject":{},"f:lastTimestamp":{},"f:message":{},"f:reason":{},"f:reportingComponent":{},"f:source":{"f:component":{}},"f:type":{}}}]},"involvedObject":{"kind":"Job","namespace":"plex","name":"testjob","uid":"e8b4c58c-3162-47cb-8ab3-af44a8bcfe13","apiVersion":"batch/v1","resourceVersion":"6314854"},"reason":"SuccessfulCreate","message":"Created pod: testjob-lnbj6","source":{"component":"job-controller"},"firstTimestamp":"2024-11-09T16:43:41Z","lastTimestamp":"2024-11-09T16:43:41Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"job-controller","reportingInstance":""}}
{"type":"ADDED","object":{"kind":"Event","apiVersion":"v1","metadata":{"name":"testjob-lnbj6.18065a81f7700fef","namespace":"plex","uid":"00cfb984-d827-438b-8d1d-10ffd97b672a","resourceVersion":"6314916","creationTimestamp":"2024-11-09T16:43:41Z","managedFields":[{"manager":"kube-scheduler","operation":"Update","apiVersion":"events.k8s.io/v1","time":"2024-11-09T16:43:41Z","fieldsType":"FieldsV1","fieldsV1":{"f:action":{},"f:eventTime":{},"f:note":{},"f:reason":{},"f:regarding":{},"f:reportingController":{},"f:reportingInstance":{},"f:type":{}}}]},"involvedObject":{"kind":"Pod","namespace":"plex","name":"testjob-lnbj6","uid":"e222f6ee-6707-4147-901b-8cb834965d2f","apiVersion":"v1","resourceVersion":"6314912"},"reason":"Scheduled","message":"Successfully assigned plex/testjob-lnbj6 to k-prod-w3","source":{},"firstTimestamp":null,"lastTimestamp":null,"type":"Normal","eventTime":"2024-11-09T16:43:41.019127Z","action":"Binding","reportingComponent":"default-scheduler","reportingInstance":"default-scheduler-k-prod-c3"}}
{"type":"ADDED","object":{"kind":"Event","apiVersion":"v1","metadata":{"name":"testjob-lnbj6.18065a82164bf27c","namespace":"plex","uid":"6507adba-0e50-495b-937b-9e09b52e0064","resourceVersion":"6314921","creationTimestamp":"2024-11-09T16:43:41Z","managedFields":[{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2024-11-09T16:43:41Z","fieldsType":"FieldsV1","fieldsV1":{"f:count":{},"f:firstTimestamp":{},"f:involvedObject":{},"f:lastTimestamp":{},"f:message":{},"f:reason":{},"f:reportingComponent":{},"f:reportingInstance":{},"f:source":{"f:component":{},"f:host":{}},"f:type":{}}}]},"involvedObject":{"kind":"Pod","namespace":"plex","name":"testjob-lnbj6","uid":"e222f6ee-6707-4147-901b-8cb834965d2f","apiVersion":"v1","resourceVersion":"6314913","fieldPath":"spec.containers{testjob}"},"reason":"Pulling","message":"Pulling image \"alpine/curl\"","source":{"component":"kubelet","host":"k-prod-w3"},"firstTimestamp":"2024-11-09T16:43:41Z","lastTimestamp":"2024-11-09T16:43:41Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"kubelet","reportingInstance":"k-prod-w3"}}
{"type":"ADDED","object":{"kind":"Event","apiVersion":"v1","metadata":{"name":"testjob-lnbj6.18065a823ce11339","namespace":"plex","uid":"c1474817-fba1-484f-b3de-7ffedfac0896","resourceVersion":"6314925","creationTimestamp":"2024-11-09T16:43:42Z","managedFields":[{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2024-11-09T16:43:42Z","fieldsType":"FieldsV1","fieldsV1":{"f:count":{},"f:firstTimestamp":{},"f:involvedObject":{},"f:lastTimestamp":{},"f:message":{},"f:reason":{},"f:reportingComponent":{},"f:reportingInstance":{},"f:source":{"f:component":{},"f:host":{}},"f:type":{}}}]},"involvedObject":{"kind":"Pod","namespace":"plex","name":"testjob-lnbj6","uid":"e222f6ee-6707-4147-901b-8cb834965d2f","apiVersion":"v1","resourceVersion":"6314913","fieldPath":"spec.containers{testjob}"},"reason":"Pulled","message":"Successfully pulled image \"alpine/curl\" in 647ms (647ms including waiting). Image size: 6188514 bytes.","source":{"component":"kubelet","host":"k-prod-w3"},"firstTimestamp":"2024-11-09T16:43:42Z","lastTimestamp":"2024-11-09T16:43:42Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"kubelet","reportingInstance":"k-prod-w3"}}

Resource files used to test:

$ cat deployment-logevents.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: logevents
  name: logevents
spec:
  replicas: 1
  selector:
    matchLabels:
      app: logevents
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: logevents
    spec:
      initContainers:
      - name: logevents
        image: alpine/curl
        imagePullPolicy: Always
        # this makes it into a sidecar (feature enabled since k8s 1.29)
        restartPolicy: Always
        command: ['sh', '-c']
        args:
        - |
          # Uncomment to cause local generation of keys, rather than wait for keys from vault
          #return 0

          # Point to the internal API server hostname
          APISERVER=https://kubernetes.default.svc

          # Path to ServiceAccount token
          SERVICEACCOUNT=/var/run/secrets/kubernetes.io/serviceaccount

          # Read this Pod's namespace
          NAMESPACE=$(cat ${SERVICEACCOUNT}/namespace)

          # Read the ServiceAccount bearer token
          TOKEN=$(cat ${SERVICEACCOUNT}/token)

          # Reference the internal certificate authority (CA)
          CACERT=${SERVICEACCOUNT}/ca.crt

          curl --cacert ${CACERT} --header "Authorization: Bearer ${TOKEN}" \
            ${APISERVER}/api/v1/namespaces/plex/events?watch=1

      containers:
      - image: alpine/curl
        name: main
        command: ['sh', '-c']
        args:
        - tail -f /dev/null
        resources: {}
status: {}

$ cat role.yaml 
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: logevents
rules:
- apiGroups: [""] # "" indicates the core API group
  resources: ["events"]
  verbs: ["watch"]

$ cat rolebinding.yaml 
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: logevents
subjects:
# You can specify more than one "subject"
- kind: ServiceAccount
  name: default
roleRef:
  kind: Role
  name: logevents
  apiGroup: rbac.authorization.k8s.io
lknite commented 1 week ago

I suspect if this were a sidecar we'd somehow be able to get access to the needed X-Plex-Token to log via the api like this: https://plexapi.dev/api-reference/log/logging-a-single-line-message

or maybe it wouldn't need a token if it were local somehow?

ressu commented 1 week ago

The kube-plex binary has all the necessary tokens to deal with the plex api. A more reliable solution would be to have an event listener where we already listen for pod state for the job. If we add an event watcher next to the job watcher, we get most of this for free.

https://github.com/ressu/kube-plex/blob/472229321407052afb2b26f24aa8b63347ad1a44/cmd/kube-plex/kubernetes.go#L120-L142

Sidecars can be tricky to manage, especially when it comes to monitoring container health. If the Plex container gets terminated for any reason, the sidecar would remain and prevent Plex from restarting.