Logging was done very quickly, needs to be thought out more though. Currently logging is running read_namespaced_pod_log with the Kubernetes python client. This gives us all current logs for the pod, up to the maximum Kubernetes itself stores (10 MB by default). Meaning when we update the pod's logs, we always have the latest 10 MB of logs.
Two possible problems:
We'll lose logs. Might be useful to have the entirety of a pods logs, thinking in the case of week/month long pod runs.
There might be a lot of churn, this would mean updating 10 MB of logs in the database every time health runs (a lot). At the least, it would be useful to have a function to only append with the diff of logs.
Bonus problem. If a pod is stopped and restarted, currently we would lose the first instances logs. That might not be preferred.
Logging was done very quickly, needs to be thought out more though. Currently logging is running
read_namespaced_pod_logwith the Kubernetes python client. This gives us all current logs for the pod, up to the maximum Kubernetes itself stores (10 MB by default). Meaning when we update the pod's logs, we always have the latest 10 MB of logs.Two possible problems: