For lack of somewhere better to send to at the moment we can start by simply archiving these logs, particularly kube-apiserver with increased verbosity could end up being useful.
The default is hourly which is not actionable for our short running demo. Going to try a 1 minute resolution.
Forward Output Plugin
Since we're going to have a small dedicated server (#145) to save demo runs and dashboards, we could include fluentd on it and use the forwarding plugin.
Within Kubernetes
Best way of collecting all the Kubernetes logs (from pods) via something like fluentd is under active discussion (kubernetes/kubernetes/issues/24677), still researching the way forward on this.
Outside Kubernetes
Systemd logs archiving to S3
For lack of somewhere better to send to at the moment we can start by simply archiving these logs, particularly kube-apiserver with increased verbosity could end up being useful.
The default is hourly which is not actionable for our short running demo. Going to try a 1 minute resolution.
Forward Output Plugin
Since we're going to have a small dedicated server (#145) to save demo runs and dashboards, we could include fluentd on it and use the forwarding plugin.
Within Kubernetes
Best way of collecting all the Kubernetes logs (from pods) via something like fluentd is under active discussion (kubernetes/kubernetes/issues/24677), still researching the way forward on this.