Save k8s logs to a volume with a retention period

[dleard]

Our pod logs currently only write to stdout & are lost when a pod is cycled. We do have access to kibana, but that is managed outside of the team, we can't download any of the logs & the retention period is out of our hands.

We should save our own set of logs to a volume & set a retention period of 2 years for prod & 3 months for dev/test environments.

There are likely several different ways we could handle this. One possible solution would be to run a sidecar container that tails the logs to a file on an interval. There's a suggestion for implementation here. One upgrade I'd make to the above implementation is to write to a different file daily (ie: app-name-yyyy-mm-dd.log).

Update After some exploration, the above implementation with the sidecar container appears to assume control of cluster-level-logging which I don't think we have at our permission level. We can access the logs for a container with the following command: oc logs <pod-name> -c <container-name>. oc logs documentation

If we can fetch the logs from a sidecar container dynamically & save them to a file the efk stack described below could read that file. A container that has k8s or openshift image that can run kubectl logs... or oc logs.... should be able to do this.

Get pod name from ENV get container name from helm values / templates oc logs --since 10m <pod name> -c <container name> (get logs for last 10 min) sleep 10 minutes

Suggested actions for this ticket:

• reach out via RC to ask if other teams are saving their app logs to a PVC / ask platform services if we're able to tap into that logging agent ourselves (programatically in k8s beyond the above oc logs.. command.
• If not, we could potentially run a cronjob that fetches each pod's logs either using a python script like trigger-k8s-cronjob or k8s image running the above command on a schedule and saving the logs to a volume that way. The cronjob will need a service account that has the permissions necessary to fetch the logs & save to volume.

Second Update Something to explore from reaching out in RC: https://github.com/bcgov/elmsd-nodejs/tree/main/packages/openshift/templates/efk-stack A couple other suggestions for where to look from RC thread: https://stackoverflow.developer.gov.bc.ca/questions/147 https://stackoverflow.developer.gov.bc.ca/questions/732

[dleard]

@pbastia @hannavovk @patriciarussellCAS @marcellmueller Here is a card for the log retention thing I've brought up recently. I've put this in cas-obps for the registration namespace, but once this is done for one namespace I'd suggest doing it for all of our app pods in the other namespaces as well.

[ayeshmcg]

@dleard @pbastia The EFK stack is ready, created helm chart and added that to the repository https://github.com/bcgov/cas-efk As discussed with @dleard EFK stack looks good, now need to look into how to access logs As of now the logs are not in the log file. So researching more on that to find a way to either read logs directly from STDOUT or do we need to generate a log file and then have our EFK stack configured.