[Best Practice Proposal]: Resource output logs should be sent to STDOUT/STDERR

[taylor]

Summary

All K8s resource output / logs should be sent to STDOUT/STDERR as event streams.

Log output, such as normal standard output as well as application errors should be sent to stdout/stderr (standard out, standard error) instead of a log file or directly to a logging system (other than a proxy/sidecar which streams to stdout/stderr).

---

Services should not concern themselves with routing or storing logs. Instead, events should be streamed, unbuffered, to STDOUT and STDERR for handling by the whatever observability / debugging process is appropriate.

Motivation

The standard output (STDOUT) and standard error (STDERR) can be easily viewed in development. In a production deployment, the CNF streams can be routed to anything, including a central logging service.

Goals

No response

Non-Goals

No response

Proposal

Send all application logs to stdout and all error logs to stderr rather than implementing or using a custom logging solution. Don't write to a log file, and don't expect log files to be managed. This then requires log rotation and log file maintenance.

Log output being sent to standard out / standard error supports open observability options and a pluggable flexible system.

By sending logs to standard out/standard error "logs will be treated like event streams" as recommended by 12 factor apps principles.

Using a sidecar to collect and stream the output is an option that is appropriate in some circumstances.

Reasons for using stdout:

• CNFs can end up with many levels and logging across many containers and Pods. stdout / stderr gives a simpler solution for all levels rather than worrying about gathering up each log from all places
• Containers are ephemeral and any logs on the file system could be lost at any point. Storing logs on the host file system could be a security risk since the host file system may be filled up by accident (or maliciously on purpose). Also log files may not be accessible for gathering in some events and configurations.

Workload Context

No response

User Stories

No response

Notes, Caveats, Constraints

Testing should verify that STDOUT/STDERR is configured for logging. For example, running kubectl get logs returns useful information for diagnosing or troubleshooting issues.

References

• https://12factor.net/logs
• https://www.dreamfactory.com/resources/whitepapers/implementing-the-twelve-factor-app-methodology/
• https://adam.herokuapp.com/past/2011/4/1/logs_are_streams_not_files/
• https://learn.microsoft.com/en-us/dotnet/architecture/microservices/architect-microservice-container-applications/resilient-high-availability-microservices
• https://medium.com/ssense-tech/keep-your-microservices-clean-using-the-twelve-factor-app-principles-f57ee42c164
• https://www.redhat.com/architect/12-factor-app
• https://dzone.com/articles/12-factor-app-principles-and-cloud-native-microser-1
• https://developer.ibm.com/articles/creating-a-12-factor-application-with-open-liberty/
• https://www.oreilly.com/library/view/architecting-modern-java/9781788393850/01019a33-e87c-4c2c-8972-b952e58b3510.xhtml - Treat logs as event streams - Architecting Modern Java EE Applications by Sebastian Daschner
• https://www.docker.com/blog/capturing-logs-in-docker-desktop/
• https://0xadada.pub/2015/11/05/on-logging/
• https://sematext.com/guides/kubernetes-logging/
• https://curity.io/resources/learn/logging-best-practices/
• https://kubernetes.io/docs/tasks/debug/
• https://logz.io/blog/a-practical-guide-to-kubernetes-logging/
• https://tanzu.vmware.com/developer/guides/app-enhancements-logging-practices/
• https://platform9.com/blog/kubernetes-logging-best-practices/
• https://github.com/kubernetes/community/blob/master/contributors/devel/sig-instrumentation/logging.md
• https://signoz.io/blog/docker-logging/
• https://kubernetes.io/docs/concepts/cluster-administration/system-logs/
• https://kubernetes.io/docs/concepts/cluster-administration/logging/
• https://www.howtogeek.com/435903/what-are-stdin-stdout-and-stderr-on-linux/
• https://docs.aws.amazon.com/AmazonECS/latest/userguide/using_awslogs.html
• https://www.coursehero.com/file/p42bdra3/Logs-Treat-logs-as-event-streams-Services-should-not-concern-themselves-with/
• https://www.bmc.com/blogs/twelve-factor-app/

[taylor]

Google doc draft https://docs.google.com/document/d/11V_mbvKfrCUFQu339HEtMpYxNme1NXGrLiRO-RZp350/edit

[taylor]

well-documented best practice. good first issue. draft started

[martin-mat]

I agree with this best practice, however I think this is one aspect of "observability". Perhaps we should consider more aspects of that:

Platform:

• centralized logging configured for platform and CNFs (fluentd, logstash etc)
• log rotation configured to prevent uncontrolled log space growth
• log search/analysis tools set up for easy log handling (elasticsearch etc)
• provide access to real time logs

CNFs:

• Log streaming using unbuffered stdout/stderr
• Setting different log levels (typically Critical, Error, Warning, Info, Debug) best using an environment variable (typically LOG_LEVEL) for easy/unified log level setting inside containers
• Logging of important lifecycle service states, including startup/available, shutdown
• Avoid logging of sensitive information (credentials, keys, secrets, personal/company data) - use tagging in logs (****) if needed
• Format of log information: Use structured logging (json etc) including contextual information - timestamp, hostname, request id etc, for easy log parsing/searching. Use standardized logging formats such as OpenMetrics and OpenTelemetry.

Some additional links https://www.cncf.io/blog/2023/07/03/kubernetes-logging-best-practices/ https://www.papertrail.com/solution/tips/cloud-native-applications-and-log-management-best-practices/ https://newrelic.com/blog/how-to-relic/log-monitoring-for-cloud-native-architectures https://www.mezmo.com/blog/what-the-cloud-native-revolution-means-for-log-management

[lixuna]

References:

• Included in CNF Certification v1.0 https://github.com/lfn-cnti/certification/blob/main/docs/CNFCertification-1.1-beta.md#use-stdoutstderr-for-logs
• Included in Test Catalog https://github.com/cnti-testcatalog/testsuite/blob/main/docs/LIST_OF_TESTS.md#use-stdoutstderr-for-logs
• Test Catalog usage https://github.com/cnti-testcatalog/testsuite/blob/main/USAGE.md#use-stdoutstderr-for-logs
• Test Catalog rationale https://github.com/cnti-testcatalog/testsuite/blob/main/RATIONALE.md#to-check-if-logs-are-being-sent-to-stdoutstderr-standard-out-standard-error-instead-of-a-log-file-log_output