Support question on log aggregation

[lyudmilalala]

My actions before raising this issue

• [x] Followed the troubleshooting guide
• [x] Read/searched the docs
• [x] Searched past issues

This is more a question rather than an issue.

I am trying to collect logs from a python flask function, and hope I can retain them as log files on disk, which will rotate once a day. During this process, I find that:

1. I don't really know how to overwrite the log provider to save logs to a file on the disk of a k8s master node. The only related tutorial I see is the static logger provider written in Golang. I hope can get some tutorials in python.
2. I cannot get logs by faas-cli logs <func_name>, I can only see logs by docker logs <container_name>.

How someone could tell me how to implement the log collection correctly.

A further question is that, if collecting logs to a file system is unrealistic, do I need to make sure my logs can be shown by faas-cli logs <func_name> before trying to collect it by Elastic Search or Grafana Loki? I am not familiar with these two things.

Expected Behaviour

Some logs show when running faas-cli logs flask-faas or kubectl logs XXX

Current Behaviour

Run faas-cli logs flask-faas shows nothing

Run kubectl logs flask-faas-57f6449bbd-z5cld -n openfaas-fn shows an error. I am not sure if it is related with openfaas or not.

Error from server: Get "https://192.168.65.4:10250/containerLogs/openfaas-fn/flask-faas-57f6449bbd-z5cld/flask-faas": open /run/config/pki/apiserver-kubelet-client.crt: no such file or directory

Only when using docker logs the logger and stdout/stderr works (2ef3e197a8b1 is the id of the function container).

$ docker logs 2ef3e197a8b1
2023/03/16 12:02:28 Version: 0.9.10     SHA: eefeb9dd8c979398a46fc0decc3297591362bfab
2023/03/16 12:02:28 Forking: python, arguments: [index.py]
2023/03/16 12:02:28 Started logging: stderr from function.
2023/03/16 12:02:28 Started logging: stdout from function.
2023/03/16 12:02:28 Watchdog mode: http fprocess: "python index.py"
2023/03/16 12:02:28 Timeouts: read: 10s write: 10s hard: 10s health: 10s
2023/03/16 12:02:28 Listening on port: 8080
2023/03/16 12:02:28 Writing lock-file to: /tmp/.lock
2023/03/16 12:02:28 Metrics listening on port: 8081
2023/03/16 12:02:29 stderr: global logger error
2023/03/16 12:06:51 stderr: global stderr
2023/03/16 12:06:51 stderr: INFO:function.handler:logger info: ImmutableMultiDict([('filepath', 'data/10ibt_H.msgpack'), ('filterStr', '00000*****')])
2023/03/16 12:06:51 stderr: ERROR:function.handler:logger error: ImmutableMultiDict([('filepath', 'data/10ibt_H.msgpack'), ('filterStr', '00000*****')])
2023/03/16 12:06:51 stderr: stderr: ImmutableMultiDict([('filepath', 'data/10ibt_H.msgpack'), ('filterStr', '00000*****')])
2023/03/16 12:06:51 stderr: INFO:function.handler:inner logger info: ImmutableMultiDict([('filepath', 'data/10ibt_H.msgpack'), ('filterStr', '00000*****')])
2023/03/16 12:06:51 stderr: ERROR:function.handler:inner logger error: ImmutableMultiDict([('filepath', 'data/10ibt_H.msgpack'), ('filterStr', '00000*****')])
2023/03/16 12:06:51 GET /?filepath=data/10ibt_H.msgpack&filterStr=00000%2A%2A%2A%2A%2A - 200 OK - ContentLength: 189B (0.0171s)

If I add PYTHONUNBUFFERED=1 as mentioned in this post, the docker log includes print and stdout information as shown below, but the faas-cli logs and kubectl logs are still the same, giving no useful logs.

2023/03/16 12:40:09 Version: 0.9.10     SHA: eefeb9dd8c979398a46fc0decc3297591362bfab
2023/03/16 12:40:09 Forking: python, arguments: [index.py]
2023/03/16 12:40:09 Started logging: stderr from function.
2023/03/16 12:40:09 Started logging: stdout from function.
2023/03/16 12:40:09 Watchdog mode: http fprocess: "python index.py"
2023/03/16 12:40:09 Timeouts: read: 10s write: 10s hard: 10s health: 10s     
2023/03/16 12:40:09 Listening on port: 8080
2023/03/16 12:40:09 Writing lock-file to: /tmp/.lock
2023/03/16 12:40:09 Metrics listening on port: 8081
2023/03/16 12:40:09 stdout: global print
2023/03/16 12:40:09 stdout: global stdout
2023/03/16 12:40:09 stderr: global logger error
2023/03/16 12:40:09 stderr: global stderr
2023/03/16 12:43:37 stderr: INFO:function.handler:logger info: ImmutableMultiDict([('filepath', 'data/10ibt_H.msgpack'), ('filterStr', '00000*****')])    
2023/03/16 12:43:37 stderr: ERROR:function.handler:logger error: ImmutableMultiDict([('filepath', 'data/10ibt_H.msgpack'), ('filterStr', '00000*****')])  
2023/03/16 12:43:37 stderr: stderr: ImmutableMultiDict([('filepath', 'data/10ibt_H.msgpack'), ('filterStr', '00000*****')])
2023/03/16 12:43:37 stdout: print: ImmutableMultiDict([('filepath', 'data/10ibt_H.msgpack'), ('filterStr', '00000*****')])
2023/03/16 12:43:37 stdout: stdout: ImmutableMultiDict([('filepath', 'data/10ibt_H.msgpack'), ('filterStr', '00000*****')])
2023/03/16 12:43:37 stdout: inner print: ImmutableMultiDict([('filepath', 'data/10ibt_H.msgpack'), ('filterStr', '00000*****')])
2023/03/16 12:43:37 stdout: inner stdout: ImmutableMultiDict([('filepath', 'data/10ibt_H.msgpack'), ('filterStr', '00000*****')])
2023/03/16 12:43:37 stderr: INFO:function.handler:inner logger info: ImmutableMultiDict([('filepath', 'data/10ibt_H.msgpack'), ('filterStr', '00000*****')])
2023/03/16 12:43:37 stderr: ERROR:function.handler:inner logger error: ImmutableMultiDict([('filepath', 'data/10ibt_H.msgpack'), ('filterStr', '00000*****')])
2023/03/16 12:43:37 stderr: inner stderr: ImmutableMultiDict([('filepath', 'data/10ibt_H.msgpack'), ('filterStr', '00000*****')])
2023/03/16 12:43:37 GET /?filepath=data/10ibt_H.msgpack&filterStr=00000%2A%2A%2A%2A%2A - 200 OK - ContentLength: 189B (0.0023s)

Steps to Reproduce (for bugs)

Create a function by template. I am using the flask template with of-watchdog:0.9.10.

faas-cli new --lang python3-http flask-faas --prefix lyudmilalala

Overwrite the handler.py in the flask-faas folder.

from flask import current_app
import sys
import logging

logger = logging.getLogger(__name__)
logger.setLevel("INFO")

logger.info("global logger info")
logger.error("global logger error")
print("global print") # cannot print
sys.stdout.write("global stdout\n")
sys.stderr.write("global stderr\n")

def handle(event, context):
    current_app.logger.info("app logger: " + repr(event.query)) # cannot print
    logger.info("logger info: " + repr(event.query))
    logger.error("logger error: " + repr(event.query))
    print("print: " + repr(event.query)) # cannot print
    sys.stdout.write("stdout: " + repr(event.query) + "\n")
    sys.stderr.write("stderr: " + repr(event.query) + "\n")

    inner(event)
    return {
        "statusCode": 200,
        "body": "Welcom to flask-demo v1.0!"
    }

def inner(event):
    current_app.logger.info("inner app logger: " + repr(event.query)) # cannot print
    logger.info("inner logger info: " + repr(event.query))
    logger.error("inner logger error: " + repr(event.query))
    print("inner print: " + repr(event.query)) # cannot print
    sys.stdout.write("inner stdout: " + repr(event.query) + "\n")
    sys.stderr.write("inner stderr: " + repr(event.query)  + "\n")

Overwrite the flask-faas.yml

192.168.1.124 is my local IP.

version: 1.0
provider:
  name: openfaas
  gateway: http://192.168.1.124:31112
functions:
  flask-faas:
    lang: python3-http
    handler: ./flask-faas
    image: lyudmilalala/flask-faas:1.0
    environment:
      write_debug: true 
      PYTHONUNBUFFERED: 1

Start the function.

faas-cil up -f flask-faas.yml

Call the function with curl.

curl --request GET 'http://192.168.1.124:31112/function/flask-faas?filepath=data/10ibt_H.msgpack&filterStr=00000*****'

Your Environment

• FaaS-CLI version ( Full output from: faas-cli version ): 0.14.2

• Docker version docker version (e.g. Docker 17.0.05 ): docker version 20.10.11

• Are you using Docker Swarm or Kubernetes (FaaS-netes)? FaaS-netes with tag 0.16.4, k8s version v1.22.4

• Operating System and version (e.g. Linux, Windows, MacOS): Windows 10 64-bit, Docker Desktop 4.3.0 with kind

• Code example or link to GitHub repo or gist to reproduce problem:

• Other diagnostic information / logs from troubleshooting guide

Docs and posts about logging I have gone through

• https://docs.openfaas.com/cli/logs/#log-retention-and-history
• https://www.openfaas.com/blog/openfaas-loki-into/
• https://github.com/openfaas/workshop/blob/master/lab4.md#making-use-of-logging
• https://github.com/openfaas/faas/issues/1654#issuecomment-1295995272
• https://github.com/openfaas/python-flask-template/issues/35

I did not find much about customized log provider. Also, because I am considering to move some of my business from python to c++ for performance improvement, I want to know more about logging for customized docker image with of-watchdog. (In my case, I use a pistachio server behind of-watchdog, and don't whether I should use a log library or just stdout logs.) I hope to see more documentations about them.

Next steps

You may join Slack for community support.

[alexellis]

I am trying to collect logs from a python flask function, and hope I can retain them as log files on disk, which will rotate once a day. During this process, I find that:

Hi, it sounds like you could solve your problem by:

• Using kubectl logs - rotation is automatic, and you can specify a "since" value
• A log aggregation solution like logz.io or Grafana Loki

Function Pods (replicas) come and go often, you shouldn't be writing anything to disk and expecting it to still be there at a future date.

Alex

[alexellis]

/set title: Support question on log aggregation

[alexellis]

/add label: support

[lyudmilalala]

I am trying to collect logs from a python flask function, and hope I can retain them as log files on disk, which will rotate once a day. During this process, I find that:

Hi, it sounds like you could solve your problem by:

• Using kubectl logs - rotation is automatic, and you can specify a "since" value
• A log aggregation solution like logz.io or Grafana Loki

Function Pods (replicas) come and go often, you shouldn't be writing anything to disk and expecting it to still be there at a future date.

Alex

Thanks for quick response, @alexellis .

I am not going to save logs on the specific worker node running the pod. I am just thinking about collecting logs to a NFS system or something similar, as I am not familiar with ES or Grafana.

Any idea on the reason why I cannot see logs by kubectl logs or faas-cli logs?

[alexellis]

You'll usually see logs for pods which still exist, so long as they haven't been deleted.

For long term aggregation, I'd suggest using whatever your cloud provider makes available, or Loki or perhaps logz.io

In a default installation - faas-cli logs maps to kubectl logs.

[lyudmilalala]

You'll usually see logs for pods which still exist, so long as they haven't been deleted.

For long term aggregation, I'd suggest using whatever your cloud provider makes available, or Loki or perhaps logz.io

In a default installation - faas-cli logs maps to kubectl logs.

So the current problem now is that my faas-cli logs and kubectl logs do not show logs.

Could you please take a look at the current behavior section and the steps to reproduce section of the issue to see what I did wrong?