cannot retrieve logs from faasd

[fundef1]

connection from faas-cli to faasd is ok login, version, deploy. describe seem to work fine:

faas-cli describe hello-py -g faasd.lan:8080
Name:                hello-py
Status:              Ready
Replicas:            1
Available replicas:  1
Invocations:         10
Image:
Function process:
URL:                 http://faasd.lan:8080/function/hello-py
Async URL:           http://faasd.lan:8080/async-function/hello-py

Current Behaviour

faas-cli logs hello-py -g faasd.lan:8080
WARNING! Communication is not secure, please consider using HTTPS. Letsencrypt.org offers free SSL/TLS certificates.
Cannot connect to OpenFaaS on URL: http://faasd.lan:8080

Your Environment

• OS and architecture: faas-cli on WIN10 / faasd on debian buster / proxmox container

• Versions:

  
  faas-cli version -g faasd.lan:8080
  ___                   _____           ____
  / _ \ _ __   ___ _ __ |  ___|_ _  __ _/ ___|
  | | | | '_ \ / _ \ '_ \| |_ / _` |/ _` \___ \
  | |_| | |_) |  __/ | | |  _| (_| | (_| |___) |
  \___/| .__/ \___|_| |_|_|  \__,_|\__,_|____/
    |_|

CLI: commit: 40555282492b1f7cfdb10d801fcdce251360ec25 version: 0.12.9

Gateway uri: http://faasd.lan:8080 version: 0.18.17 sha: 18f6c720b50db7da5f9c410f9fd3369ed7aff379 commit: Extract a caching function_query type

Provider name: faasd orchestration: containerd version: 0.9.2 sha: 5b92e7793dfebbd8e9b1700c2676cc5f56fad83e

containerd -version
```containerd github.com/containerd/containerd v1.4.0 09814d48d50816305a8e6c1a4ae3e2bcc4ba725a```

uname -a
```Linux faasd 5.4.44-2-pve #1 SMP PVE 5.4.44-2 (Wed, 01 Jul 2020 16:37:57 +0200) x86_64 GNU/Linux```

[LucasRoesler]

@fundef1 thank you for the report, unfortunately, I have been working on this already for the last couple of weeks. I have made some progress in #93, but not enough to actually close this.

One of the issues seems to be an unfortunate behavior where faasd code would try to read logs from the journalctl output before journalctl had written any output. This is probably what you are seeing now

The code (once cleaned up) in #93 addresses this issue.

However, there seems to be another issue also happening.

In some personal branches, i have added additional logging in faasd and the gateway code. I have determined that

1. the faasd log provider does correctly tail the logs,
2. that it correctly sends the logs to the faasd log handler
3. the log handler correctly sends the logs to the gateway and
4. the gateway log handler correctly writes the logs to the ResponseWriter as they are received.

Basically, everything inside the openfaas authored code is working (or can at least log the expected log line at each moment)

It seems like there might be some kind of buffering on the response writer. For example, seems to hold the last line of the logs for some time and I can not determine the exact way to force it to actually flush to the client. Note that we already use the Flusher interface. I have been able to get it to "tail" the logs by just spamming the function with invocations. Eventually it does send logs, but it takes 15 -- 20s for each batch.

To me, the fact that the Gateway sees and is able to log every expected function log line suggests that is some bug (a missing config or header) in the Gateway or with how faasd configures the provider server.

[alexellis]

@LucasRoesler is right, there's something we're looking into that is making the CLI's logs feature fail in some circumstances.

Here's a workaround that you can use to access logs:

export NAME="figlet"
journalctl -t openfaas-fn:$NAME

You'll need to run it over SSH or SSH into the faasd host to execute it.

[fundef1]

Thanks for the update. Good to know it's not something on my end.

[LucasRoesler]

@alexellis this is how I was debugging the issue up to the gateway

Update the gateway image to use docker.io/theaxer/gateway:508b17b6, it is based on this branch

sudo vim /var/lib/faasd/docker-compose.yaml

Deploy the branch version of faasd

make local \
    && sudo pkill -9 faasd \
    && sudo cp bin/faasd /usr/local/bin

Stop the faasd components and restart

sudo systemctl stop faasd
sudo -i
cd /var/lib/faasd

pkill faasd && faasd up

and

sudo -i
cd /var/lib/faasd-provider

faasd provider

Redeploy the function to avoid any networking issues

faas-cli deploy --image=functions/nodeinfo-http:latest --name=nodeinfo
faas-cli invoke nodeinfo <<< "" && date
faas-cli logs nodeinfo

Get the raw logs response using curl

curl  -u "$(sudo cat /var/lib/faasd/secrets/basic-auth-user):$(sudo cat /var/lib/faasd/secrets/basic-auth-password)" -v -H "Connection: keep-alive" "http://localhost:8080/system/logs?follow=true&name=nodeinfo&tail=-1"

[alexellis]

I think this can be closed now. @fundef1 please can you retest? I'll reopen the issue if it's still a problem in the latest version