search

openfaas / faas-swarm

OpenFaaS provider for Docker Swarm
https://github.com/openfaas/faas
MIT License
81 stars 37 forks source link

Error with logs handler caused a panic #70

Open mangobowl opened 4 years ago

mangobowl commented 4 years ago

Expected Behaviour

Faas-swarm works fine.

Current Behaviour

Faas-swarm crashed soon after starting.

Possible Solution

No solution has been found yet (not familiar with golang).

Steps to Reproduce (for bugs)

  1. Follow the instructions on the official website to deploy openfaas on the swarm cluster.
  2. Deploy a few of our own business functions.
  3. Faas-swarm crashes and restarts repeatedly, the service log is as follows: 
    
2020/06/18 03:07:20 Docker API version: 1.39, 18.09.4
2020/06/18 03:07:20 HTTP Read Timeout: 5m5s
2020/06/18 03:07:20 HTTP Write Timeout: 5m5s
2020/06/18 03:07:20 Basic authentication: true
2020/06/18 03:07:21 ReplicaReader - reading function: 1230-1229-debug
panic: runtime error: slice bounds out of range

goroutine 11 [running]: github.com/openfaas/faas-swarm/handlers.parseLogStream(0x91f280, 0xc0000be300, 0xc00042e2b6, 0xd, 0xc0003ac2a0, 0x91d100, 0xc0003609c0) /go/src/github.com/openfaas/faas-swarm/handlers/logs.go:86 +0x780 created by github.com/openfaas/faas-swarm/handlers.LogRequester.Query /go/src/github.com/openfaas/faas-swarm/handlers/logs.go:65 +0x226

## Context
<!--- How has this issue affected you? What are you trying to accomplish? -->
<!--- Providing context helps us come up with a solution that is most useful in the real world -->

## Your Environment
<!--- Include as many relevant details about the environment you experienced the bug in -->
* FaaS-CLI version ( Full output from: `faas-cli version` ):

/ \ __ | | _ / | | | | | ' \ / \ ' | | / ` |/ ` _ \ | || | |) | / | | | | (| | (| |) | _/| ./ _|| ||| _,|_,_|__/ |_|

CLI: commit: b7a67fe8d6a02aef35caae615ba56333e7337bfe version: 0.9.5

faas-swarm version: 0.8.5

* Docker version `docker version` (e.g. Docker 17.0.05 ):

Client: Version: 18.09.9 API version: 1.39 Go version: go1.11.13 Git commit: 039a7df9ba Built: Wed Sep 4 16:57:28 2019 OS/Arch: linux/amd64 Experimental: false

Server: Docker Engine - Community Engine: Version: 18.09.9 API version: 1.39 (minimum version 1.12) Go version: go1.11.13 Git commit: 039a7df Built: Wed Sep 4 16:19:38 2019 OS/Arch: linux/amd64 Experimental: false


* Are you using Docker Swarm or Kubernetes (FaaS-netes)?
```Docker Swarm```
* Operating System and version (e.g. Linux, Windows, MacOS):
 ```Ubuntu 16.04.6 LTS (GNU/Linux 4.4.0-142-generic x86_64)```
* Link to your project or a code example to reproduce issue:
alexellis commented 4 years ago

Hi @mangobowl,

Thanks for your interest in OpenFaaS

For us to be able to help you, we will need either the functions you are deploying, or similar ones where you can re-create the issue.

faas-swarm hasn't been changed significantly for quite some time, so I suspect this has something to do with the naming of your functions or perhaps something else specific to your environment.

If you want a quick resolution, I'd suggest you use OpenFaaS on Kubernetes - with k3s or a managed service like AKS, GKE etc. This is what we recommend for business / work and production use and what we actively support. See the note here -> https://docs.openfaas.com/deployment/

Alex

alexellis commented 4 years ago

/set title: Error with logs handler caused a panic

alexellis commented 4 years ago

Specifically, it looks like you are trying to access logs from the call-stack, is that right? Did you run faas-cli logs immediately after deploying? cc @LucasRoesler

LucasRoesler commented 4 years ago

that is really weird, the line that panics is here https://github.com/openfaas/faas-swarm/blob/master/handlers/logs.go#L86

which suggests that the logs stream was completely empty or a partial load of the logs that doesn't contain the full docker log prefix.

Perhaps the cleanest way to change this is to try and reuse their StdCopy implementation to get stderr and stdout into a new reader.

I have an idea on how to handle it

mangobowl commented 4 years ago

@alexellis @LucasRoesler Because there are too many functions in my production environment, it took a long time to troubleshoot the problem, sorry for the delay.

This problem is related to a special function log. The steps to reproduce are as follows:

  1. Deployment the function 1213-test0608(The image in docker hub is mangobowl/debug-function:debug)
  2. Invoke the function
  3. Get the function log by the api /system/logs?name=1213-test0608, then the faas-swarm will crash.

The full docker json-file log of the function is as follows: 4cf1fbee11253f846dcc8613403262e4c534279fc1268704994ca5c7420a678b-json.log

The api response will terminate at line 127:

{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T07:59:04.093038594Z","text":"Forking - python [index.py]"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T07:59:04.093354612Z","text":"2020/06/23 07:59:04 Started logging stderr from function."}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T07:59:04.093391118Z","text":"2020/06/23 07:59:04 Started logging stdout from function."}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T07:59:04.094015443Z","text":"2020/06/23 07:59:04 OperationalMode: http"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T07:59:04.094192224Z","text":"2020/06/23 07:59:04 Timeouts: read: 5m0s, write: 5m0s hard: 5m0s."}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T07:59:04.094219234Z","text":"2020/06/23 07:59:04 Listening on port: 8080"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T07:59:04.094241148Z","text":"2020/06/23 07:59:04 Writing lock-file to: /tmp/.lock"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T07:59:04.094247609Z","text":"2020/06/23 07:59:04 Metrics listening on port: 8081"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.836543308Z","text":"2020/06/23 08:00:03 stdout: args: ImmutableMultiDict([])"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.836595126Z","text":"2020/06/23 08:00:03 stdout: data: b''"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.836608521Z","text":"2020/06/23 08:00:03 stdout: headers: Content-Type: application/json"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.836616671Z","text":"2020/06/23 08:00:03 stdout: Content-Length: 0"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.83662435Z","text":"2020/06/23 08:00:03 stdout: Host: 1213-test0608:8080"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.836632386Z","text":"2020/06/23 08:00:03 stdout: User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36 OPR/68.0.3618.173"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.836643619Z","text":"2020/06/23 08:00:03 stdout: Accept: application/json, text/plain, */*"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.836654495Z","text":"2020/06/23 08:00:03 stdout: Accept-Encoding: gzip, deflate"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.836662488Z","text":"2020/06/23 08:00:03 stdout: Accept-Language: zh-CN,zh;q=0.9,en-US;q=0.8,en;q=0.7"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.836671071Z","text":"2020/06/23 08:00:03 stdout: Cookie: monitorCustomerKey=5a0f0cd7-c748-4841-be8b-7269b8b99276-1590409358674; ajs_anonymous_id=%224b72111e-907f-4751-be00-6514a2f92809%22; _ga=GA1.2.1466632365.1590558477; NX-ANTI-CSRF-TOKEN=0.19181169709565116; NXSESSIONID=99302d0b-d5ab-449b-93d0-ee08706fdfe8; _gid=GA1.2.890040766.1592893980; csrftoken=LmvUCRceTv9dBCX3ivfWwe2ikikwpwHd42URP4X1haeZeLFWMm2gzHSAjv6i9dIh; sessionid=8ovt402pinpqvsln462ojhm8g77zqfil"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.836681754Z","text":"2020/06/23 08:00:03 stdout: Origin: http://nexus.hiktest.com:18080"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.836691391Z","text":"2020/06/23 08:00:03 stdout: Referer: http://nexus.hiktest.com:18080/ui/"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.836699264Z","text":"2020/06/23 08:00:03 stdout: X-Call-Id: cedc3de5-96a2-4140-a336-2b44f80e1cb6"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.836706689Z","text":"2020/06/23 08:00:03 stdout: X-Forwarded-For: 10.65.143.143:59798"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.836713597Z","text":"2020/06/23 08:00:03 stdout: X-Forwarded-Host: nexus.hiktest.com:18080"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.836720863Z","text":"2020/06/23 08:00:03 stdout: X-Start-Time: 1592899203820666171"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.836728198Z","text":"2020/06/23 08:00:03 stdout:"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.836735333Z","text":"2020/06/23 08:00:03 stdout:"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.836742508Z","text":"2020/06/23 08:00:03 stdout: form: ImmutableMultiDict([])"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.83674973Z","text":"2020/06/23 08:00:03 stdout: /home/app/function/handler.py[line:18] - INFO: 0"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.846679046Z","text":"2020/06/23 08:00:03 stdout: /home/app/function/handler.py[line:18] - INFO: 1"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.856989128Z","text":"2020/06/23 08:00:03 stdout: /home/app/function/handler.py[line:18] - INFO: 2"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.867285774Z","text":"2020/06/23 08:00:03 stdout: /home/app/function/handler.py[line:18] - INFO: 3"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.877715103Z","text":"2020/06/23 08:00:03 stdout: /home/app/function/handler.py[line:18] - INFO: 4"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.888026179Z","text":"2020/06/23 08:00:03 stdout: /home/app/function/handler.py[line:18] - INFO: 5"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.898369725Z","text":"2020/06/23 08:00:03 stdout: /home/app/function/handler.py[line:18] - INFO: 6"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.908723221Z","text":"2020/06/23 08:00:03 stdout: /home/app/function/handler.py[line:18] - INFO: 7"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.919067403Z","text":"2020/06/23 08:00:03 stdout: /home/app/function/handler.py[line:18] - INFO: 8"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.929382731Z","text":"2020/06/23 08:00:03 stdout: /home/app/function/handler.py[line:18] - INFO: 9"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.939759512Z","text":"2020/06/23 08:00:03 stdout: /home/app/function/handler.py[line:18] - INFO: 10"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.950042229Z","text":"2020/06/23 08:00:03 stdout: /home/app/function/handler.py[line:18] - INFO: 11"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.96043778Z","text":"2020/06/23 08:00:03 stdout: /home/app/function/handler.py[line:18] - INFO: 12"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.970743751Z","text":"2020/06/23 08:00:03 stdout: /home/app/function/handler.py[line:18] - INFO: 13"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.981130142Z","text":"2020/06/23 08:00:03 stdout: /home/app/function/handler.py[line:18] - INFO: 14"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.991489161Z","text":"2020/06/23 08:00:03 stdout: /home/app/function/handler.py[line:18] - INFO: 15"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.001823858Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 16"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.012123535Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 17"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.022488806Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 18"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.032778573Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 19"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.043156302Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 20"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.053537687Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 21"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.06384826Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 22"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.074121599Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 23"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.084547571Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 24"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.094843591Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 25"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.105151356Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 26"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.115524652Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 27"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.12575426Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 28"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.136141716Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 29"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.146465667Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 30"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.156823622Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 31"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.167119495Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 32"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.177464524Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 33"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.187849047Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 34"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.198082664Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 35"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.208420074Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 36"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.218838849Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 37"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.229138286Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 38"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.239421116Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 39"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.249788228Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 40"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.260134189Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 41"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.270443818Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 42"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.280824262Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 43"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.29115372Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 44"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.301498122Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 45"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.311828915Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 46"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.322141878Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 47"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.332510622Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 48"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.34286626Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 49"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.353182707Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 50"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.363507019Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 51"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.37389845Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 52"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.38413531Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 53"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.39454411Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 54"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.404923711Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 55"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.415259059Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 56"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.42557142Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 57"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.435952609Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 58"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.446289249Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 59"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.4566382Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 60"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.466905115Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 61"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.477119001Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 62"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.487382285Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 63"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.497769548Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 64"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.508079199Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 65"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.5184419Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 66"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.528722073Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 67"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.539104804Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 68"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.549429336Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 69"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.559708485Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 70"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.570090412Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 71"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.580416997Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 72"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.590738452Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 73"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.601024329Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 74"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.611344263Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 75"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.621618129Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 76"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.631950804Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 77"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.642345212Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 78"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.652622042Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 79"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.662947679Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 80"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.673355996Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 81"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.683650997Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 82"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.69391356Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 83"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.704220725Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 84"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.714611659Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 85"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.724945149Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 86"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.735180053Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 87"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.745561638Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 88"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.755843638Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 89"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.766159469Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 90"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.776533527Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 91"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.786923126Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 92"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.797226723Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 93"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.807511983Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 94"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.817872731Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 95"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.828284132Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 96"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.838588434Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 97"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.848903252Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 98"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.85923018Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 99"}

The function yml config is as follows: 1213-test0608.zip

LucasRoesler commented 4 years ago

I don't understand what is special about line 127

{"log":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 99\n","stream":"stdout","time":"2020-06-23T08:00:04.85923018Z"}

The API response has a default timeout, is it possible that it terminates there just because of the request timeout? Can you share the fully cli command you used or the curl request you used?

ag2uki commented 4 years ago

I got the similar issue. faas-swarm was getting crash when I tried to get the logs using faas-cli logs. The logs is similar with the first post. https://github.com/openfaas/faas-swarm/issues/70#issue-640880684

My workaround was trying to remove the function and then redeploy it. $ docker service rm function-xyz $ faas-cli deploy -f function-xyz.yml

And faas-cli logs works as expected then. And also, faas-swarm was not getting crash anymore.

alexellis commented 4 years ago

Not sure how removing the function could prevent a crash. Can either of you give us full reproduction instructions?