Upstream HTTP killed due to exec_timeout: 10m0s, but it's after a minute

tomaszsluszniak commented 2 years ago

Hi, I developed a python function to run ansible playbooks. Some of them are long-running operations, that's why I increased the exec_timeout to 10m. The function is calling as an async function. For testing purposes, I created a simple playbook that has only a pause task (sleep for 5 min). My intention was to receive information after the playbook is finished, so I passed the X-Callback-Url header.

Current Behaviour

I set the function exec_timeout to 10m, but the callback URL is calling after a minute from the function execution, and the function is still executed. Why is that? How to deal with the function timeouts?

Here is a function execution log: It started at 20:17:05, and the HTTP request error appeared at 20:18:02

May 12 20:17:05 faasd-gw openfaas-fn:bps-on-demand-api[18032]: 2022/05/12 18:17:05 stdout: 
May 12 20:17:05 faasd-gw openfaas-fn:bps-on-demand-api[18032]: 2022/05/12 18:17:05 stdout: PLAY [Create BPS instance] *****************************************************
May 12 20:17:05 faasd-gw openfaas-fn:bps-on-demand-api[18032]: 2022/05/12 18:17:05 stdout: 
May 12 20:17:05 faasd-gw openfaas-fn:bps-on-demand-api[18032]: 2022/05/12 18:17:05 stdout: TASK [Gathering Facts] *********************************************************
May 12 20:17:07 faasd-gw openfaas-fn:bps-on-demand-api[18032]: 2022/05/12 18:17:07 stdout: ok: [172.21.17.120]
May 12 20:17:07 faasd-gw openfaas-fn:bps-on-demand-api[18032]: 2022/05/12 18:17:07 stdout: 
May 12 20:17:07 faasd-gw openfaas-fn:bps-on-demand-api[18032]: 2022/05/12 18:17:07 stdout: TASK [Sleep] *******************************************************************
May 12 20:17:07 faasd-gw openfaas-fn:bps-on-demand-api[18032]: 2022/05/12 18:17:07 stdout: Pausing for 300 seconds
May 12 20:17:07 faasd-gw openfaas-fn:bps-on-demand-api[18032]: 2022/05/12 18:17:07 stdout: (ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
May 12 20:18:02 faasd-gw openfaas-fn:bps-on-demand-api[18032]: 2022/05/12 18:18:02 Upstream HTTP request error: Post http://127.0.0.1:5000/: context canceled
May 12 20:18:02 faasd-gw openfaas-fn:bps-on-demand-api[18032]: 2022/05/12 18:18:02 Upstream HTTP killed due to exec_timeout: 10m0s
May 12 20:22:07 faasd-gw openfaas-fn:bps-on-demand-api[18032]: 2022/05/12 18:22:07 stdout: ok: [172.21.17.120]
May 12 20:22:07 faasd-gw openfaas-fn:bps-on-demand-api[18032]: 2022/05/12 18:22:07 stdout: 
May 12 20:22:07 faasd-gw openfaas-fn:bps-on-demand-api[18032]: 2022/05/12 18:22:07 stdout: TASK [BPS Bootstrap Summary] ***************************************************
May 12 20:22:07 faasd-gw openfaas-fn:bps-on-demand-api[18032]: 2022/05/12 18:22:07 stdout: ok: [172.21.17.120] => {
May 12 20:22:07 faasd-gw openfaas-fn:bps-on-demand-api[18032]: 2022/05/12 18:22:07 stdout:     "msg": [

Here is also python http.server log (callback is sent here)

ubuntu@faasd-devel:~/bps-on-demand-api$ python3 -m http.server 8888
Serving HTTP on 0.0.0.0 port 8888 (http://0.0.0.0:8888/) ...
172.21.17.120 - - [12/May/2022 20:18:02] code 501, message Unsupported method ('POST')
172.21.17.120 - - [12/May/2022 20:18:02] "POST / HTTP/1.1" 501 -

Expected Behaviour

I expect that the callback URL will be called after the function is finished.

Possible Solution

Steps to Reproduce (for bugs)

Context

Your Environment

Docker version: n/a
Are you using Docker Swarm or Kubernetes (FaaS-netes)? faasd
Operating System and version (e.g. Linux, Windows, MacOS): Linux Ubuntu 22.04
Link to your project or a code example to reproduce issue:

welteki commented 2 years ago

Hi @tomaszsluszniak

Could you please take some time to fill out the issue template? That makes it a lot easier for us to understand your problem.

Did you already look at: https://docs.openfaas.com/tutorials/expanded-timeouts/

The book Serverless for Everyone Else also has a chapter on timeouts and asynchronous functions.

Regards Han

tomaszsluszniak commented 2 years ago

Hi @welteki thanks for the response. I edited the issue description. Are these expanded timeouts also applicable for faasd? If yes, how to set them?

alexellis commented 1 year ago

Hi @tomaszsluszniak Han already answered the question.

The configuration of faasd is covered in the linked eBook.

This isn't an issue with the watchdog.

Alex

alexellis commented 1 year ago

/lock: resolved

openfaas / of-watchdog