Long running python3 function gives error 500 - Internal Server Error null

unicman commented 7 years ago

Any python3 function that runs more than few seconds gives 500 Internal Server Error. I earlier posts that some fix went in server.go, but it seems this error is coming in proxy.go - Can't .

I've set read_timeout (65) and write_timeout (65) in docker-compose.yml and ran deploy_stack.ps1 and confirmed that docker service inspect func_gateway shows *_timeout values.

Expected Behaviour

It should send back response JSON that is sent through python handler.handle() function.

Current Behaviour

I see that docker service logs shows the output. But it doesn't make it to REST API response. REST API response still fails. The moment sleep(10) is removed, it sends proper response.

Possible Solution

I think the issue is in proxy.go where something is timing out.

Steps to Reproduce (for bugs)

Create handler.py with function code below.
Deploy in openfaas (0.6.5)
Ensure that the function shows up in http://192.168.99.100:8080/ui (since it is in boot2loader).
Hit 'Invoke'.
If there is a sleep, it will give error. If there is no sleep, it will send back response.

Context

OpenFaas 0.6.5. Want a simple function that fetches mail and pushes it to queue / DB / file.

Your Environment

Docker version docker version (e.g. Docker 17.0.05 ): Docker version 17.06.2-ce, build cec0b72 win client. Docker version 17.09.0-ce, build afdb6d4 boot2loader server.
Are you using Docker Swarm or Kubernetes (FaaS-netes)? Docker swarm
Operating System and version (e.g. Linux, Windows, MacOS): VirtualBox 5.1.28 based boot2loader on Windows 10.
Link to your project or a code example to reproduce issue:

handler.py

def handle(st):
    sleep(10)
    mailList = [ { "test" : "test1" }, { "test2" : "test3" } ]

    print(json.dumps(mailList))

docker-compose.yml

version: "3.2"
services:
    gateway:
        volumes:
            - "/var/run/docker.sock:/var/run/docker.sock"
        ports:
            - 8080:8080
        image: functions/gateway:0.6.3
        networks:
            - functions
        environment:
            dnsrr: "true"  # Temporarily use dnsrr in place of VIP while issue persists on PWD
            read_timeout: 65 # seconds
            write_timeout: 65 # seconds
        deploy:
            placement:
                constraints:
                    - 'node.role == manager'
                    - 'node.platform.os == linux'
    prometheus:
        image: functions/prometheus:latest  # autobuild from Dockerfile in repo.
        command: "-config.file=/etc/prometheus/prometheus.yml -storage.local.path=/prometheus -storage.local.memory-chunks=10000 --alertmanager.url=http://alertmanager:9093"
        ports:
            - 9090:9090
        depends_on:
            - gateway
            - alertmanager
        environment:
            no_proxy: "gateway"
        networks:
            - functions
        deploy:
            placement:
                constraints:
                    - 'node.role == manager'
                    - 'node.platform.os == linux'

    alertmanager:
        image: functions/alertmanager:latest    # autobuild from Dockerfile in repo.
        environment:
            no_proxy: "gateway"
#        volumes:
#            - ./prometheus/alertmanager.yml:/alertmanager.yml
        command:
            - '-config.file=/alertmanager.yml'
        networks:
            - functions
        ports:
            - 9093:9093
        deploy:
            placement:
                constraints:
                    - 'node.role == manager'
                    - 'node.platform.os == linux'
    # Sample functions go here.

    # Service label of "function" allows functions to show up in UI on http://gateway:8080/
    webhookstash:
        image: functions/webhookstash:latest
        labels:
            function: "true"
        depends_on:
            - gateway
        networks:
            - functions
        environment:
            no_proxy: "gateway"
            https_proxy: $https_proxy
        deploy:
            placement:
                constraints:
                    - 'node.platform.os == linux'

    # Pass a username as an argument to find how many images user has pushed to Docker Hub.
    hubstats:
        image: functions/hubstats:latest 
        labels:
            function: "true"
        depends_on:
            - gateway
        networks:
            - functions
        environment:
            no_proxy: "gateway"
            https_proxy: $https_proxy
        deploy:
            placement:
                constraints:
                    - 'node.platform.os == linux'

    # Node.js gives OS info about the node (Host)
    nodeinfo:
        image: functions/nodeinfo:latest
        labels:
            function: "true"
        depends_on:
            - gateway
        networks:
            - functions
        environment:
            no_proxy: "gateway"
            https_proxy: $https_proxy
        deploy:
            placement:
                constraints:
                    - 'node.platform.os == linux'

    # Uses `cat` to echo back response, fastest function to execute.
    echoit:
        image: functions/alpine:health
        labels:
            function: "true"
        depends_on:
            - gateway
        networks:
            - functions
        environment:
            fprocess: "cat"
            no_proxy: "gateway"
            https_proxy: $https_proxy
        deploy:
            placement:
                constraints:
                    - 'node.platform.os == linux'

    # Counts words in request with `wc` utility
    wordcount:
        image: functions/alpine:health
        labels:
            function: "true"
            com.faas.max_replicas: "10"
        depends_on:
            - gateway
        networks:
            - functions
        environment:
            fprocess: "wc"
            no_proxy: "gateway"
            https_proxy: $https_proxy
        deploy:
            placement:
                constraints:
                    - 'node.platform.os == linux'

    # Calculates base64 representation of request body.
    base64:
        image: functions/alpine:health
        labels:
            function: "true"
        depends_on:
            - gateway
        networks:
            - functions
        environment:
            fprocess: "base64"
            no_proxy: "gateway"
            https_proxy: $https_proxy
        deploy:
            placement:
                constraints:
                    - 'node.platform.os == linux'

    # Decodes base64 representation of request body.
    decodebase64:
        image: functions/alpine:health
        labels:
            function: "true"
        depends_on:
            - gateway
        networks:
            - functions
        environment:
            fprocess: "base64 -d"
            no_proxy: "gateway"
            https_proxy: $https_proxy
        deploy:
            placement:
                constraints:
                    - 'node.platform.os == linux'

    # Converts body in (markdown format) -> (html)
    markdown:
        image: alexellis2/faas-markdownrender:latest
        labels:
            function: "true"
        depends_on:
            - gateway
        networks:
            - functions
        environment:
            no_proxy: "gateway"
            https_proxy: $https_proxy
        deploy:
            placement:
                constraints:
                    - 'node.platform.os == linux'

networks:
    functions:
        driver: overlay
        # Docker does not support this option yet - maybe create outside of the stack and reference as "external"?
        #attachable: true

function_name.yml

provider:
  name: faas
  #gateway: http://localhost:8080
  gateway: http://192.168.99.100:8080

functions:
  process_mail_em_alerts:
    lang: python3
    handler: ./process_mail
    image: myhost:5000/process_mail

alexellis commented 7 years ago

The troubleshooting guide explain later that a timeout must be set at the function and gateway level. https://github.com/openfaas/faas/blob/master/guide/troubleshooting.md

Btw you don't need to edit the yaml of OpenFaaS to deploy functions or set environmental variables. Use the CLI which is also documented in the guides.

unicman commented 7 years ago

I'm deploying my function using faas-cli deploy -f function_name.yml

Added exec_timeout = 0 to docker_compose.yml, ran deploy_stack.ps1 and remove+deploy function using CLI. That didn't work.

So changed function_name.yml like this,

provider:
  name: faas
  #gateway: http://localhost:8080
  gateway: http://192.168.99.100:8080

functions:
  process_mail_em_alerts:
    lang: python3
    handler: ./process_mail
    image: myhost:5000/process_mail
    environment:
      read_timeout: 65 # seconds
      write_timeout: 65 # seconds
      exec_timeout: 0 # disable

And deployed function using faas-cli. That worked!

Thanks!

drpaneas commented 6 years ago

@alexellis I applied the same configuration with @unicman with the ENV variables like that into the *.yml but it doesn't work for me. The time it's always 15 seconds ... :/

# time curl --request POST --data-binary '{ "owner": "os-autoinst", "repo": "os-autoinst-distri-opensuse", "file": "caasp" }' http://d100.qam.suse.de:31112/function/list-pr
curl: (52) Empty reply from server

real    0m14.999s
user    0m0.008s
sys 0m0.000s

and

# echo '{ "owner": "os-autoinst", "repo": "os-autoinst-distri-opensuse", "file": "caasp" }'  | faas-cli invoke list-pr --gateway http://d100.qam.suse.de:31112/

Post http://d100.qam.suse.de:31112/function/list-pr: EOF
cannot connect to OpenFaaS on URL: http://d100.qam.suse.de:31112

I am running it on Kubernetes but I haven't configured anything on that side apart from the list-pr.yml.

alexellis commented 6 years ago

@drpaneas please open a new issue. Thank you.

As a side, you will need to set the timeout values you want on Kubernetes also. https://github.com/openfaas/faas-netes

openfaas / faas