search

openfaas / faas

OpenFaaS - Serverless Functions Made Simple
https://www.openfaas.com
MIT License
25.02k stars 1.93k forks source link

Support question on Docker Swarm and faas-idler #1507

Closed mabuaisha closed 4 years ago

mabuaisha commented 4 years ago

Expected Behaviour

I'm expecting that auto scaling will triggered and the faas-idler also work

Current Behaviour

Right now I'm testing openfaas on docker swarm and the auto scaling is not triggered and also faas-idler seems not working after the function being idle for 5 minutes

Steps to Reproduce (for bugs)

  1. Using Docker Swarm Stack
  2. Deploy faas-idler with dry-run=false faas-idler
  3. Run any simple function and deploy it to swarm
  4. My cluster contains 1 manager, 2 workers node
  5. The basic auth is enabled

Your Environment

Found deprecated go-style flags in command, translating to new format:
  faas-cli version
  ___                   _____           ____
 / _ \ _ __   ___ _ __ |  ___|_ _  __ _/ ___|
| | | | '_ \ / _ \ '_ \| |_ / _` |/ _` \___ \
| |_| | |_) |  __/ | | |  _| (_| | (_| |___) |
 \___/| .__/ \___|_| |_|_|  \__,_|\__,_|____/
      |_|

CLI:
 commit:  2d183c713b32385831dc7f69c073e57c06e3b76c
 version: 0.12.2
Client: Docker Engine - Community
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        afacb8b
 Built:             Wed Mar 11 01:27:04 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.7
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.17
  Git commit:       7141c199a2
  Built:            Wed Mar  4 01:22:45 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Troubleshooting Info

$ docker service ls
ID                  NAME                     MODE                REPLICAS            IMAGE                                PORTS
v09d7sjrbtsi        fibfunction              replicated          1/1                 mabuaisha/fibfunction:latest
bgvos5s1dpg7        func_alertmanager        replicated          1/1                 prom/alertmanager:v0.18.0
0zgy9vvwjxza        func_basic-auth-plugin   replicated          1/1                 openfaas/basic-auth-plugin:0.18.13
8he6hggcekc4        func_faas-idler          replicated          1/1                 openfaas/faas-idler:0.1.9
ugmdakqqvk8p        func_faas-swarm          replicated          1/1                 openfaas/faas-swarm:0.8.5
iroairtnvcn9        func_gateway             replicated          1/1                 openfaas/gateway:0.18.13             *:8080->8080/tcp
5t2mlcp8tzck        func_nats                replicated          1/1                 nats-streaming:0.11.2
ad9jklbhw3v6        func_prometheus          replicated          1/1                 prom/prometheus:v2.11.0              *:9090->9090/tcp
ez5bximopf49        func_queue-worker        replicated          1/1                 openfaas/queue-worker:0.9.0
$ docker service logs func_alertmanager
func_alertmanager.1.ld7mpzymjuhm@serverless-env-manager-docker-swarm.openstacklocal    | level=info ts=2020-04-14T19:42:13.512Z caller=main.go:197 msg="Starting Alertmanager" version="(version=0.18.0, branch=HEAD, revision=1ace0f76b7101cccc149d7298022df36039858ca)"
func_alertmanager.1.ld7mpzymjuhm@serverless-env-manager-docker-swarm.openstacklocal    | level=info ts=2020-04-14T19:42:13.512Z caller=main.go:198 build_context="(go=go1.12.6, user=root@868685ed3ed0, date=20190708-14:31:49)"
func_alertmanager.1.ld7mpzymjuhm@serverless-env-manager-docker-swarm.openstacklocal    | level=info ts=2020-04-14T19:42:13.516Z caller=cluster.go:161 component=cluster msg="setting advertise address explicitly" addr=172.18.0.6 port=9094
func_alertmanager.1.ld7mpzymjuhm@serverless-env-manager-docker-swarm.openstacklocal    | level=info ts=2020-04-14T19:42:13.527Z caller=cluster.go:623 component=cluster msg="Waiting for gossip to settle..." interval=2s
func_alertmanager.1.ld7mpzymjuhm@serverless-env-manager-docker-swarm.openstacklocal    | level=info ts=2020-04-14T19:42:13.559Z caller=coordinator.go:119 component=configuration msg="Loading configuration file" file=/alertmanager.yml
func_alertmanager.1.ld7mpzymjuhm@serverless-env-manager-docker-swarm.openstacklocal    | level=info ts=2020-04-14T19:42:13.560Z caller=coordinator.go:131 component=configuration msg="Completed loading of configuration file" file=/alertmanager.yml
func_alertmanager.1.ld7mpzymjuhm@serverless-env-manager-docker-swarm.openstacklocal    | level=info ts=2020-04-14T19:42:13.563Z caller=main.go:429 msg=Listening address=:9093
func_alertmanager.1.ld7mpzymjuhm@serverless-env-manager-docker-swarm.openstacklocal    | level=info ts=2020-04-14T19:42:15.528Z caller=cluster.go:648 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.00025569s
func_alertmanager.1.ld7mpzymjuhm@serverless-env-manager-docker-swarm.openstacklocal    | level=info ts=2020-04-14T19:42:23.529Z caller=cluster.go:640 component=cluster msg="gossip settled; proceeding" elapsed=10.001257238s
$ docker service logs func_prometheus
func_prometheus.1.5sxpg3soohhj@serverless-env-manager-docker-swarm.openstacklocal    | level=info ts=2020-04-14T19:42:11.152Z caller=main.go:293 msg="no time or size retention was set so using the default time retention" duration=15d
func_prometheus.1.5sxpg3soohhj@serverless-env-manager-docker-swarm.openstacklocal    | level=info ts=2020-04-14T19:42:11.152Z caller=main.go:329 msg="Starting Prometheus" version="(version=2.11.0, branch=HEAD, revision=4ef66003d9855ed2b7a41e987b33828ec36db34d)"
func_prometheus.1.5sxpg3soohhj@serverless-env-manager-docker-swarm.openstacklocal    | level=info ts=2020-04-14T19:42:11.153Z caller=main.go:330 build_context="(go=go1.12.7, user=root@0dc27cf95f36, date=20190709-09:54:35)"
func_prometheus.1.5sxpg3soohhj@serverless-env-manager-docker-swarm.openstacklocal    | level=info ts=2020-04-14T19:42:11.153Z caller=main.go:331 host_details="(Linux 3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 29 14:49:43 UTC 2018 x86_64 d1df9af9b922 (none))"
func_prometheus.1.5sxpg3soohhj@serverless-env-manager-docker-swarm.openstacklocal    | level=info ts=2020-04-14T19:42:11.153Z caller=main.go:332 fd_limits="(soft=1048576, hard=1048576)"
func_prometheus.1.5sxpg3soohhj@serverless-env-manager-docker-swarm.openstacklocal    | level=info ts=2020-04-14T19:42:11.153Z caller=main.go:333 vm_limits="(soft=unlimited, hard=unlimited)"
func_prometheus.1.5sxpg3soohhj@serverless-env-manager-docker-swarm.openstacklocal    | level=info ts=2020-04-14T19:42:11.156Z caller=main.go:652 msg="Starting TSDB ..."
func_prometheus.1.5sxpg3soohhj@serverless-env-manager-docker-swarm.openstacklocal    | level=info ts=2020-04-14T19:42:11.166Z caller=main.go:667 fs_type=XFS_SUPER_MAGIC
func_prometheus.1.5sxpg3soohhj@serverless-env-manager-docker-swarm.openstacklocal    | level=info ts=2020-04-14T19:42:11.166Z caller=main.go:668 msg="TSDB started"
func_prometheus.1.5sxpg3soohhj@serverless-env-manager-docker-swarm.openstacklocal    | level=info ts=2020-04-14T19:42:11.166Z caller=main.go:738 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
func_prometheus.1.5sxpg3soohhj@serverless-env-manager-docker-swarm.openstacklocal    | level=info ts=2020-04-14T19:42:11.169Z caller=web.go:448 component=web msg="Start listening for connections" address=0.0.0.0:9090
func_prometheus.1.5sxpg3soohhj@serverless-env-manager-docker-swarm.openstacklocal    | level=info ts=2020-04-14T19:42:11.169Z caller=main.go:766 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml
func_prometheus.1.5sxpg3soohhj@serverless-env-manager-docker-swarm.openstacklocal    | level=info ts=2020-04-14T19:42:11.169Z caller=main.go:621 msg="Server is ready to receive web requests."
$ docker node ls
ID                            HOSTNAME                                              STATUS              AVAILABILITY        MANAGER STATUS      ENGINE VERSION
un5pu000c6bew0jit8mxmbutt *   serverless-env-manager-docker-swarm.openstacklocal    Ready               Active              Leader              19.03.7
kntlas8z66bmwd1kc4h6w4khs     serverless-env-worker-docker-swarm-0.openstacklocal   Ready               Active                                  19.03.7
xz6ho0it5hbd13vha4dmtk4c7     serverless-env-worker-docker-swarm-1.openstacklocal   Ready               Active                                  19.03.7
alexellis commented 4 years ago

Hi there,

I don't think you are following the documentation correctly, but I can't be sure.

Can you share how you are deploying the functions? Give me the exact commands please.

https://docs.openfaas.com/architecture/autoscaling/#1-faas-idler

Alex

alexellis commented 4 years ago

/set title: Support question on Docker Swarm and faas-idler

mabuaisha commented 4 years ago

@alexellis I followed the instructions based on the link doc you shared.

  1. Deploying idler using the following command 
    
curl https://raw.githubusercontent.com/openfaas-incubator/faas-idler/master/docker-compose.yml -o faas-idler.yml

docker stack deploy func -c faas-idler.yml


2. Deploying functions

faas-cli deploy -yaml MY_FUNCTION_YAML.yaml [I already have my function build and pushed to docker hub]

alexellis commented 4 years ago

I need to see your stack.yml file please.

mabuaisha commented 4 years ago 
version: 1.0
provider:
name: openfaas
# This can be override upon request/need
gateway: http://gateway.openfaas.local
functions:
fibfunction:
   lang: node12
   handler: ./fibfunction
   image: mabuaisha/fibfunction:latest
   labels:
     com.openfaas.scale.min: 1
     com.openfaas.scale.max: 20
     com.openfaas.scale.zero: true
   environment:
     read_timeout: 30s
     write_timeout: 30s
     write_debug: true

the gateway.openfaas.local already added to my hosts file

alexellis commented 4 years ago

Have you invoked your function at least once, to generate a series of data for it?

mabuaisha commented 4 years ago

Yeah I did invoke my functions before and it working fine but for scaling and idler never fired

alexellis commented 4 years ago

What are the logs of the idler?

viveksyngh commented 4 years ago

I deployed a sample function on on swarm with idler and it is working fine for me.

Logs from idler:

docker service logs -f func_faas-idler
func_faas-idler.1.yc04kgo1z40e@docker-desktop    | 2020/04/15 12:16:38 Gateway version: 0.18.13, SHA: f6526929665cb05ceaca7c9c90a6bd7c5f32dca2
func_faas-idler.1.yc04kgo1z40e@docker-desktop    | dry_run: false
func_faas-idler.1.yc04kgo1z40e@docker-desktop    | gateway_url: http://gateway:8080/
func_faas-idler.1.yc04kgo1z40e@docker-desktop    | inactivity_duration: 5m0s
func_faas-idler.1.yc04kgo1z40e@docker-desktop    |
func_faas-idler.1.yc04kgo1z40e@docker-desktop    |
func_faas-idler.1.yc04kgo1z40e@docker-desktop    |
func_faas-idler.1.yc04kgo1z40e@docker-desktop    |
func_faas-idler.1.yc04kgo1z40e@docker-desktop    |
func_faas-idler.1.yc04kgo1z40e@docker-desktop    |
func_faas-idler.1.yc04kgo1z40e@docker-desktop    |
func_faas-idler.1.yc04kgo1z40e@docker-desktop    |
func_faas-idler.1.yc04kgo1z40e@docker-desktop    |
func_faas-idler.1.yc04kgo1z40e@docker-desktop    | {{200 figlet} [1.586953298319e+09 0.016109163453030834]}
func_faas-idler.1.yc04kgo1z40e@docker-desktop    | figlet   active: 0.016109
func_faas-idler.1.yc04kgo1z40e@docker-desktop    |
func_faas-idler.1.yc04kgo1z40e@docker-desktop    | {{200 figlet} [1.586953328297e+09 0.01444700793078587]}
func_faas-idler.1.yc04kgo1z40e@docker-desktop    | figlet   active: 0.014447
func_faas-idler.1.yc04kgo1z40e@docker-desktop    |
func_faas-idler.1.yc04kgo1z40e@docker-desktop    | {{200 figlet} [1.586953358279e+09 0.014032707921329095]}
func_faas-idler.1.yc04kgo1z40e@docker-desktop    | figlet   active: 0.014033
func_faas-idler.1.yc04kgo1z40e@docker-desktop    |
func_faas-idler.1.yc04kgo1z40e@docker-desktop    | {{200 figlet} [1.586953388258e+09 0.013843906168928699]}
func_faas-idler.1.yc04kgo1z40e@docker-desktop    | figlet   active: 0.013844
func_faas-idler.1.yc04kgo1z40e@docker-desktop    |
func_faas-idler.1.yc04kgo1z40e@docker-desktop    | {{200 figlet} [1.586953418243e+09 0.0137364002258654]}
func_faas-idler.1.yc04kgo1z40e@docker-desktop    | figlet   active: 0.013736
func_faas-idler.1.yc04kgo1z40e@docker-desktop    |
func_faas-idler.1.yc04kgo1z40e@docker-desktop    | {{200 figlet} [1.586953448235e+09 0.013667229880131497]}
func_faas-idler.1.yc04kgo1z40e@docker-desktop    | figlet   active: 0.013667
func_faas-idler.1.yc04kgo1z40e@docker-desktop    |
func_faas-idler.1.yc04kgo1z40e@docker-desktop    | {{200 figlet} [1.586953478232e+09 0.013619549436795995]}
func_faas-idler.1.yc04kgo1z40e@docker-desktop    | figlet   active: 0.013620
func_faas-idler.1.yc04kgo1z40e@docker-desktop    |
func_faas-idler.1.yc04kgo1z40e@docker-desktop    | {{200 figlet} [1.586953508216e+09 0.013583318586010957]}
func_faas-idler.1.yc04kgo1z40e@docker-desktop    | figlet   active: 0.013583
func_faas-idler.1.yc04kgo1z40e@docker-desktop    |
func_faas-idler.1.yc04kgo1z40e@docker-desktop    | {{200 figlet} [1.586953538206e+09 0.01355565357155764]}
func_faas-idler.1.yc04kgo1z40e@docker-desktop    | figlet   active: 0.013556
func_faas-idler.1.yc04kgo1z40e@docker-desktop    |
func_faas-idler.1.yc04kgo1z40e@docker-desktop    | {{200 figlet} [1.586953568189e+09 0.013533483028033056]}
func_faas-idler.1.yc04kgo1z40e@docker-desktop    | figlet   active: 0.013533
func_faas-idler.1.yc04kgo1z40e@docker-desktop    |
func_faas-idler.1.yc04kgo1z40e@docker-desktop    | {{200 figlet} [1.586953598171e+09 0]}
func_faas-idler.1.yc04kgo1z40e@docker-desktop    | figlet   idle
func_faas-idler.1.yc04kgo1z40e@docker-desktop    | 2020/04/15 12:26:38 Scale figlet 202 0
func_faas-idler.1.yc04kgo1z40e@docker-desktop    |
func_faas-idler.1.yc04kgo1z40e@docker-desktop    | {{200 figlet} [1.586953628179e+09 0]}
func_faas-idler.1.yc04kgo1z40e@docker-desktop    | figlet   idle
func_faas-idler.1.yc04kgo1z40e@docker-desktop    |
func_faas-idler.1.yc04kgo1z40e@docker-desktop    | {{200 figlet} [1.586953658185e+09 0]}
func_faas-idler.1.yc04kgo1z40e@docker-desktop    | figlet   idle
func_faas-idler.1.yc04kgo1z40e@docker-desktop    |
func_faas-idler.1.yc04kgo1z40e@docker-desktop    | {{200 figlet} [1.586953688185e+09 0]}
func_faas-idler.1.yc04kgo1z40e@docker-desktop    | figlet   idle
func_faas-idler.1.yc04kgo1z40e@docker-desktop    |
func_faas-idler.1.yc04kgo1z40e@docker-desktop    | {{200 figlet} [1.586953718183e+09 0]}
func_faas-idler.1.yc04kgo1z40e@docker-desktop    | figlet   idle
func_faas-idler.1.yc04kgo1z40e@docker-desktop    |
func_faas-idler.1.yc04kgo1z40e@docker-desktop    | {{200 figlet} [1.586953748179e+09 0]}
func_faas-idler.1.yc04kgo1z40e@docker-desktop    | figlet   idle
func_faas-idler.1.yc04kgo1z40e@docker-desktop    |
func_faas-idler.1.yc04kgo1z40e@docker-desktop    | {{200 figlet} [1.586953778176e+09 0.003393914710923314]}
func_faas-idler.1.yc04kgo1z40e@docker-desktop    | figlet   active: 0.003394
func_faas-idler.1.yc04kgo1z40e@docker-desktop    |
func_faas-idler.1.yc04kgo1z40e@docker-desktop    | {{200 figlet} [1.586953808161e+09 0.003393891673765557]}
func_faas-idler.1.yc04kgo1z40e@docker-desktop    | figlet   active: 0.003394
func_faas-idler.1.yc04kgo1z40e@docker-desktop    |
func_faas-idler.1.yc04kgo1z40e@docker-desktop    | {{200 figlet} [1.586953838145e+09 0.003393972305185989]}
func_faas-idler.1.yc04kgo1z40e@docker-desktop    | figlet   active: 0.003394
func_faas-idler.1.yc04kgo1z40e@docker-desktop    |
func_faas-idler.1.yc04kgo1z40e@docker-desktop    | {{200 figlet} [1.586953868134e+09 0.003393983824273093]}
func_faas-idler.1.yc04kgo1z40e@docker-desktop    | figlet   active: 0.003394
mabuaisha commented 4 years ago

@viveksyngh @alexellis when I checked the logs for my functions it was only contains the info about the settings, I did not have the full logs but what I can remembered is that only the the configuration info was shown to me 

dry_run: false
gateway_url: http://gateway:8080/
 inactivity_duration: 5m0s

But did not get the other logs as you attached

viveksyngh commented 4 years ago

Did you add this label com.openfaas.scale.zero=true to the deployed function ?

You can do that while deploying the function. faas-cli store deploy figlet --label "com.openfaas.scale.zero=true"

mabuaisha commented 4 years ago

@viveksyngh yeah I'm aware of this and this is my stack.yaml

version: 1.0
provider:
name: openfaas
# This can be override upon request/need
gateway: http://gateway.openfaas.local
functions:
fibfunction:
   lang: node12
   handler: ./fibfunction
   image: mabuaisha/fibfunction:latest
   labels:
     com.openfaas.scale.min: 1
     com.openfaas.scale.max: 20
     com.openfaas.scale.zero: true
   environment:
     read_timeout: 30s
     write_timeout: 30s
     write_debug: true

in all cases I will give a try and use a sample function from the store and see how it goes for both scaling and idler because the issue for me was for both auto scaling and idler

alexellis commented 4 years ago

Have you messed about with any of the settings in the compose file or elsewhere? I see you set an odd looking gateway URL.

viveksyngh commented 4 years ago

@mabuaisha I don't see much difference. I guess there is some other problem.

mabuaisha commented 4 years ago

@alexellis well that http://gateway.openfaas.local:8080 is actually mapped in hosts files which map the ip with that name and use the same compose file provided without any changes

@viveksyngh will give a try today and see if this is happening again or not

alexellis commented 4 years ago

Closing as a support issue since we cannot reproduce this and the software appears to be working as designed. Feel free to use Slack or if this is for work, contact OpenFaaS Ltd for paid support.

alexellis commented 4 years ago

Vivek thank you for investigating.