Support issue with Docker Swarm

iamjuush commented 4 years ago

My actions before raising this issue

[√] Followed the troubleshooting guide
[√] Read/searched the docs
[√] Searched past issues

Expected Behaviour

faas-swarm service should be listening on port 8080.

Current Behaviour

docker service logs func_gateway output:

func_gateway.1.unkmhzkco7hx@uni-vmsvr30038    | 2020/05/14 08:26:15 HTTP Read Timeout: 10m5s
func_gateway.1.unkmhzkco7hx@uni-vmsvr30038    | 2020/05/14 08:26:15 HTTP Write Timeout: 10m5s
func_gateway.1.unkmhzkco7hx@uni-vmsvr30038    | 2020/05/14 08:26:15 Binding to external function provider: http://faas-swarm:8080/
func_gateway.1.unkmhzkco7hx@uni-vmsvr30038    | 2020/05/14 08:26:15 Async enabled: Using NATS Streaming.
func_gateway.1.unkmhzkco7hx@uni-vmsvr30038    | 2020/05/14 08:26:15 Opening connection to nats://nats:4222
func_gateway.1.unkmhzkco7hx@uni-vmsvr30038    | 2020/05/14 08:26:15 Connect: nats://nats:4222
func_gateway.1.unkmhzkco7hx@uni-vmsvr30038    | 2020/05/14 08:26:20 Get http://faas-swarm:8080/system/functions: dial tcp 10.0.4.2:8080: connect: connection refused
func_gateway.1.unkmhzkco7hx@uni-vmsvr30038    | 2020/05/14 08:26:25 Get http://faas-swarm:8080/system/functions: dial tcp 10.0.4.2:8080: connect: connection refused
func_gateway.1.unkmhzkco7hx@uni-vmsvr30038    | 2020/05/14 08:26:30 Get http://faas-swarm:8080/system/functions: dial tcp 10.0.4.2:8080: connect: connection refused
func_gateway.1.unkmhzkco7hx@uni-vmsvr30038    | 2020/05/14 08:26:35 Get http://faas-swarm:8080/system/functions: dial tcp 10.0.4.2:8080: connect: connection refused

docker service logs func_faas-swarm return no output.

Exec into Faas-swarm container and ran this to check if it is listening on 8080 and realised it is not indeed:

~ # netstat -l
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 127.0.0.11:38545        0.0.0.0:*               LISTEN
udp        0      0 127.0.0.11:49635        0.0.0.0:*
Active UNIX domain sockets (only servers)
Proto RefCnt Flags       Type       State         I-Node Path

Steps to Reproduce (for bugs)

run deploy_stack.sh
docker exec -it -u 0 sh
netstat -l

Context

Trying to Deploy OpenFaaS on an ubuntu server.

Your Environment

FaaS-CLI version ( Full output from: faas-cli version ): commit: f7c29ea19b5df9d7aa87e9c70aacf4d9315da2cd version: 0.12.4
Docker version docker version (e.g. Docker 17.0.05 ): Client: Docker Engine - Community Version: 19.03.5 API version: 1.40 Go version: go1.12.12 Git commit: 633a0ea838 Built: Wed Nov 13 07:29:52 2019 OS/Arch: linux/amd64 Experimental: false

Server: Docker Engine - Community Engine: Version: 19.03.5 API version: 1.40 (minimum version 1.12) Go version: go1.12.12 Git commit: 633a0ea838 Built: Wed Nov 13 07:28:22 2019 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.2.13 GitCommit: 7ad184331fa3e55e52b890ea95e65ba581ae3429 runc: Version: 1.0.0-rc10 GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd docker-init: Version: 0.18.0 GitCommit: fec3683

Are you using Docker Swarm or Kubernetes (FaaS-netes)? Docker swarm (single node)
Operating System and version (e.g. Linux, Windows, MacOS): Ubuntu 18.04.4 LTS (GNU/Linux 4.15.0-88-generic x86_64)
Code example or link to GitHub repo or gist to reproduce problem:

Other diagnostic information / logs from troubleshooting guide docker service ls

ID                  NAME                                    MODE                REPLICAS            IMAGE                                                                         PORTS
bj1ligi1tiwy        func_alertmanager                       replicated          1/1                 prom/alertmanager:v0.16.1                                                     *:9096->9093/tcp
alrg1mlji8db        func_basic-auth-plugin                  replicated          1/1                 openfaas/basic-auth-plugin:0.1.1
73ousvpp5zwd        func_faas-swarm                         replicated          1/1                 openfaas/faas-swarm:0.6.2
dj46bsjkxnfe        func_gateway                            replicated          1/1                 openfaas/gateway:0.13.9                                                       *:1753->8080/tcp
3l0yufvdvazs        func_nats                               replicated          1/1                 nats-streaming:0.11.2
ejr2ccnnxpqm        func_prometheus                         replicated          1/1                 prom/prometheus:v2.7.1                                                        *:9095->9090/tcp
98nc2jb1gpbh        func_queue-worker                       replicated          1/1                 openfaas/queue-worker:0.7.2

alexellis commented 4 years ago

/set title: Support issue with Docker Swarm

alexellis commented 4 years ago

faas-swarm has had no recent changes and neither has the docker-compose.yml file, so this is very likely to be environmental (with your setup)

Can you provide some more context?

# Find the logs
docker service logs --tail 100 func_faas-swarm

# Check your Kernel
curl -SLsf https://raw.githubusercontent.com/docker/docker/master/contrib/check-config.sh | bash

Then perhaps try again by docker swarm leave --force and then starting over.

iamjuush commented 4 years ago

Hi, thanks for your response.

docker service logs --tail 100 func_faas-swarm returns no output.

$ curl -SLsf https://raw.githubusercontent.com/docker/docker/master/contrib/check-config.sh | bash
warning: /proc/config.gz does not exist, searching other paths for kernel config ...
info: reading kernel config from /boot/config-4.15.0-88-generic ...

Generally Necessary:
- cgroup hierarchy: properly mounted [/sys/fs/cgroup]
- apparmor: enabled and tools installed
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled (as module)
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_NF_NAT_IPV4: enabled (as module)
- CONFIG_IP_NF_FILTER: enabled (as module)
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
- CONFIG_IP_NF_NAT: enabled (as module)
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_NF_NAT_NEEDED: enabled
- CONFIG_POSIX_MQUEUE: enabled

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_MEMCG_SWAP: enabled
- CONFIG_MEMCG_SWAP_ENABLED: missing
    (cgroup swap accounting is currently not enabled, you can enable it by setting boot option "swapaccount=1")
- CONFIG_LEGACY_VSYSCALL_EMULATE: enabled
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: enabled
- CONFIG_IOSCHED_CFQ: enabled
- CONFIG_CFQ_GROUP_IOSCHED: enabled
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: enabled
- CONFIG_NET_CLS_CGROUP: enabled (as module)
- CONFIG_CGROUP_NET_PRIO: enabled
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: missing
- CONFIG_IP_NF_TARGET_REDIRECT: enabled (as module)
- CONFIG_IP_VS: enabled (as module)
- CONFIG_IP_VS_NFCT: enabled
- CONFIG_IP_VS_PROTO_TCP: enabled
- CONFIG_IP_VS_PROTO_UDP: enabled
- CONFIG_IP_VS_RR: enabled (as module)
- CONFIG_EXT4_FS: enabled
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: enabled (as module)
    - CONFIG_BRIDGE_VLAN_FILTERING: enabled
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled
      - CONFIG_CRYPTO_GCM: enabled
      - CONFIG_CRYPTO_SEQIV: enabled
      - CONFIG_CRYPTO_GHASH: enabled
      - CONFIG_XFRM: enabled
      - CONFIG_XFRM_USER: enabled (as module)
      - CONFIG_XFRM_ALGO: enabled (as module)
      - CONFIG_INET_ESP: enabled (as module)
      - CONFIG_INET_XFRM_MODE_TRANSPORT: enabled (as module)
  - "ipvlan":
    - CONFIG_IPVLAN: enabled (as module)
  - "macvlan":
    - CONFIG_MACVLAN: enabled (as module)
    - CONFIG_DUMMY: enabled (as module)
  - "ftp,tftp client in container":
    - CONFIG_NF_NAT_FTP: enabled (as module)
    - CONFIG_NF_CONNTRACK_FTP: enabled (as module)
    - CONFIG_NF_NAT_TFTP: enabled (as module)
    - CONFIG_NF_CONNTRACK_TFTP: enabled (as module)
- Storage Drivers:
  - "aufs":
    - CONFIG_AUFS_FS: enabled (as module)
  - "btrfs":
    - CONFIG_BTRFS_FS: enabled (as module)
    - CONFIG_BTRFS_FS_POSIX_ACL: enabled
  - "devicemapper":
    - CONFIG_BLK_DEV_DM: enabled
    - CONFIG_DM_THIN_PROVISIONING: enabled (as module)
  - "overlay":
    - CONFIG_OVERLAY_FS: enabled (as module)
  - "zfs":
    - /dev/zfs: missing
    - zfs command: missing
    - zpool command: missing

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

ran docker swarm leave --force and still no output from docker service logs --tail 100 func_faas-swarm

ran docker service logs -f func_gateway

 docker service logs -f func_gateway
func_gateway.1.srdxikgpmvos@uni-vmsvr30038    | 2020/05/14 09:04:53 HTTP Read Timeout: 10m5s
func_gateway.1.srdxikgpmvos@uni-vmsvr30038    | 2020/05/14 09:04:53 HTTP Write Timeout: 10m5s
func_gateway.1.srdxikgpmvos@uni-vmsvr30038    | 2020/05/14 09:04:53 Binding to external function provider: http://faas-swarm:8080/
func_gateway.1.srdxikgpmvos@uni-vmsvr30038    | 2020/05/14 09:04:53 Async enabled: Using NATS Streaming.
func_gateway.1.srdxikgpmvos@uni-vmsvr30038    | 2020/05/14 09:04:53 Opening connection to nats://nats:4222
func_gateway.1.srdxikgpmvos@uni-vmsvr30038    | 2020/05/14 09:04:53 Connect: nats://nats:4222
func_gateway.1.srdxikgpmvos@uni-vmsvr30038    | 2020/05/14 09:04:58 Get http://faas-swarm:8080/system/functions: dial tcp: lookup faas-swarm on 127.0.0.11:53: server misbehaving
func_gateway.1.srdxikgpmvos@uni-vmsvr30038    | 2020/05/14 09:05:03 Get http://faas-swarm:8080/system/functions: dial tcp: lookup faas-swarm on 127.0.0.11:53: server misbehaving
func_gateway.1.srdxikgpmvos@uni-vmsvr30038    | 2020/05/14 09:05:08 Get http://faas-swarm:8080/system/functions: dial tcp: lookup faas-swarm on 127.0.0.11:53: server misbehaving
func_gateway.1.srdxikgpmvos@uni-vmsvr30038    | 2020/05/14 09:05:13 Get http://faas-swarm:8080/system/functions: dial tcp 10.0.1.15:8080: connect: connection refused
func_gateway.1.srdxikgpmvos@uni-vmsvr30038    | 2020/05/14 09:05:18 Get http://faas-swarm:8080/system/functions: dial tcp 10.0.1.15:8080: connect: connection refused
func_gateway.1.srdxikgpmvos@uni-vmsvr30038    | 2020/05/14 09:05:23 Get http://faas-swarm:8080/system/functions: dial tcp 10.0.1.15:8080: connect: connection refused
func_gateway.1.srdxikgpmvos@uni-vmsvr30038    | 2020/05/14 09:05:28 Get http://faas-swarm:8080/system/functions: dial tcp 10.0.1.15:8080: connect: connection refused
func_gateway.1.srdxikgpmvos@uni-vmsvr30038    | 2020/05/14 09:05:33 Get http://faas-swarm:8080/system/functions: dial tcp 10.0.1.15:8080: connect: connection refused
func_gateway.1.srdxikgpmvos@uni-vmsvr30038    | 2020/05/14 09:05:38 Get http://faas-swarm:8080/system/functions: dial tcp 10.0.1.15:8080: connect: connection refused
func_gateway.1.srdxikgpmvos@uni-vmsvr30038    | 2020/05/14 09:05:43 Get http://faas-swarm:8080/system/functions: dial tcp 10.0.1.15:8080: connect: connection refused

BobTheDestroyer commented 4 years ago

Go a similar issue on docker swarm for the arm64 image.

docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES e09bab7d61ee prom/prometheus:v2.11.0 "/bin/prometheus --c…" 44 minutes ago Up 44 minutes 9090/tcp func_prometheus.1.ihn6810rcuib7l8jmft5jygl1 747b0dd8d60e openfaas/gateway:0.18.13-arm64 "./gateway" 45 minutes ago Up 44 minutes 8080/tcp, 8082/tcp func_gateway.1.wdf6i13bjey2w3exuxvpvrdjb 6e70229880a1 openfaas/faas-swarm:0.8.5-arm64 "./faas-swarm" 46 minutes ago Up 45 minutes 8080/tcp func_faas-swarm.1.lzcxr5zepi1jlefurgqa7hm69

The docker image for gateway did not expose its ports, it seems.

Sorry for the formating, dont know how to do the scrolling box thing.

alexellis commented 4 years ago

The compose file needs updating for Swarm on arm64 - @aidun has taken that, we're waiting for a fix from him - it's mainly updating the images to match the main compose file and keeping the arm64 suffix for everything but the queue-worker.

alexellis commented 4 years ago

Users can seek out technical support on Slack

alexellis commented 4 years ago

/msg: slack

openfaas / faas