search

moby / libnetwork

networking for containers
Apache License 2.0
2.15k stars 880 forks source link

Docker services cannot exchange data between separate swarm nodes when network encryption is enabled #2337

Open jayb1122 opened 5 years ago

jayb1122 commented 5 years ago

Any help would be greatly appreciated as this issue has persisted across multiple docker releases and many attempts to debug. Perhaps it's something simple?

Could it be that Docker Networking, with encryption enabled, be using some stealth port between swarm nodes that's being blocked by AWS?

Issue

Configuration

Environment

Compose YML File Settings version: '3.6' services: . . . networks: my-net:

Uncomment for encryption enabled, Comment out for encryption disabled

    driver_opts:
        encrypted: "true"
    ipam:
        config:
            - subnet: 10.0.0.0/16

Docker Exec into Nginx communicating with Service on separate node (encryption DISABLED) $ nslookup sage.service Server: 127.0.0.11 Address: 127.0.0.11#53 Non-authoritative answer: Name: sage.service Address: 10.0.0.2 $ ping sage.service PING sage.service (10.0.0.2) 56(84) bytes of data. 64 bytes from ip-10-0-0-2.us-gov-west-1.compute.internal (10.0.0.2): icmp_seq=1 ttl=64 time=0.047 ms $ curl -X GET http://sage.service:3000/ Hello from SAGE REST Service

Docker Exec into Nginx communicating with Service on separate node (encryption ENABLED) $ nslookup sage.service Server: 127.0.0.11 Address: 127.0.0.11#53 Non-authoritative answer: Name: sage.service Address: 10.0.0.2 $ ping sage.service PING sage.service (10.0.0.2) 56(84) bytes of data. 64 bytes from ip-10-0-0-2.us-gov-west-1.compute.internal (10.0.0.2): icmp_seq=1 ttl=64 time=0.047 ms $ curl -X GET http://sage.service:3000/ About to connect() to sage.service port 3000 (#0) Trying 10.0.0.2... Connection timed out Failed connect to sage.service:3000; Connection timed out Closing connection 0 curl: (7) Failed connect to sage.service:3000; Connection timed out

=====================================================

Docker Info $ docker info Containers: 3 Running: 3 Paused: 0 Stopped: 0 Images: 4 Server Version: 18.09.1 Storage Driver: overlay2 Backing Filesystem: xfs Supports d_type: true Native Overlay Diff: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: active NodeID: a65mhgfrbzim59ot5169atdb4 Is Manager: true ClusterID: ka7xbrxuaasc3d94pb4bbumxk Managers: 3 Nodes: 3 Default Address Pool: 10.0.0.0/8 SubnetSize: 24 Orchestration: Task History Retention Limit: 5 Raft: Snapshot Interval: 10000 Number of Old Snapshots to Retain: 0 Heartbeat Tick: 1 Election Tick: 10 Dispatcher: Heartbeat Period: 5 seconds CA Configuration: Expiry Duration: 3 months Force Rotate: 0 Autolock Managers: false Root Rotation In Progress: false Node Address: 10.1.0.207 Manager Addresses: 10.1.0.126:2377 10.1.0.200:2377 10.1.0.207:2377 Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: 9754871865f7fe2f4e74d43e2fc7ccd237edcbce runc version: 96ec2177ae841256168fcf76954f7177af9446eb init version: fec3683 Security Options: seccomp Profile: default Kernel Version: 3.10.0-957.5.1.el7.x86_64 Operating System: CentOS Linux 7 (Core) OSType: linux Architecture: x86_64 CPUs: 8 Total Memory: 31.26GiB Name: myserver.xyz.com ID: JGQL:6KUA:QQ2Z:RU4M:XIWI:NEJA:MCFJ:GG3A:S4RW:DKNT:42A5:RCFB Docker Root Dir: /var/lib/docker Debug Mode (client): false Debug Mode (server): false Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: iaidev3:5000 127.0.0.0/8 Live Restore Enabled: false Product License: Community Engine WARNING: bridge-nf-call-ip6tables is disabled

=====================================================

IP TABLES (same on all swarm nodes) NOTE...Enabled Ports for Docker: 2376 (TCP), 2377 (TCP), 4789 (TCP and UDP), 7946 (TCP and UDP)

[iaidev4]$ sudo iptables -nL

Chain INPUT (policy DROP) target prot opt source destination ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 / 001 accept all to lo interface / ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 / 002 accept all related established input traffic / state RELATED,ESTABLISHED ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 multiport sports 67,68 multiport dports 67,68 / 003 accept dhcp in / ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0 / 004 accept ping in / icmptype 8 ACCEPT esp -- 0.0.0.0/0 0.0.0.0/0 / 005 accept all ipsec input traffic / ACCEPT udp -- 10.1.2.11 0.0.0.0/0 multiport sports 53 multiport dports 53 / 010 allow udp DNS lookups to 10.1.2.11 / ACCEPT udp -- 10.1.3.11 0.0.0.0/0 multiport sports 53 multiport dports 53 / 010 allow udp DNS lookups to 10.1.3.11 / ACCEPT udp -- 10.1.2.11 0.0.0.0/0 multiport sports 88,464 / 015 allow kerberos UDP in from 10.1.2.11 / ACCEPT udp -- 10.1.3.11 0.0.0.0/0 multiport sports 88,464 / 015 allow kerberos UDP in from 10.1.3.11 / ACCEPT udp -- 10.1.2.240 0.0.0.0/0 multiport sports 2049 / 015 allow nfs udp lookups from fs1.mabiaf.com / ACCEPT udp -- 10.1.2.12 0.0.0.0/0 multiport sports 2049 / 015 allow nfs udp lookups from nfsserver.mabiaf.com / ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 22 / 020 allow ssh in from 0.0.0.0/0.0.0.0 / limit: avg 100/min burst 100 ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 multiport sports 123 multiport dports 123 / 020 allow udp ntp lookups from NTP Server / ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 111 / 025 allow incoming tcp connections on port 111 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 1161 / 025 allow incoming tcp connections on port 1161 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 1883 / 025 allow incoming tcp connections on port 1883 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 20048 / 025 allow incoming tcp connections on port 20048 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 2049 / 025 allow incoming tcp connections on port 2049 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 22 / 025 allow incoming tcp connections on port 22 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 2376:2377 / 025 allow incoming tcp connections on port 2376-2377 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 3000:3029 / 025 allow incoming tcp connections on port 3000-3029 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 443 / 025 allow incoming tcp connections on port 443 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 46360 / 025 allow incoming tcp connections on port 46360 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 4789 / 025 allow incoming tcp connections on port 4789 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 50 / 025 allow incoming tcp connections on port 50 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 500 / 025 allow incoming tcp connections on port 500 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 5000 / 025 allow incoming tcp connections on port 5000 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 50411:50539 / 025 allow incoming tcp connections on port 50411-50539 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 50555:50556 / 025 allow incoming tcp connections on port 50555-50556 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 51800:51820 / 025 allow incoming tcp connections on port 51800-51820 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 51900:51920 / 025 allow incoming tcp connections on port 51900-51920 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 5432 / 025 allow incoming tcp connections on port 5432 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 55097 / 025 allow incoming tcp connections on port 55097 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 61616 / 025 allow incoming tcp connections on port 61616 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 7946 / 025 allow incoming tcp connections on port 7946 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 80 / 025 allow incoming tcp connections on port 80 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 8010 / 025 allow incoming tcp connections on port 8010 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 8080 / 025 allow incoming tcp connections on port 8080 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 8082:8083 / 025 allow incoming tcp connections on port 8082-8083 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 8161 / 025 allow incoming tcp connections on port 8161 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 8443 / 025 allow incoming tcp connections on port 8443 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 9012:9013 / 025 allow incoming tcp connections on port 9012-9013 / state NEW ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 111 / 025 allow incoming udp connections on port 111 / state NEW ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 20048 / 025 allow incoming udp connections on port 20048 / state NEW ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 20100:20101 / 025 allow incoming udp connections on port 20100-20101 / state NEW ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 2049 / 025 allow incoming udp connections on port 2049 / state NEW ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 4789 / 025 allow incoming udp connections on port 4789 / state NEW ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 50 / 025 allow incoming udp connections on port 50 / state NEW ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 500 / 025 allow incoming udp connections on port 500 / state NEW ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 50411:50475 / 025 allow incoming udp connections on port 50411-50475 / state NEW ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 52116 / 025 allow incoming udp connections on port 52116 / state NEW ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 54053 / 025 allow incoming udp connections on port 54053 / state NEW ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 7946 / 025 allow incoming udp connections on port 7946 / state NEW ACCEPT udp -- 10.1.2.57 0.0.0.0/0 multiport dports 8082,8083 / 027 HBSS Incoming udp / state NEW ACCEPT tcp -- 10.1.2.57 0.0.0.0/0 multiport dports 591,8591 / 028 HBSS Incoming tcp / state NEW LOG tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports !137,138,139,445 / 990 Log dropped tcp input chain / limit: avg 2/min burst 5 LOG flags 0 level 7 prefix "[IPTABLES]:INPUT:DROP:" LOG udp -- 0.0.0.0/0 0.0.0.0/0 multiport dports !137,138,139,445 / 990 Log dropped udp input chain / limit: avg 2/min burst 5 LOG flags 0 level 7 prefix "[IPTABLES]:INPUT:DROP:" DROP all -- 0.0.0.0/0 0.0.0.0/0 / 991 deny all other input requests / ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 policy match dir in pol ipsec udp dpt:4789 u32 "0x0>>0x16&0x3c@0xc&0xffffff00=0x100300" DROP udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:4789 u32 "0x0>>0x16&0x3c@0xc&0xffffff00=0x100300"

Chain FORWARD (policy DROP) target prot opt source destination DOCKER-USER all -- 0.0.0.0/0 0.0.0.0/0 DOCKER-INGRESS all -- 0.0.0.0/0 0.0.0.0/0 DOCKER-ISOLATION-STAGE-1 all -- 0.0.0.0/0 0.0.0.0/0 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED DOCKER all -- 0.0.0.0/0 0.0.0.0/0 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED DOCKER all -- 0.0.0.0/0 0.0.0.0/0 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 DROP all -- 0.0.0.0/0 0.0.0.0/0 LOG all -- 0.0.0.0/0 0.0.0.0/0 / 990 Log dropped forward chain / limit: avg 2/min burst 5 LOG flags 0 level 7 prefix "[IPTABLES]:FORWARD:DROP:" DROP all -- 0.0.0.0/0 0.0.0.0/0 / 991 deny all other forward requests /

Chain OUTPUT (policy DROP) target prot opt source destination ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 / 001 accept all from lo interface / ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 / 002 accept all related established output traffic / state RELATED,ESTABLISHED ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 multiport sports 67,68 multiport dports 67,68 / 003 accept dhcp out / ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0 / 004 accept ping out / icmptype 8 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 8140 / 006 accept puppet out / ACCEPT esp -- 0.0.0.0/0 0.0.0.0/0 / 006 allow all ipsec output traffic / ACCEPT tcp -- 0.0.0.0/0 10.1.2.11 multiport dports 53 / 010 allow DNS tcp lookups to 10.1.2.11 / state NEW ACCEPT tcp -- 0.0.0.0/0 10.1.3.11 multiport dports 53 / 010 allow DNS tcp lookups to 10.1.3.11 / state NEW ACCEPT udp -- 0.0.0.0/0 10.1.2.11 multiport dports 53 / 010 allow DNS udp lookups to 10.1.2.11 / state NEW ACCEPT udp -- 0.0.0.0/0 10.1.3.11 multiport dports 53 / 010 allow DNS udp lookups to 10.1.3.11 / state NEW ACCEPT udp -- 0.0.0.0/0 10.1.2.11 multiport dports 88,464,389,636 / 015 allow kerberos/ldap UDP out to 10.1.2.11 / state NEW ACCEPT udp -- 0.0.0.0/0 10.1.3.11 multiport dports 88,464,389,636 / 015 allow kerberos/ldap UDP out to 10.1.3.11 / state NEW ACCEPT tcp -- 0.0.0.0/0 10.1.2.11 multiport dports 88,749,389,445,139,3268,636 / 015 allow kerberos/ldap/netbios TCP out to 10.1.2.11 / state NEW ACCEPT tcp -- 0.0.0.0/0 10.1.3.11 multiport dports 88,749,389,445,139,3268,636 / 015 allow kerberos/ldap/netbios TCP out to 10.1.3.11 / state NEW ACCEPT tcp -- 0.0.0.0/0 10.1.2.240 multiport dports 2049 / 015 allow nfs tcp lookups to fs1.mabiaf.com / state NEW ACCEPT tcp -- 0.0.0.0/0 10.1.2.12 multiport dports 2049 / 015 allow nfs tcp lookups to nfsserver.mabiaf.com / state NEW ACCEPT udp -- 0.0.0.0/0 10.1.2.240 multiport dports 2049 / 015 allow nfs udp lookups to fs1.mabiaf.com / state NEW ACCEPT udp -- 0.0.0.0/0 10.1.2.12 multiport dports 2049 / 015 allow nfs udp lookups to nfsserver.mabiaf.com / state NEW ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 123 / 020 allow ntp udp lookups to NTP Server / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 22 / 020 allow ssh out to 0.0.0.0/0.0.0.0 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 25 / 025 allow SMTP out / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 80,443,5000,2376,5432,30015,514,9200,5044,514,515 / 025 allow http, https, and docker TCP out to anywhere / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 111 / 025 allow outgoing tcp connections on port 111 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 1161 / 025 allow outgoing tcp connections on port 1161 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 1883 / 025 allow outgoing tcp connections on port 1883 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 20048 / 025 allow outgoing tcp connections on port 20048 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 2049 / 025 allow outgoing tcp connections on port 2049 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 22 / 025 allow outgoing tcp connections on port 22 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 2376:2377 / 025 allow outgoing tcp connections on port 2376-2377 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 3000:3029 / 025 allow outgoing tcp connections on port 3000-3029 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 443 / 025 allow outgoing tcp connections on port 443 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 46360 / 025 allow outgoing tcp connections on port 46360 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 4789 / 025 allow outgoing tcp connections on port 4789 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 50 / 025 allow outgoing tcp connections on port 50 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 500 / 025 allow outgoing tcp connections on port 500 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 5000 / 025 allow outgoing tcp connections on port 5000 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 50411:50539 / 025 allow outgoing tcp connections on port 50411-50539 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 50555:50556 / 025 allow outgoing tcp connections on port 50555-50556 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 51800:51820 / 025 allow outgoing tcp connections on port 51800-51820 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 51900:51920 / 025 allow outgoing tcp connections on port 51900-51920 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 5432 / 025 allow outgoing tcp connections on port 5432 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 55097 / 025 allow outgoing tcp connections on port 55097 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 61616 / 025 allow outgoing tcp connections on port 61616 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 7946 / 025 allow outgoing tcp connections on port 7946 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 80 / 025 allow outgoing tcp connections on port 80 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 8010 / 025 allow outgoing tcp connections on port 8010 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 8080 / 025 allow outgoing tcp connections on port 8080 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 8082:8083 / 025 allow outgoing tcp connections on port 8082-8083 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 8161 / 025 allow outgoing tcp connections on port 8161 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 8443 / 025 allow outgoing tcp connections on port 8443 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 9012:9013 / 025 allow outgoing tcp connections on port 9012-9013 / state NEW ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 111 / 025 allow outgoing udp connections on port 111 / state NEW ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 20048 / 025 allow outgoing udp connections on port 20048 / state NEW ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 20100:20101 / 025 allow outgoing udp connections on port 20100-20101 / state NEW ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 2049 / 025 allow outgoing udp connections on port 2049 / state NEW ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 4789 / 025 allow outgoing udp connections on port 4789 / state NEW ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 50 / 025 allow outgoing udp connections on port 50 / state NEW ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 500 / 025 allow outgoing udp connections on port 500 / state NEW ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 50411:50475 / 025 allow outgoing udp connections on port 50411-50475 / state NEW ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 52116 / 025 allow outgoing udp connections on port 52116 / state NEW ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 54053 / 025 allow outgoing udp connections on port 54053 / state NEW ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 7946 / 025 allow outgoing udp connections on port 7946 / state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 9092,9093,9094,9095,9096,8443 / 026 allow http, kafka related ports / state NEW ACCEPT tcp -- 0.0.0.0/0 10.1.2.57 multiport dports 443,8443,8444 / 029 HBSS Outgoing / state NEW LOG all -- 0.0.0.0/0 0.0.0.0/0 / 990 Log dropped output chain / limit: avg 2/min burst 5 LOG flags 0 level 7 prefix "[IPTABLES]:OUTPUT:DROP:" DROP all -- 0.0.0.0/0 0.0.0.0/0 / 991 deny all other output requests /

Chain DOCKER (2 references) target prot opt source destination

Chain DOCKER-INGRESS (1 references) target prot opt source destination RETURN all -- 0.0.0.0/0 0.0.0.0/0

Chain DOCKER-ISOLATION (0 references) target prot opt source destination

Chain DOCKER-ISOLATION-STAGE-1 (1 references) target prot opt source destination DOCKER-ISOLATION-STAGE-2 all -- 0.0.0.0/0 0.0.0.0/0 DOCKER-ISOLATION-STAGE-2 all -- 0.0.0.0/0 0.0.0.0/0 RETURN all -- 0.0.0.0/0 0.0.0.0/0

Chain DOCKER-ISOLATION-STAGE-2 (2 references) target prot opt source destination DROP all -- 0.0.0.0/0 0.0.0.0/0 DROP all -- 0.0.0.0/0 0.0.0.0/0 RETURN all -- 0.0.0.0/0 0.0.0.0/0

Chain DOCKER-USER (1 references) target prot opt source destination RETURN all -- 0.0.0.0/0 0.0.0.0/0

Chain KUBE-FIREWALL (0 references) target prot opt source destination

Chain KUBE-SERVICES (0 references) target prot opt source destination

avkudryashov commented 5 years ago

You solved this problem?

jayb1122 commented 4 years ago

Sadly, I believe there's still an issue here. The issue can be replicated in an multi-node environment by closing all ports on all nodes except for the bare minimum required to run docker swarm in encrypted mode... plus any port needed to communicate between containers running on separate nodes.