search

docker-archive / for-aws

92 stars 26 forks source link

Requests to ELB hang indefinitely #121

Open Mobe91 opened 6 years ago

Mobe91 commented 6 years ago

I have a swarm running an nginx service with a single replica. The service exposes ports 80 and 443. Due to frequent configuration changes I frequently removed and recreated the service along with its configs (using docker service rm and docker config rm).

This worked without problems for a while but at some point requests to the public DNS of the ELB started to hang indefinitely. It seems that the requests do not get through to nginx as there are no new entries appended to the nginx access logs (previous successful requests were logged).

I can confirm that the requests are reaching the ELB itself as the surge queue length spikes. Thus, the requests seem to be unable to reach the backend systems (in this case, the swarm running the nginx service).

So to me this looks like an issue with swarm networking. I even tried to recreate the complete nginx stack using docker stack rm nginx which also removes the overlay network. But even a fresh network does not resolve this issue.

Here is the output of docker service ls:

ID                  NAME                MODE                REPLICAS            IMAGE                 PORTS
e9ll32t87yf5        nginx_nginx         global              1/1                 nginx:1.13.6-alpine   *:80->80/tcp,*:443->443/tcp
Mobe91 commented 6 years ago

What I also tried now is to restart the EC2-Instance running the manager: no success After that, I restarted the worker EC2-Instance running the nginx container: this resolved the issue

What could be the cause for this behavior?

FrenchBen commented 6 years ago

@Mobe91 Could you provide the output of: $curl -iL http://<ELB-endpoint> $curl -iL http://<machine-public-ip> $curl -iL http://<machine-private-ip> $curl -iL http://localhost

Mobe91 commented 6 years ago

I executed the commands from within the nginx container. The results are as follows:

$curl -iL http://<ELB-endpoint>
HTTP/1.1 200 OK
Server: nginx/1.13.6
Date: Sun, 19 Nov 2017 16:21:00 GMT
Content-Type: text/html
Content-Length: 13346
Connection: keep-alive
Last-Modified: Wed, 15 Nov 2017 13:44:53 GMT
ETag: "5a0c44d5-3422"
Accept-Ranges: bytes

<lots of html>
$curl -iL http://<machine-public-ip>
curl: (7) Failed to connect to 18.194.240.53 port 80: Operation timed out
$curl -iL http://<machine-private-ip>
HTTP/1.1 200 OK
Server: nginx/1.13.6
Date: Sun, 19 Nov 2017 16:27:26 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Fri, 03 Nov 2017 22:37:08 GMT
Connection: keep-alive
ETag: "59fcef94-264"
Accept-Ranges: bytes

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
$curl -iL http://localhost
HTTP/1.1 200 OK
Server: nginx/1.13.6
Date: Sun, 19 Nov 2017 16:29:48 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Fri, 03 Nov 2017 22:37:08 GMT
Connection: keep-alive
ETag: "59fcef94-264"
Accept-Ranges: bytes

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
Mobe91 commented 6 years ago

Ok it just happened again, here are the outputs of the commands in an erroneous state:

$curl -iL http://<ELB-endpoint>

Runs for minutes and then returns curl: (52) Empty reply from server.

$curl -iL http://<machine-public-ip>
curl: (7) Failed to connect to 18.194.240.53 port 80: Operation timed out
$curl -iL http://<machine-private-ip>
curl: (7) Failed to connect to 172.31.0.232 port 80: Operation timed out
$curl -iL http://localhost
HTTP/1.1 200 OK
Server: nginx/1.13.6
Date: Sun, 19 Nov 2017 17:11:29 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Fri, 03 Nov 2017 22:37:08 GMT
Connection: keep-alive
ETag: "59fcef94-264"
Accept-Ranges: bytes

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

And again, a reboot of the worker instance resolved the issue.

FrenchBen commented 6 years ago

could you run a docker-diagnose command and provide us with the diagnostic ID? https://docs.docker.com/docker-for-aws/faqs/#where-do-i-report-problems-or-bugs

Mobe91 commented 6 years ago 
OK hostname=ip-172-31-4-42-eu-central-1-compute-internal session=1511295594-hrXIAwGyjlYx4lvk3b9ddaedsSsISe8t
OK hostname=ip-172-31-0-232-eu-central-1-compute-internal session=1511295594-hrXIAwGyjlYx4lvk3b9ddaedsSsISe8t
Done requesting diagnostics.
Your diagnostics session ID is 1511295594-hrXIAwGyjlYx4lvk3b9ddaedsSsISe8t
Please provide this session ID to the maintainer debugging your issue.
FrenchBen commented 6 years ago

Could you look in AWS for the ASG history? i'm seeing a lot of churn on your worker VM. It seems to be requesting to join the swarm almost every 5mins, which means that the healthcheck at the ELB level could have failed and the node is getting replaced.

if it's easier, please join our Community slack channel: https://blog.docker.com/2016/11/introducing-docker-community-directory-docker-community-slack/

Mobe91 commented 6 years ago

Here is the complete history of worker and manager ASG. Does not look like instances are being replaced.

Worker ASG History

  Status Description Start Time End Time
  Successful Terminating EC2 instance: i-00cf0b9db161166bf 2017 October 28 10:34:45 UTC+2 2017 October 28 10:36:11 UTC+2
  Successful Launching a new EC2 instance: i-00cf0b9db161166bf 2017 October 28 09:47:08 UTC+2 2017 October 28 09:47:41 UTC+2
  Successful Launching a new EC2 instance: i-07b94f434982ff51f 2017 October 27 22:58:49 UTC+2 2017 October 27 22:59:22 UTC+2

Manager ASG History

  Status Description Start Time End Time
  Successful Launching a new EC2 instance: i-0555e410bad276e6a 2017 October 27 22:56:28 UTC+2 2017 October 27 22:57:01 UTC+2
Mobe91 commented 6 years ago

Just happened again

EDIT: ok, false alarm this time it was a different issue..

joel1st commented 6 years ago

We've seen the same issue:

Connectivity inside of the docker network is stable, but there is intermittent connectivity for a period of time after deploying/updating a stack or service when attempting to hit the ec2 or load balancer publicly.

We resolved this by setting the following in our compose/stack file: Note the ports mode host and deploy mode global. It now works immediately as soon as the service is up and running with no intermittent connectivity issues. 

  gateway:
    image: 'nginx'
    ports:
      - target: 80
        published: 80
        protocol: tcp
        mode: host
      - target: 443
        published: 443
        protocol: tcp
        mode: host
    deploy:
      mode: global

This seems to have presented a new and exciting bug, where you have to manually add port 80 and port 443 to your listening ports on the ELB as it doesn't automatically propogate to the ELB like the regular ports do (and in fact it can get overwritten on stack changes).

FrenchBen commented 6 years ago

@joel1st Thanks for sharing - What you see as a "bug" is actually working as expected by design. See the docs: https://docs.docker.com/engine/swarm/services/#publish-ports By enabling mode: host you are exposing the port on a specific node, and thus the ELB is pointless, as only 1 node can respond to a request on that port.

FYI: there's a PR in flight to get some of the delay that you're experiencing fixed: https://github.com/docker/libnetwork/pull/1824

joel1st commented 6 years ago

Will be interesting to see if that PR fixes the issues we are seeing. Combining port mode: host with deploy mode: global means that the port is exposed on all nodes, and thus using the ELB is not pointless.

FrenchBen commented 6 years ago

@joel1st I see what you're saying - Seems a bit of overkill to get your services to work.

joel1st commented 6 years ago

Yeah definitely feels overkill - the alternative (before the PR gets accepted) is 30/40 minutes of intermittent connectivity - which isn't acceptable for prod. For the meantime we've scripted a post release step to manually update the load balancer to open port 80 and 443.

Mobe91 commented 6 years ago

@FrenchBen Ok now it just happened again during config rotation on one of my services. I.e. I was running docker service update --config-rm <old-config> --config-add <new-config> and the update was successful, i.e. all tasks (just 1 in this case) restarted correctly using the new config. But then all connections through my ELB to this service hung indefinitely. After rebooting the worker running the service task, everything went back to normal.

My service uses the deploy mode global.

~ $ docker-diagnose
OK hostname=ip-172-31-4-42-eu-central-1-compute-internal session=1512726823-uTtGwtZgRUNxblKQe7LOlJlOmo1N2TtP
OK hostname=ip-172-31-0-232-eu-central-1-compute-internal session=1512726823-uTtGwtZgRUNxblKQe7LOlJlOmo1N2TtP
Done requesting diagnostics.
Your diagnostics session ID is 1512726823-uTtGwtZgRUNxblKQe7LOlJlOmo1N2TtP
Please provide this session ID to the maintainer debugging your issue.
FrenchBen commented 6 years ago

@mobe91 can you also provide the logs associated with the l4controller that match he service update?

Mobe91 commented 6 years ago

I will try to retrieve the wanted logs the next time this happens.

Mobe91 commented 6 years ago

@FrenchBen It happened again, this time with a service that does not use deploy mode global. I currently have 3 managers (A, B and C) in my swarm and I retrieved the l4controller logs as requested but I am not sure if these logs actually belong to my service update as the timestamps do not really match. However, there are no other logs for the l4controller...

Even when running docker service update --force <my-service> the log output did not change.

The respective service has a port mapping 8082:8080.

Managers A and B have about the same log output:

time="2018-02-27T15:47:47Z" level=info msg="9 matches found. Processing."
time="2018-02-27T15:47:47Z" level=info msg="exposedPorts:  map[80:{ tcp 80 80 ingress}]"
time="2018-02-27T15:47:47Z" level=info msg="requestedPublishPorts:  map[80:[80]]"
time="2018-02-27T15:47:47Z" level=info msg="exposed:  { tcp 80 80 ingress}"
time="2018-02-27T15:47:47Z" level=info msg="Cert:  <nil>"
time="2018-02-27T15:47:47Z" level=info msg="urlString:  tcp://:80"
time="2018-02-27T15:47:47Z" level=info msg="exposedPorts:  map[8081:{ tcp 8080 8081 ingress}]"
time="2018-02-27T15:47:47Z" level=info msg="requestedPublishPorts:  map[8080:[8081]]"
time="2018-02-27T15:47:47Z" level=info msg="exposed:  { tcp 8080 8081 ingress}"
time="2018-02-27T15:47:47Z" level=info msg="Cert:  <nil>"
time="2018-02-27T15:47:47Z" level=info msg="urlString:  tcp://:8081"
time="2018-02-27T15:47:47Z" level=info msg="exposedPorts:  map[8084:{ tcp 80 8084 ingress}]"
time="2018-02-27T15:47:47Z" level=info msg="requestedPublishPorts:  map[80:[8084]]"
time="2018-02-27T15:47:47Z" level=info msg="exposed:  { tcp 80 8084 ingress}"
time="2018-02-27T15:47:47Z" level=info msg="Cert:  <nil>"
time="2018-02-27T15:47:47Z" level=info msg="urlString:  tcp://:8084"
time="2018-02-27T15:47:47Z" level=info msg="exposedPorts:  map[8085:{ tcp 8080 8085 ingress}]"
time="2018-02-27T15:47:47Z" level=info msg="requestedPublishPorts:  map[8080:[8085]]"
time="2018-02-27T15:47:47Z" level=info msg="exposed:  { tcp 8080 8085 ingress}"
time="2018-02-27T15:47:47Z" level=info msg="Cert:  <nil>"
time="2018-02-27T15:47:47Z" level=info msg="urlString:  tcp://:8085"
time="2018-02-27T15:47:47Z" level=info msg="exposedPorts:  map[8080:{ tcp 8080 8080 ingress}]"
time="2018-02-27T15:47:47Z" level=info msg="requestedPublishPorts:  map[8080:[8080]]"
time="2018-02-27T15:47:47Z" level=info msg="exposed:  { tcp 8080 8080 ingress}"
time="2018-02-27T15:47:47Z" level=info msg="Cert:  <nil>"
time="2018-02-27T15:47:47Z" level=info msg="urlString:  tcp://:8080"
time="2018-02-27T15:47:47Z" level=info msg="exposedPorts:  map[11111:{ tcp 11111 11111 ingress}]"
time="2018-02-27T15:47:47Z" level=info msg="requestedPublishPorts:  map[11111:[11111]]"
time="2018-02-27T15:47:47Z" level=info msg="exposed:  { tcp 11111 11111 ingress}"
time="2018-02-27T15:47:47Z" level=info msg="Cert:  <nil>"
time="2018-02-27T15:47:47Z" level=info msg="urlString:  tcp://:11111"
time="2018-02-27T15:47:47Z" level=info msg="exposedPorts:  map[8086:{ tcp 8080 8086 ingress}]"
time="2018-02-27T15:47:47Z" level=info msg="requestedPublishPorts:  map[8080:[8086]]"
time="2018-02-27T15:47:47Z" level=info msg="exposed:  { tcp 8080 8086 ingress}"
time="2018-02-27T15:47:47Z" level=info msg="Cert:  <nil>"
time="2018-02-27T15:47:47Z" level=info msg="urlString:  tcp://:8086"
time="2018-02-27T15:47:47Z" level=info msg="exposedPorts:  map[8082:{ tcp 8080 8082 ingress}]"
time="2018-02-27T15:47:47Z" level=info msg="requestedPublishPorts:  map[8080:[8082]]"
time="2018-02-27T15:47:47Z" level=info msg="exposed:  { tcp 8080 8082 ingress}"
time="2018-02-27T15:47:47Z" level=info msg="Cert:  <nil>"
time="2018-02-27T15:47:47Z" level=info msg="urlString:  tcp://:8082"
time="2018-02-27T15:47:47Z" level=info msg="exposedPorts:  map[8083:{ tcp 8080 8083 ingress}]"
time="2018-02-27T15:47:47Z" level=info msg="requestedPublishPorts:  map[8080:[8083]]"
time="2018-02-27T15:47:47Z" level=info msg="exposed:  { tcp 8080 8083 ingress}"
time="2018-02-27T15:47:47Z" level=info msg="Cert:  <nil>"
time="2018-02-27T15:47:47Z" level=info msg="urlString:  tcp://:8083"
time="2018-02-27T15:47:47Z" level=info msg="Read config: # hostname : ELB_name\n127.0.0.1: DockerBac-Internal-AYCKM81LV1SW\nlocalhost: DockerBac-Internal-AYCKM81LV1SW\ndefault: DockerBac-Internal-AYCKM81LV1SW\n# hostname : ELB_name\n127.0.0.1: DockerBac-Internal-AYCKM81LV1SW\nlocalhost: DockerBac-Internal-AYCKM81LV1SW\ndefault: DockerBac-Internal-AYCKM81LV1SW\n"
time="2018-02-27T15:47:47Z" level=info msg="ELB mapping: map[127.0.0.1:DockerBac-Internal-AYCKM81LV1SW localhost:DockerBac-Internal-AYCKM81LV1SW default:DockerBac-Internal-AYCKM81LV1SW]"
time="2018-02-27T15:47:47Z" level=info msg="ELB Client in region eu-central-1"
time="2018-02-27T15:47:47Z" level=info msg="Located external load balancer DockerBac-Internal-AYCKM81LV1SW for 127.0.0.1"
time="2018-02-27T15:47:47Z" level=info msg="Located external load balancer DockerBac-Internal-AYCKM81LV1SW for localhost"
time="2018-02-27T15:47:47Z" level=info msg="Located external load balancer DockerBac-Internal-AYCKM81LV1SW for default"
time="2018-02-27T15:47:47Z" level=info msg="Configuring DockerBac-Internal-AYCKM81LV1SW"
2018/02/27 15:47:47 DEBUG: Request elasticloadbalancing/DescribeLoadBalancers Details:
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: elasticloadbalancing.eu-central-1.amazonaws.com
User-Agent: aws-sdk-go/1.4.3 (go1.7.6; linux; amd64)
Content-Length: 106
Authorization: AWS4-HMAC-SHA256 Credential=ASIAIGJ3GRWONGJJU3NQ/20180227/eu-central-1/elasticloadbalancing/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date;x-amz-security-token, Signature=ce854ea4df677da2c28854a0f404b18c74fc9c034a89819d3d249ceca408b76a
Content-Type: application/x-www-form-urlencoded; charset=utf-8
X-Amz-Date: 20180227T154747Z
X-Amz-Security-Token: FQoDYXdzEMD//////////wEaDBmkdc1j1d/mcaqV8yK9AzbZyUqXnu2w2Isp5po7QzB1ivEIZyBrveSuQYTMHWrYMcmbQu3Zx+XB3+MhphxLyUOsLnBNZYRFzXNFfTFmxXqkrXn49eskhYekZ881mO/s61sCZKErDFmid8NkwYwHSKYjGMZaEFbu2xCsrRO/1lPwG5kfdq3MB0oX9bdSJoHkCQbwVAiK60O+U9k/7nF7moMNY7J4NCWtI0TPxflsgTCUODNRYlAR4WP6ZQ92u8E3Q0JV70HDaupWC0HAUe4Vm3Ir1sODD9Sg/GEa6YRraxKOSMj7+u+1x7iGGycxblUnImMXbLKwqo8JZ+ka8Uwif+KE0MqYEvkN/HFGCCMbrGVtmMbkzsy61NhRm1iWD+yqjuNShl2KXcxKWUp0oeJ9CzF3j7tDa9HCH2P0mSwl1ZSuYob0xGdkfpZnCr8JW9hlikgB3Ic9dtdVz4DZNo/gOVqUAZJVR0IQkdW00R0XNFW2iZFZuyxqvw2s5F6TIWigIm2zOM5TY8XQP4SL1N0I5EnmDpJTSeeuDDScfjfqbcTcGGt8+MNkppG36R3kDHFX13Q4MvoKP8RmnoWqUR4DYn13QcgtBXjMQCIPeYQo4unV1AU=
Accept-Encoding: gzip

Action=DescribeLoadBalancers&LoadBalancerNames.member.1=DockerBac-Internal-AYCKM81LV1SW&Version=2012-06-01
-----------------------------------------------------
2018/02/27 15:47:47 DEBUG: Response elasticloadbalancing/DescribeLoadBalancers Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: text/xml
Date: Tue, 27 Feb 2018 15:47:47 GMT
Vary: Accept-Encoding
X-Amzn-Requestid: 8f2cec9c-1bd5-11e8-85cd-abe0e118119e

15b1
<DescribeLoadBalancersResponse xmlns="http://elasticloadbalancing.amazonaws.com/doc/2012-06-01/">
  <DescribeLoadBalancersResult>
    <LoadBalancerDescriptions>
      <member>
        <VPCId>vpc-c913e2a2</VPCId>
        <CanonicalHostedZoneNameID>Z215JYRZR1TBD5</CanonicalHostedZoneNameID>
        <Scheme>internal</Scheme>
        <Instances>
          <member>
            <InstanceId>i-0684f04b68548f888</InstanceId>
          </member>
          <member>
            <InstanceId>i-0ef35af86c7718b55</InstanceId>
          </member>
          <member>
            <InstanceId>i-0da1a7304ffad71fc</InstanceId>
          </member>
          <member>
            <InstanceId>i-0ba03b9a2159c3204</InstanceId>
          </member>
          <member>
            <InstanceId>i-0341ad76d2638d2c5</InstanceId>
          </member>
          <member>
            <InstanceId>i-0b16e11b4e52a18a1</InstanceId>
          </member>
        </Instances>
        <Policies>
          <AppCookieStickinessPolicies/>
          <LBCookieStickinessPolicies/>
          <OtherPolicies/>
        </Policies>
        <AvailabilityZones>
          <member>eu-central-1a</member>
          <member>eu-central-1b</member>
        </AvailabilityZones>
        <DNSName>internal-DockerBac-Internal-AYCKM81LV1SW-1848568591.eu-central-1.elb.amazonaws.com</DNSName>
        <BackendServerDescriptions/>
        <SourceSecurityGroup>
          <OwnerAlias>295620090465</OwnerAlias>
          <GroupName>DockerBackend-InternalLoadBalancerSG-HLRQQBXLTCDD</GroupName>
        </SourceSecurityGroup>
        <ListenerDescriptions>
          <member>
            <Listener>
              <InstancePort>8084</InstancePort>
              <InstanceProtocol>TCP</InstanceProtocol>
              <Protocol>TCP</Protocol>
              <LoadBalancerPort>8084</LoadBalancerPort>
            </Listener>
            <PolicyNames/>
          </member>
          <member>
            <Listener>
              <InstancePort>8085</InstancePort>
              <InstanceProtocol>TCP</InstanceProtocol>
              <Protocol>TCP</Protocol>
              <LoadBalancerPort>8085</LoadBalancerPort>
            </Listener>
            <PolicyNames/>
          </member>
          <member>
            <Listener>
              <InstancePort>8086</InstancePort>
              <InstanceProtocol>TCP</InstanceProtocol>
              <Protocol>TCP</Protocol>
              <LoadBalancerPort>8086</LoadBalancerPort>
            </Listener>
            <PolicyNames/>
          </member>
          <member>
            <Listener>
              <InstancePort>8083</InstancePort>
              <InstanceProtocol>TCP</InstanceProtocol>
              <Protocol>TCP</Protocol>
              <LoadBalancerPort>8083</LoadBalancerPort>
            </Listener>
            <PolicyNames/>
          </member>
          <member>
            <Listener>
              <InstancePort>80</InstancePort>
              <InstanceProtocol>TCP</InstanceProtocol>
              <Protocol>TCP</Protocol>
              <LoadBalancerPort>80</LoadBalancerPort>
            </Listener>
            <PolicyNames/>
          </member>
          <member>
            <Listener>
              <InstancePort>8080</InstancePort>
              <InstanceProtocol>TCP</InstanceProtocol>
              <Protocol>TCP</Protocol>
              <LoadBalancerPort>8080</LoadBalancerPort>
            </Listener>
            <PolicyNames/>
          </member>
          <member>
            <Listener>
              <InstancePort>8081</InstancePort>
              <InstanceProtocol>TCP</InstanceProtocol>
              <Protocol>TCP</Protocol>
              <LoadBalancerPort>8081</LoadBalancerPort>
            </Listener>
            <PolicyNames/>
          </member>
          <member>
            <Listener>
              <InstancePort>11111</InstancePort>
              <InstanceProtocol>TCP</InstanceProtocol>
              <Protocol>TCP</Protocol>
              <LoadBalancerPort>11111</LoadBalancerPort>
            </Listener>
            <PolicyNames/>
          </member>
          <member>
            <Listener>
              <InstancePort>7</InstancePort>
              <InstanceProtocol>TCP</InstanceProtocol>
              <Protocol>TCP</Protocol>
              <LoadBalancerPort>7</LoadBalancerPort>
            </Listener>
            <PolicyNames/>
          </member>
          <member>
            <Listener>
              <InstancePort>8082</InstancePort>
              <InstanceProtocol>TCP</InstanceProtocol>
              <Protocol>TCP</Protocol>
              <LoadBalancerPort>8082</LoadBalancerPort>
            </Listener>
            <PolicyNames/>
          </member>
        </ListenerDescriptions>
        <LoadBalancerName>DockerBac-Internal-AYCKM81LV1SW</LoadBalancerName>
        <HealthCheck>
          <UnhealthyThreshold>4</UnhealthyThreshold>
          <Interval>10</Interval>
          <HealthyThreshold>2</HealthyThreshold>
          <Timeout>8</Timeout>
          <Target>HTTP:44554/</Target>
        </HealthCheck>
        <CreatedTime>2017-12-12T00:32:55.650Z</CreatedTime>
        <SecurityGroups>
          <member>sg-2d2c5347</member>
        </SecurityGroups>
        <Subnets>
          <member>subnet-16728c6b</member>
          <member>subnet-4872f723</member>
        </Subnets>
      </member>
    </LoadBalancerDescriptions>
  </DescribeLoadBalancersResult>
  <ResponseMetadata>
    <RequestId>8f2cec9c-1bd5-11e8-85cd-abe0e118119e</RequestId>
  </ResponseMetadata>
</DescribeLoadBalancersResponse>

0

-----------------------------------------------------
time="2018-02-27T15:47:47Z" level=info msg="Listeners to sync with ELB: [service(ordami-website_ordami-website):80 ==> tcp://:80 service(nominatim_nominatim):8081 ==> tcp://:8081 service(ordami-backoffice-staging_ordami-backoffice):8084 ==> tcp://:8084 service(ordami-wko-api-staging_ordami-wko-api):8085 ==> tcp://:8085 service(ordami-backend-staging_ordami-backend):8080 ==> tcp://:8080 service(graphhopper_graph-hopper):11111 ==> tcp://:11111 service(ordami-wko-sync-staging_ordami-wko-sync):8086 ==> tcp://:8086 service(keycloak_keycloak):8082 ==> tcp://:8082 service(visualizer_visualizer):8083 ==> tcp://:8083]"
time="2018-02-27T15:47:47Z" level=info msg="cert is nil, or port  80  Is NOT in  [{443 SSL}]"
time="2018-02-27T15:47:47Z" level=info msg="cert is nil, or port  80  Is NOT in  [{443 SSL}]"
time="2018-02-27T15:47:47Z" level=info msg="cert is nil, or port  8081  Is NOT in  [{443 SSL}]"
time="2018-02-27T15:47:47Z" level=info msg="cert is nil, or port  8081  Is NOT in  [{443 SSL}]"
time="2018-02-27T15:47:47Z" level=info msg="cert is nil, or port  8084  Is NOT in  [{443 SSL}]"
time="2018-02-27T15:47:47Z" level=info msg="cert is nil, or port  8084  Is NOT in  [{443 SSL}]"
time="2018-02-27T15:47:47Z" level=info msg="cert is nil, or port  8085  Is NOT in  [{443 SSL}]"
time="2018-02-27T15:47:47Z" level=info msg="cert is nil, or port  8085  Is NOT in  [{443 SSL}]"
time="2018-02-27T15:47:47Z" level=info msg="cert is nil, or port  8080  Is NOT in  [{443 SSL}]"
time="2018-02-27T15:47:47Z" level=info msg="cert is nil, or port  8080  Is NOT in  [{443 SSL}]"
time="2018-02-27T15:47:47Z" level=info msg="cert is nil, or port  11111  Is NOT in  [{443 SSL}]"
time="2018-02-27T15:47:47Z" level=info msg="cert is nil, or port  11111  Is NOT in  [{443 SSL}]"
time="2018-02-27T15:47:47Z" level=info msg="cert is nil, or port  8086  Is NOT in  [{443 SSL}]"
time="2018-02-27T15:47:47Z" level=info msg="cert is nil, or port  8086  Is NOT in  [{443 SSL}]"
time="2018-02-27T15:47:47Z" level=info msg="cert is nil, or port  8082  Is NOT in  [{443 SSL}]"
time="2018-02-27T15:47:47Z" level=info msg="cert is nil, or port  8082  Is NOT in  [{443 SSL}]"
time="2018-02-27T15:47:47Z" level=info msg="cert is nil, or port  8083  Is NOT in  [{443 SSL}]"
time="2018-02-27T15:47:47Z" level=info msg="cert is nil, or port  8083  Is NOT in  [{443 SSL}]"
time="2018-02-27T15:47:47Z" level=info msg="keeping protocol= TCP port= 8084 instancePort= 8084"
time="2018-02-27T15:47:47Z" level=info msg="keeping protocol= TCP port= 8085 instancePort= 8085"
time="2018-02-27T15:47:47Z" level=info msg="keeping protocol= TCP port= 8086 instancePort= 8086"
time="2018-02-27T15:47:47Z" level=info msg="keeping protocol= TCP port= 8083 instancePort= 8083"
time="2018-02-27T15:47:47Z" level=info msg="keeping protocol= TCP port= 80 instancePort= 80"
time="2018-02-27T15:47:47Z" level=info msg="keeping protocol= TCP port= 8080 instancePort= 8080"
time="2018-02-27T15:47:47Z" level=info msg="keeping protocol= TCP port= 8081 instancePort= 8081"
time="2018-02-27T15:47:47Z" level=info msg="keeping protocol= TCP port= 11111 instancePort= 11111"
time="2018-02-27T15:47:47Z" level=info msg="keeping protocol= TCP port= 7 instancePort= 7"
time="2018-02-27T15:47:47Z" level=info msg="keeping protocol= TCP port= 8082 instancePort= 8082"
time="2018-02-27T15:47:47Z" level=info msg="listeners to create: []"
time="2018-02-27T15:47:47Z" level=info msg="listeners to change: []"
time="2018-02-27T15:47:47Z" level=info msg="listeners to remove: []"

Weirdly, manager C only has this output for the respective period:

time="2018-02-27T15:47:01Z" level=info msg="Not a leader.  Check back later"
time="2018-02-27T15:47:04Z" level=info msg="Not a leader.  Check back later"
time="2018-02-27T15:47:07Z" level=info msg="Not a leader.  Check back later"
time="2018-02-27T15:47:10Z" level=info msg="Not a leader.  Check back later"
time="2018-02-27T15:47:13Z" level=info msg="Not a leader.  Check back later"
time="2018-02-27T15:47:16Z" level=info msg="Not a leader.  Check back later"
time="2018-02-27T15:47:19Z" level=info msg="Not a leader.  Check back later"
time="2018-02-27T15:47:22Z" level=info msg="Not a leader.  Check back later"
time="2018-02-27T15:47:25Z" level=info msg="Not a leader.  Check back later"
time="2018-02-27T15:47:28Z" level=info msg="Not a leader.  Check back later"
time="2018-02-27T15:47:31Z" level=info msg="Not a leader.  Check back later"
time="2018-02-27T15:47:34Z" level=info msg="Not a leader.  Check back later"
time="2018-02-27T15:47:37Z" level=info msg="Not a leader.  Check back later"
time="2018-02-27T15:47:40Z" level=info msg="Not a leader.  Check back later"
time="2018-02-27T15:47:43Z" level=info msg="Not a leader.  Check back later"
time="2018-02-27T15:47:46Z" level=info msg="Not a leader.  Check back later"
time="2018-02-27T15:47:49Z" level=info msg="Not a leader.  Check back later"
time="2018-02-27T15:47:52Z" level=info msg="Not a leader.  Check back later"
time="2018-02-27T15:47:56Z" level=info msg="Not a leader.  Check back later"
time="2018-02-27T15:47:58Z" level=info msg="Not a leader.  Check back later"

It seems that manager C has never logged anything else than that.

I then removed the corresponding stack entirely by running docker stack rm <stack> and only then, the log output in managers A and B was updated. This time, there was a difference because only the first logged the removal of the ELB listener.

Manager A:

time="2018-02-27T17:21:30Z" level=info msg="8 matches found. Processing."
time="2018-02-27T17:21:30Z" level=info msg="exposedPorts:  map[80:{ tcp 80 80 ingress}]"
time="2018-02-27T17:21:30Z" level=info msg="requestedPublishPorts:  map[80:[80]]"
time="2018-02-27T17:21:30Z" level=info msg="exposed:  { tcp 80 80 ingress}"
time="2018-02-27T17:21:30Z" level=info msg="Cert:  <nil>"
time="2018-02-27T17:21:30Z" level=info msg="urlString:  tcp://:80"
time="2018-02-27T17:21:30Z" level=info msg="exposedPorts:  map[8081:{ tcp 8080 8081 ingress}]"
time="2018-02-27T17:21:30Z" level=info msg="requestedPublishPorts:  map[8080:[8081]]"
time="2018-02-27T17:21:30Z" level=info msg="exposed:  { tcp 8080 8081 ingress}"
time="2018-02-27T17:21:30Z" level=info msg="Cert:  <nil>"
time="2018-02-27T17:21:30Z" level=info msg="urlString:  tcp://:8081"
time="2018-02-27T17:21:30Z" level=info msg="exposedPorts:  map[8084:{ tcp 80 8084 ingress}]"
time="2018-02-27T17:21:30Z" level=info msg="requestedPublishPorts:  map[80:[8084]]"
time="2018-02-27T17:21:30Z" level=info msg="exposed:  { tcp 80 8084 ingress}"
time="2018-02-27T17:21:30Z" level=info msg="Cert:  <nil>"
time="2018-02-27T17:21:30Z" level=info msg="urlString:  tcp://:8084"
time="2018-02-27T17:21:30Z" level=info msg="exposedPorts:  map[8085:{ tcp 8080 8085 ingress}]"
time="2018-02-27T17:21:30Z" level=info msg="requestedPublishPorts:  map[8080:[8085]]"
time="2018-02-27T17:21:30Z" level=info msg="exposed:  { tcp 8080 8085 ingress}"
time="2018-02-27T17:21:30Z" level=info msg="Cert:  <nil>"
time="2018-02-27T17:21:30Z" level=info msg="urlString:  tcp://:8085"
time="2018-02-27T17:21:30Z" level=info msg="exposedPorts:  map[8080:{ tcp 8080 8080 ingress}]"
time="2018-02-27T17:21:30Z" level=info msg="requestedPublishPorts:  map[8080:[8080]]"
time="2018-02-27T17:21:30Z" level=info msg="exposed:  { tcp 8080 8080 ingress}"
time="2018-02-27T17:21:30Z" level=info msg="Cert:  <nil>"
time="2018-02-27T17:21:30Z" level=info msg="urlString:  tcp://:8080"
time="2018-02-27T17:21:30Z" level=info msg="exposedPorts:  map[11111:{ tcp 11111 11111 ingress}]"
time="2018-02-27T17:21:30Z" level=info msg="requestedPublishPorts:  map[11111:[11111]]"
time="2018-02-27T17:21:30Z" level=info msg="exposed:  { tcp 11111 11111 ingress}"
time="2018-02-27T17:21:30Z" level=info msg="Cert:  <nil>"
time="2018-02-27T17:21:30Z" level=info msg="urlString:  tcp://:11111"
time="2018-02-27T17:21:30Z" level=info msg="exposedPorts:  map[8086:{ tcp 8080 8086 ingress}]"
time="2018-02-27T17:21:30Z" level=info msg="requestedPublishPorts:  map[8080:[8086]]"
time="2018-02-27T17:21:30Z" level=info msg="exposed:  { tcp 8080 8086 ingress}"
time="2018-02-27T17:21:30Z" level=info msg="Cert:  <nil>"
time="2018-02-27T17:21:30Z" level=info msg="urlString:  tcp://:8086"
time="2018-02-27T17:21:30Z" level=info msg="exposedPorts:  map[8083:{ tcp 8080 8083 ingress}]"
time="2018-02-27T17:21:30Z" level=info msg="requestedPublishPorts:  map[8080:[8083]]"
time="2018-02-27T17:21:30Z" level=info msg="exposed:  { tcp 8080 8083 ingress}"
time="2018-02-27T17:21:30Z" level=info msg="Cert:  <nil>"
time="2018-02-27T17:21:30Z" level=info msg="urlString:  tcp://:8083"
time="2018-02-27T17:21:30Z" level=info msg="Read config: # hostname : ELB_name\n127.0.0.1: DockerBac-Internal-AYCKM81LV1SW\nlocalhost: DockerBac-Internal-AYCKM81LV1SW\ndefault: DockerBac-Internal-AYCKM81LV1SW\n# hostname : ELB_name\n127.0.0.1: DockerBac-Internal-AYCKM81LV1SW\nlocalhost: DockerBac-Internal-AYCKM81LV1SW\ndefault: DockerBac-Internal-AYCKM81LV1SW\n"
time="2018-02-27T17:21:30Z" level=info msg="ELB mapping: map[127.0.0.1:DockerBac-Internal-AYCKM81LV1SW localhost:DockerBac-Internal-AYCKM81LV1SW default:DockerBac-Internal-AYCKM81LV1SW]"
time="2018-02-27T17:21:31Z" level=info msg="ELB Client in region eu-central-1"
time="2018-02-27T17:21:31Z" level=info msg="Located external load balancer DockerBac-Internal-AYCKM81LV1SW for 127.0.0.1"
time="2018-02-27T17:21:31Z" level=info msg="Located external load balancer DockerBac-Internal-AYCKM81LV1SW for localhost"
time="2018-02-27T17:21:31Z" level=info msg="Located external load balancer DockerBac-Internal-AYCKM81LV1SW for default"
time="2018-02-27T17:21:31Z" level=info msg="Configuring DockerBac-Internal-AYCKM81LV1SW"
2018/02/27 17:21:31 DEBUG: Request elasticloadbalancing/DescribeLoadBalancers Details:
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: elasticloadbalancing.eu-central-1.amazonaws.com
User-Agent: aws-sdk-go/1.4.3 (go1.7.6; linux; amd64)
Content-Length: 106
Authorization: AWS4-HMAC-SHA256 Credential=ASIAIUBVHMMZSCPOV6YQ/20180227/eu-central-1/elasticloadbalancing/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date;x-amz-security-token, Signature=f1b546ed4fe7ce2d9d8feee49abe732fbfd65bee7769b1d071a0e1842e4e92b2
Content-Type: application/x-www-form-urlencoded; charset=utf-8
X-Amz-Date: 20180227T172131Z
X-Amz-Security-Token: FQoDYXdzEML//////////wEaDOdLjWgGDpcsIL5KjiK9A+QaLwF23DgYn1dDX0n3Ie6Z7YpY3Hk7jjxU1ejxstA0NsU4Kwn8VEAxEFBRNxfcp3kaKMz0VmVz9WII4/gm+yLtfgC3bIgpHF26fDGUx41U+Ru+IfZerFZGNq2aQvriZIVWIhEQzDNHfRx0CAeMK0J7uY5GSk4R4hAaW85Wxb9gElQUdfG4r84ZA1VKo/zDlrvxUHpA0Kugwknad6ZR0X8IRINPq4VWzpVwWKmDQ5WYJUnz1cFZf0G2QtC5M0mglC0ZOiqwD6mVva7Xlild8+FWMb/CH/Q4PWvSQqOdV4r6SYyaGGD0wddonv20auGJEsKCLiiCbZPhFOVesKUfKsRiccH2CtqrXlwmOzv/OJUWwhlwog5nz2lDsojjkyFMniUhGWYZFfrUfrWl05W+wFUX8BJw8tdbB0rG247MYo7dN6IfGvWZMUFZYY++4vhMNolB07Hi74vn5g7vEIo4QprDVGe0S1tU0BSt0SSDYCTXrcyiwz7QlLLjdcncV06hPiAm+1kZCQA/43CQB9f/rkG5wL9KmtlzjHSP87DIMPBpKqvNsvCbLQ/rsZpL2gZqstCTfrLVB1RZm3vRV5ooz5DW1AU=
Accept-Encoding: gzip

Action=DescribeLoadBalancers&LoadBalancerNames.member.1=DockerBac-Internal-AYCKM81LV1SW&Version=2012-06-01
-----------------------------------------------------
2018/02/27 17:21:31 DEBUG: Response elasticloadbalancing/DescribeLoadBalancers Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: text/xml
Date: Tue, 27 Feb 2018 17:21:30 GMT
Vary: Accept-Encoding
X-Amzn-Requestid: a6e0d05e-1be2-11e8-bbb0-e79fca18f3bc

15b1
<DescribeLoadBalancersResponse xmlns="http://elasticloadbalancing.amazonaws.com/doc/2012-06-01/">
  <DescribeLoadBalancersResult>
    <LoadBalancerDescriptions>
      <member>
        <CanonicalHostedZoneNameID>Z215JYRZR1TBD5</CanonicalHostedZoneNameID>
        <VPCId>vpc-c913e2a2</VPCId>
        <Scheme>internal</Scheme>
        <Instances>
          <member>
            <InstanceId>i-0684f04b68548f888</InstanceId>
          </member>
          <member>
            <InstanceId>i-0ef35af86c7718b55</InstanceId>
          </member>
          <member>
            <InstanceId>i-0da1a7304ffad71fc</InstanceId>
          </member>
          <member>
            <InstanceId>i-0ba03b9a2159c3204</InstanceId>
          </member>
          <member>
            <InstanceId>i-0341ad76d2638d2c5</InstanceId>
          </member>
          <member>
            <InstanceId>i-0b16e11b4e52a18a1</InstanceId>
          </member>
        </Instances>
        <Policies>
          <AppCookieStickinessPolicies/>
          <LBCookieStickinessPolicies/>
          <OtherPolicies/>
        </Policies>
        <AvailabilityZones>
          <member>eu-central-1a</member>
          <member>eu-central-1b</member>
        </AvailabilityZones>
        <DNSName>internal-DockerBac-Internal-AYCKM81LV1SW-1848568591.eu-central-1.elb.amazonaws.com</DNSName>
        <BackendServerDescriptions/>
        <SourceSecurityGroup>
          <OwnerAlias>295620090465</OwnerAlias>
          <GroupName>DockerBackend-InternalLoadBalancerSG-HLRQQBXLTCDD</GroupName>
        </SourceSecurityGroup>
        <LoadBalancerName>DockerBac-Internal-AYCKM81LV1SW</LoadBalancerName>
        <ListenerDescriptions>
          <member>
            <Listener>
              <InstancePort>8084</InstancePort>
              <InstanceProtocol>TCP</InstanceProtocol>
              <Protocol>TCP</Protocol>
              <LoadBalancerPort>8084</LoadBalancerPort>
            </Listener>
            <PolicyNames/>
          </member>
          <member>
            <Listener>
              <InstancePort>8085</InstancePort>
              <InstanceProtocol>TCP</InstanceProtocol>
              <Protocol>TCP</Protocol>
              <LoadBalancerPort>8085</LoadBalancerPort>
            </Listener>
            <PolicyNames/>
          </member>
          <member>
            <Listener>
              <InstancePort>8086</InstancePort>
              <InstanceProtocol>TCP</InstanceProtocol>
              <Protocol>TCP</Protocol>
              <LoadBalancerPort>8086</LoadBalancerPort>
            </Listener>
            <PolicyNames/>
          </member>
          <member>
            <Listener>
              <InstancePort>8083</InstancePort>
              <InstanceProtocol>TCP</InstanceProtocol>
              <Protocol>TCP</Protocol>
              <LoadBalancerPort>8083</LoadBalancerPort>
            </Listener>
            <PolicyNames/>
          </member>
          <member>
            <Listener>
              <InstancePort>80</InstancePort>
              <InstanceProtocol>TCP</InstanceProtocol>
              <Protocol>TCP</Protocol>
              <LoadBalancerPort>80</LoadBalancerPort>
            </Listener>
            <PolicyNames/>
          </member>
          <member>
            <Listener>
              <InstancePort>8080</InstancePort>
              <InstanceProtocol>TCP</InstanceProtocol>
              <Protocol>TCP</Protocol>
              <LoadBalancerPort>8080</LoadBalancerPort>
            </Listener>
            <PolicyNames/>
          </member>
          <member>
            <Listener>
              <InstancePort>8081</InstancePort>
              <InstanceProtocol>TCP</InstanceProtocol>
              <Protocol>TCP</Protocol>
              <LoadBalancerPort>8081</LoadBalancerPort>
            </Listener>
            <PolicyNames/>
          </member>
          <member>
            <Listener>
              <InstancePort>11111</InstancePort>
              <InstanceProtocol>TCP</InstanceProtocol>
              <Protocol>TCP</Protocol>
              <LoadBalancerPort>11111</LoadBalancerPort>
            </Listener>
            <PolicyNames/>
          </member>
          <member>
            <Listener>
              <InstancePort>7</InstancePort>
              <InstanceProtocol>TCP</InstanceProtocol>
              <Protocol>TCP</Protocol>
              <LoadBalancerPort>7</LoadBalancerPort>
            </Listener>
            <PolicyNames/>
          </member>
          <member>
            <Listener>
              <InstancePort>8082</InstancePort>
              <InstanceProtocol>TCP</InstanceProtocol>
              <Protocol>TCP</Protocol>
              <LoadBalancerPort>8082</LoadBalancerPort>
            </Listener>
            <PolicyNames/>
          </member>
        </ListenerDescriptions>
        <HealthCheck>
          <UnhealthyThreshold>4</UnhealthyThreshold>
          <Interval>10</Interval>
          <HealthyThreshold>2</HealthyThreshold>
          <Timeout>8</Timeout>
          <Target>HTTP:44554/</Target>
        </HealthCheck>
        <CreatedTime>2017-12-12T00:32:55.650Z</CreatedTime>
        <SecurityGroups>
          <member>sg-2d2c5347</member>
        </SecurityGroups>
        <Subnets>
          <member>subnet-16728c6b</member>
          <member>subnet-4872f723</member>
        </Subnets>
      </member>
    </LoadBalancerDescriptions>
  </DescribeLoadBalancersResult>
  <ResponseMetadata>
    <RequestId>a6e0d05e-1be2-11e8-bbb0-e79fca18f3bc</RequestId>
  </ResponseMetadata>
</DescribeLoadBalancersResponse>

0

-----------------------------------------------------
time="2018-02-27T17:21:31Z" level=info msg="Listeners to sync with ELB: [service(ordami-website_ordami-website):80 ==> tcp://:80 service(nominatim_nominatim):8081 ==> tcp://:8081 service(ordami-backoffice-staging_ordami-backoffice):8084 ==> tcp://:8084 service(ordami-wko-api-staging_ordami-wko-api):8085 ==> tcp://:8085 service(ordami-backend-staging_ordami-backend):8080 ==> tcp://:8080 service(graphhopper_graph-hopper):11111 ==> tcp://:11111 service(ordami-wko-sync-staging_ordami-wko-sync):8086 ==> tcp://:8086 service(visualizer_visualizer):8083 ==> tcp://:8083]"
time="2018-02-27T17:21:31Z" level=info msg="cert is nil, or port  80  Is NOT in  [{443 SSL}]"
time="2018-02-27T17:21:31Z" level=info msg="cert is nil, or port  80  Is NOT in  [{443 SSL}]"
time="2018-02-27T17:21:31Z" level=info msg="cert is nil, or port  8081  Is NOT in  [{443 SSL}]"
time="2018-02-27T17:21:31Z" level=info msg="cert is nil, or port  8081  Is NOT in  [{443 SSL}]"
time="2018-02-27T17:21:31Z" level=info msg="cert is nil, or port  8084  Is NOT in  [{443 SSL}]"
time="2018-02-27T17:21:31Z" level=info msg="cert is nil, or port  8084  Is NOT in  [{443 SSL}]"
time="2018-02-27T17:21:31Z" level=info msg="cert is nil, or port  8085  Is NOT in  [{443 SSL}]"
time="2018-02-27T17:21:31Z" level=info msg="cert is nil, or port  8085  Is NOT in  [{443 SSL}]"
time="2018-02-27T17:21:31Z" level=info msg="cert is nil, or port  8080  Is NOT in  [{443 SSL}]"
time="2018-02-27T17:21:31Z" level=info msg="cert is nil, or port  8080  Is NOT in  [{443 SSL}]"
time="2018-02-27T17:21:31Z" level=info msg="cert is nil, or port  11111  Is NOT in  [{443 SSL}]"
time="2018-02-27T17:21:31Z" level=info msg="cert is nil, or port  11111  Is NOT in  [{443 SSL}]"
time="2018-02-27T17:21:31Z" level=info msg="cert is nil, or port  8086  Is NOT in  [{443 SSL}]"
time="2018-02-27T17:21:31Z" level=info msg="cert is nil, or port  8086  Is NOT in  [{443 SSL}]"
time="2018-02-27T17:21:31Z" level=info msg="cert is nil, or port  8083  Is NOT in  [{443 SSL}]"
time="2018-02-27T17:21:31Z" level=info msg="cert is nil, or port  8083  Is NOT in  [{443 SSL}]"
time="2018-02-27T17:21:31Z" level=info msg="keeping protocol= TCP port= 8084 instancePort= 8084"
time="2018-02-27T17:21:31Z" level=info msg="keeping protocol= TCP port= 8085 instancePort= 8085"
time="2018-02-27T17:21:31Z" level=info msg="keeping protocol= TCP port= 8086 instancePort= 8086"
time="2018-02-27T17:21:31Z" level=info msg="keeping protocol= TCP port= 8083 instancePort= 8083"
time="2018-02-27T17:21:31Z" level=info msg="keeping protocol= TCP port= 80 instancePort= 80"
time="2018-02-27T17:21:31Z" level=info msg="keeping protocol= TCP port= 8080 instancePort= 8080"
time="2018-02-27T17:21:31Z" level=info msg="keeping protocol= TCP port= 8081 instancePort= 8081"
time="2018-02-27T17:21:31Z" level=info msg="keeping protocol= TCP port= 11111 instancePort= 11111"
time="2018-02-27T17:21:31Z" level=info msg="keeping protocol= TCP port= 7 instancePort= 7"
time="2018-02-27T17:21:31Z" level=info msg="listeners to create: []"
time="2018-02-27T17:21:31Z" level=info msg="listeners to change: []"
time="2018-02-27T17:21:31Z" level=info msg="listeners to remove: [service(delete):8082 ==> tcp://:8082]"
time="2018-02-27T17:21:31Z" level=info msg="REMOVE on DockerBac-Internal-AYCKM81LV1SW listener service(delete):8082 ==> tcp://:8082"
2018/02/27 17:21:31 DEBUG: Request elasticloadbalancing/DeleteLoadBalancerListeners Details:
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: elasticloadbalancing.eu-central-1.amazonaws.com
User-Agent: aws-sdk-go/1.4.3 (go1.7.6; linux; amd64)
Content-Length: 134
Authorization: AWS4-HMAC-SHA256 Credential=ASIAIUBVHMMZSCPOV6YQ/20180227/eu-central-1/elasticloadbalancing/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date;x-amz-security-token, Signature=c5582975a8c52987d7227fc14ed51fa97bba46780933fe54744e57fe1084144d
Content-Type: application/x-www-form-urlencoded; charset=utf-8
X-Amz-Date: 20180227T172131Z
X-Amz-Security-Token: FQoDYXdzEML//////////wEaDOdLjWgGDpcsIL5KjiK9A+QaLwF23DgYn1dDX0n3Ie6Z7YpY3Hk7jjxU1ejxstA0NsU4Kwn8VEAxEFBRNxfcp3kaKMz0VmVz9WII4/gm+yLtfgC3bIgpHF26fDGUx41U+Ru+IfZerFZGNq2aQvriZIVWIhEQzDNHfRx0CAeMK0J7uY5GSk4R4hAaW85Wxb9gElQUdfG4r84ZA1VKo/zDlrvxUHpA0Kugwknad6ZR0X8IRINPq4VWzpVwWKmDQ5WYJUnz1cFZf0G2QtC5M0mglC0ZOiqwD6mVva7Xlild8+FWMb/CH/Q4PWvSQqOdV4r6SYyaGGD0wddonv20auGJEsKCLiiCbZPhFOVesKUfKsRiccH2CtqrXlwmOzv/OJUWwhlwog5nz2lDsojjkyFMniUhGWYZFfrUfrWl05W+wFUX8BJw8tdbB0rG247MYo7dN6IfGvWZMUFZYY++4vhMNolB07Hi74vn5g7vEIo4QprDVGe0S1tU0BSt0SSDYCTXrcyiwz7QlLLjdcncV06hPiAm+1kZCQA/43CQB9f/rkG5wL9KmtlzjHSP87DIMPBpKqvNsvCbLQ/rsZpL2gZqstCTfrLVB1RZm3vRV5ooz5DW1AU=
Accept-Encoding: gzip

Action=DeleteLoadBalancerListeners&LoadBalancerName=DockerBac-Internal-AYCKM81LV1SW&LoadBalancerPorts.member.1=8082&Version=2012-06-01
-----------------------------------------------------
2018/02/27 17:21:31 DEBUG: Response elasticloadbalancing/DeleteLoadBalancerListeners Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Content-Length: 289
Content-Type: text/xml
Date: Tue, 27 Feb 2018 17:21:30 GMT
X-Amzn-Requestid: a6e711ef-1be2-11e8-bbb0-e79fca18f3bc

<DeleteLoadBalancerListenersResponse xmlns="http://elasticloadbalancing.amazonaws.com/doc/2012-06-01/">
  <DeleteLoadBalancerListenersResult/>
  <ResponseMetadata>
    <RequestId>a6e711ef-1be2-11e8-bbb0-e79fca18f3bc</RequestId>
  </ResponseMetadata>
</DeleteLoadBalancerListenersResponse>

-----------------------------------------------------
time="2018-02-27T17:21:31Z" level=info msg="REMOVED on DockerBac-Internal-AYCKM81LV1SW listener service(delete):8082 ==> tcp://:8082"

Manager B:

time="2018-02-27T17:21:32Z" level=info msg="8 matches found. Processing."
time="2018-02-27T17:21:32Z" level=info msg="exposedPorts:  map[80:{ tcp 80 80 ingress}]"
time="2018-02-27T17:21:32Z" level=info msg="requestedPublishPorts:  map[80:[80]]"
time="2018-02-27T17:21:32Z" level=info msg="exposed:  { tcp 80 80 ingress}"
time="2018-02-27T17:21:32Z" level=info msg="Cert:  <nil>"
time="2018-02-27T17:21:32Z" level=info msg="urlString:  tcp://:80"
time="2018-02-27T17:21:32Z" level=info msg="exposedPorts:  map[8081:{ tcp 8080 8081 ingress}]"
time="2018-02-27T17:21:32Z" level=info msg="requestedPublishPorts:  map[8080:[8081]]"
time="2018-02-27T17:21:32Z" level=info msg="exposed:  { tcp 8080 8081 ingress}"
time="2018-02-27T17:21:32Z" level=info msg="Cert:  <nil>"
time="2018-02-27T17:21:32Z" level=info msg="urlString:  tcp://:8081"
time="2018-02-27T17:21:32Z" level=info msg="exposedPorts:  map[8084:{ tcp 80 8084 ingress}]"
time="2018-02-27T17:21:32Z" level=info msg="requestedPublishPorts:  map[80:[8084]]"
time="2018-02-27T17:21:32Z" level=info msg="exposed:  { tcp 80 8084 ingress}"
time="2018-02-27T17:21:32Z" level=info msg="Cert:  <nil>"
time="2018-02-27T17:21:32Z" level=info msg="urlString:  tcp://:8084"
time="2018-02-27T17:21:32Z" level=info msg="exposedPorts:  map[8085:{ tcp 8080 8085 ingress}]"
time="2018-02-27T17:21:32Z" level=info msg="requestedPublishPorts:  map[8080:[8085]]"
time="2018-02-27T17:21:32Z" level=info msg="exposed:  { tcp 8080 8085 ingress}"
time="2018-02-27T17:21:32Z" level=info msg="Cert:  <nil>"
time="2018-02-27T17:21:32Z" level=info msg="urlString:  tcp://:8085"
time="2018-02-27T17:21:32Z" level=info msg="exposedPorts:  map[8080:{ tcp 8080 8080 ingress}]"
time="2018-02-27T17:21:32Z" level=info msg="requestedPublishPorts:  map[8080:[8080]]"
time="2018-02-27T17:21:32Z" level=info msg="exposed:  { tcp 8080 8080 ingress}"
time="2018-02-27T17:21:32Z" level=info msg="Cert:  <nil>"
time="2018-02-27T17:21:32Z" level=info msg="urlString:  tcp://:8080"
time="2018-02-27T17:21:32Z" level=info msg="exposedPorts:  map[11111:{ tcp 11111 11111 ingress}]"
time="2018-02-27T17:21:32Z" level=info msg="requestedPublishPorts:  map[11111:[11111]]"
time="2018-02-27T17:21:32Z" level=info msg="exposed:  { tcp 11111 11111 ingress}"
time="2018-02-27T17:21:32Z" level=info msg="Cert:  <nil>"
time="2018-02-27T17:21:32Z" level=info msg="urlString:  tcp://:11111"
time="2018-02-27T17:21:32Z" level=info msg="exposedPorts:  map[8086:{ tcp 8080 8086 ingress}]"
time="2018-02-27T17:21:32Z" level=info msg="requestedPublishPorts:  map[8080:[8086]]"
time="2018-02-27T17:21:32Z" level=info msg="exposed:  { tcp 8080 8086 ingress}"
time="2018-02-27T17:21:32Z" level=info msg="Cert:  <nil>"
time="2018-02-27T17:21:32Z" level=info msg="urlString:  tcp://:8086"
time="2018-02-27T17:21:32Z" level=info msg="exposedPorts:  map[8083:{ tcp 8080 8083 ingress}]"
time="2018-02-27T17:21:32Z" level=info msg="requestedPublishPorts:  map[8080:[8083]]"
time="2018-02-27T17:21:32Z" level=info msg="exposed:  { tcp 8080 8083 ingress}"
time="2018-02-27T17:21:32Z" level=info msg="Cert:  <nil>"
time="2018-02-27T17:21:32Z" level=info msg="urlString:  tcp://:8083"
time="2018-02-27T17:21:32Z" level=info msg="Read config: # hostname : ELB_name\n127.0.0.1: DockerBac-Internal-AYCKM81LV1SW\nlocalhost: DockerBac-Internal-AYCKM81LV1SW\ndefault: DockerBac-Internal-AYCKM81LV1SW\n# hostname : ELB_name\n127.0.0.1: DockerBac-Internal-AYCKM81LV1SW\nlocalhost: DockerBac-Internal-AYCKM81LV1SW\ndefault: DockerBac-Internal-AYCKM81LV1SW\n"
time="2018-02-27T17:21:32Z" level=info msg="ELB mapping: map[default:DockerBac-Internal-AYCKM81LV1SW 127.0.0.1:DockerBac-Internal-AYCKM81LV1SW localhost:DockerBac-Internal-AYCKM81LV1SW]"
time="2018-02-27T17:21:32Z" level=info msg="ELB Client in region eu-central-1"
time="2018-02-27T17:21:32Z" level=info msg="Located external load balancer DockerBac-Internal-AYCKM81LV1SW for 127.0.0.1"
time="2018-02-27T17:21:32Z" level=info msg="Located external load balancer DockerBac-Internal-AYCKM81LV1SW for localhost"
time="2018-02-27T17:21:32Z" level=info msg="Located external load balancer DockerBac-Internal-AYCKM81LV1SW for default"
time="2018-02-27T17:21:32Z" level=info msg="Configuring DockerBac-Internal-AYCKM81LV1SW"
2018/02/27 17:21:32 DEBUG: Request elasticloadbalancing/DescribeLoadBalancers Details:
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: elasticloadbalancing.eu-central-1.amazonaws.com
User-Agent: aws-sdk-go/1.4.3 (go1.7.6; linux; amd64)
Content-Length: 106
Authorization: AWS4-HMAC-SHA256 Credential=ASIAIF6XHGIVWMKWFZQQ/20180227/eu-central-1/elasticloadbalancing/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date;x-amz-security-token, Signature=c63a57f275cbdcbd319b9c858d689aadceea58b911f3e2085165ec376a6af395
Content-Type: application/x-www-form-urlencoded; charset=utf-8
X-Amz-Date: 20180227T172132Z
X-Amz-Security-Token: FQoDYXdzEML//////////wEaDAHYvpHJgXFTU/mpMiK9A2YUU8zonGoI57qvsfsg5kNViuGSt/gSiBzZlxa8tIeFrVTNH+bFaaDc5qk5Q7JLmGISUrPqMXRlrs69fKXimbk5VGQ6IZPF0ic1YP5ahcpCfsrnipLYh7UZPUEUHq6hi0/uNEaJcv07Znn0ySy1yGF1tEuCfh6f1ECf5pAY33clI/GmrBPlut/FVEDFhdvirJylVuhJau/HTfb74/5/5STLU5tTY4I2+/VQjcLrVw0TPPpeap3lRP7afkDeIZv8jA9F5pKCwenDjykzQ7YpO1uRTgQO1GQWWCORibk1598jtbDyE4yduRpiEcmznngcnR6R5567DlQybFW8Oluv54JUQLkcv4DAiPlBarY78QpfbLEbhxSi6V0J3LYIurB64pwrB6Ew/CjcEYYtGUXgCCDj5fC2gN4J6dUGasT6MGmsC/O2TLsqpR4AN3HVKYzrN9PnOeNDpsuj+7+39anKdcHW6TTnjLQTqb+tuwFX08YXUOLEz2ToNuRp8HbKpfsKFtw3nciPf//Zv25p9YJ5JDNWHsfy6VQZ5h9tIn5esYUTNL4ihsAYaRgJpbnY07rEv1Zq0e4zK6ck4mrzxx8ogqLW1AU=
Accept-Encoding: gzip

Action=DescribeLoadBalancers&LoadBalancerNames.member.1=DockerBac-Internal-AYCKM81LV1SW&Version=2012-06-01
-----------------------------------------------------
2018/02/27 17:21:32 DEBUG: Response elasticloadbalancing/DescribeLoadBalancers Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: text/xml
Date: Tue, 27 Feb 2018 17:21:31 GMT
Vary: Accept-Encoding
X-Amzn-Requestid: a7b5ac38-1be2-11e8-8727-8da6750745df

147a
<DescribeLoadBalancersResponse xmlns="http://elasticloadbalancing.amazonaws.com/doc/2012-06-01/">
  <DescribeLoadBalancersResult>
    <LoadBalancerDescriptions>
      <member>
        <VPCId>vpc-c913e2a2</VPCId>
        <CanonicalHostedZoneNameID>Z215JYRZR1TBD5</CanonicalHostedZoneNameID>
        <Scheme>internal</Scheme>
        <Instances>
          <member>
            <InstanceId>i-0684f04b68548f888</InstanceId>
          </member>
          <member>
            <InstanceId>i-0ef35af86c7718b55</InstanceId>
          </member>
          <member>
            <InstanceId>i-0da1a7304ffad71fc</InstanceId>
          </member>
          <member>
            <InstanceId>i-0ba03b9a2159c3204</InstanceId>
          </member>
          <member>
            <InstanceId>i-0341ad76d2638d2c5</InstanceId>
          </member>
          <member>
            <InstanceId>i-0b16e11b4e52a18a1</InstanceId>
          </member>
        </Instances>
        <Policies>
          <AppCookieStickinessPolicies/>
          <LBCookieStickinessPolicies/>
          <OtherPolicies/>
        </Policies>
        <AvailabilityZones>
          <member>eu-central-1a</member>
          <member>eu-central-1b</member>
        </AvailabilityZones>
        <DNSName>internal-DockerBac-Internal-AYCKM81LV1SW-1848568591.eu-central-1.elb.amazonaws.com</DNSName>
        <BackendServerDescriptions/>
        <SourceSecurityGroup>
          <OwnerAlias>295620090465</OwnerAlias>
          <GroupName>DockerBackend-InternalLoadBalancerSG-HLRQQBXLTCDD</GroupName>
        </SourceSecurityGroup>
        <LoadBalancerName>DockerBac-Internal-AYCKM81LV1SW</LoadBalancerName>
        <ListenerDescriptions>
          <member>
            <Listener>
              <InstancePort>8084</InstancePort>
              <InstanceProtocol>TCP</InstanceProtocol>
              <Protocol>TCP</Protocol>
              <LoadBalancerPort>8084</LoadBalancerPort>
            </Listener>
            <PolicyNames/>
          </member>
          <member>
            <Listener>
              <InstancePort>8085</InstancePort>
              <InstanceProtocol>TCP</InstanceProtocol>
              <Protocol>TCP</Protocol>
              <LoadBalancerPort>8085</LoadBalancerPort>
            </Listener>
            <PolicyNames/>
          </member>
          <member>
            <Listener>
              <InstancePort>8086</InstancePort>
              <InstanceProtocol>TCP</InstanceProtocol>
              <Protocol>TCP</Protocol>
              <LoadBalancerPort>8086</LoadBalancerPort>
            </Listener>
            <PolicyNames/>
          </member>
          <member>
            <Listener>
              <InstancePort>8083</InstancePort>
              <InstanceProtocol>TCP</InstanceProtocol>
              <Protocol>TCP</Protocol>
              <LoadBalancerPort>8083</LoadBalancerPort>
            </Listener>
            <PolicyNames/>
          </member>
          <member>
            <Listener>
              <InstancePort>80</InstancePort>
              <InstanceProtocol>TCP</InstanceProtocol>
              <Protocol>TCP</Protocol>
              <LoadBalancerPort>80</LoadBalancerPort>
            </Listener>
            <PolicyNames/>
          </member>
          <member>
            <Listener>
              <InstancePort>8080</InstancePort>
              <InstanceProtocol>TCP</InstanceProtocol>
              <Protocol>TCP</Protocol>
              <LoadBalancerPort>8080</LoadBalancerPort>
            </Listener>
            <PolicyNames/>
          </member>
          <member>
            <Listener>
              <InstancePort>8081</InstancePort>
              <InstanceProtocol>TCP</InstanceProtocol>
              <Protocol>TCP</Protocol>
              <LoadBalancerPort>8081</LoadBalancerPort>
            </Listener>
            <PolicyNames/>
          </member>
          <member>
            <Listener>
              <InstancePort>11111</InstancePort>
              <InstanceProtocol>TCP</InstanceProtocol>
              <Protocol>TCP</Protocol>
              <LoadBalancerPort>11111</LoadBalancerPort>
            </Listener>
            <PolicyNames/>
          </member>
          <member>
            <Listener>
              <InstancePort>7</InstancePort>
              <InstanceProtocol>TCP</InstanceProtocol>
              <Protocol>TCP</Protocol>
              <LoadBalancerPort>7</LoadBalancerPort>
            </Listener>
            <PolicyNames/>
          </member>
        </ListenerDescriptions>
        <HealthCheck>
          <UnhealthyThreshold>4</UnhealthyThreshold>
          <Interval>10</Interval>
          <HealthyThreshold>2</HealthyThreshold>
          <Timeout>8</Timeout>
          <Target>HTTP:44554/</Target>
        </HealthCheck>
        <CreatedTime>2017-12-12T00:32:55.650Z</CreatedTime>
        <SecurityGroups>
          <member>sg-2d2c5347</member>
        </SecurityGroups>
        <Subnets>
          <member>subnet-16728c6b</member>
          <member>subnet-4872f723</member>
        </Subnets>
      </member>
    </LoadBalancerDescriptions>
  </DescribeLoadBalancersResult>
  <ResponseMetadata>
    <RequestId>a7b5ac38-1be2-11e8-8727-8da6750745df</RequestId>
  </ResponseMetadata>
</DescribeLoadBalancersResponse>

0

-----------------------------------------------------
time="2018-02-27T17:21:32Z" level=info msg="Listeners to sync with ELB: [service(ordami-website_ordami-website):80 ==> tcp://:80 service(nominatim_nominatim):8081 ==> tcp://:8081 service(ordami-backoffice-staging_ordami-backoffice):8084 ==> tcp://:8084 service(ordami-wko-api-staging_ordami-wko-api):8085 ==> tcp://:8085 service(ordami-backend-staging_ordami-backend):8080 ==> tcp://:8080 service(graphhopper_graph-hopper):11111 ==> tcp://:11111 service(ordami-wko-sync-staging_ordami-wko-sync):8086 ==> tcp://:8086 service(visualizer_visualizer):8083 ==> tcp://:8083]"
time="2018-02-27T17:21:32Z" level=info msg="cert is nil, or port  80  Is NOT in  [{443 SSL}]"
time="2018-02-27T17:21:32Z" level=info msg="cert is nil, or port  80  Is NOT in  [{443 SSL}]"
time="2018-02-27T17:21:32Z" level=info msg="cert is nil, or port  8081  Is NOT in  [{443 SSL}]"
time="2018-02-27T17:21:32Z" level=info msg="cert is nil, or port  8081  Is NOT in  [{443 SSL}]"
time="2018-02-27T17:21:32Z" level=info msg="cert is nil, or port  8084  Is NOT in  [{443 SSL}]"
time="2018-02-27T17:21:32Z" level=info msg="cert is nil, or port  8084  Is NOT in  [{443 SSL}]"
time="2018-02-27T17:21:32Z" level=info msg="cert is nil, or port  8085  Is NOT in  [{443 SSL}]"
time="2018-02-27T17:21:32Z" level=info msg="cert is nil, or port  8085  Is NOT in  [{443 SSL}]"
time="2018-02-27T17:21:32Z" level=info msg="cert is nil, or port  8080  Is NOT in  [{443 SSL}]"
time="2018-02-27T17:21:32Z" level=info msg="cert is nil, or port  8080  Is NOT in  [{443 SSL}]"
time="2018-02-27T17:21:32Z" level=info msg="cert is nil, or port  11111  Is NOT in  [{443 SSL}]"
time="2018-02-27T17:21:32Z" level=info msg="cert is nil, or port  11111  Is NOT in  [{443 SSL}]"
time="2018-02-27T17:21:32Z" level=info msg="cert is nil, or port  8086  Is NOT in  [{443 SSL}]"
time="2018-02-27T17:21:32Z" level=info msg="cert is nil, or port  8086  Is NOT in  [{443 SSL}]"
time="2018-02-27T17:21:32Z" level=info msg="cert is nil, or port  8083  Is NOT in  [{443 SSL}]"
time="2018-02-27T17:21:32Z" level=info msg="cert is nil, or port  8083  Is NOT in  [{443 SSL}]"
time="2018-02-27T17:21:32Z" level=info msg="keeping protocol= TCP port= 8084 instancePort= 8084"
time="2018-02-27T17:21:32Z" level=info msg="keeping protocol= TCP port= 8085 instancePort= 8085"
time="2018-02-27T17:21:32Z" level=info msg="keeping protocol= TCP port= 8086 instancePort= 8086"
time="2018-02-27T17:21:32Z" level=info msg="keeping protocol= TCP port= 8083 instancePort= 8083"
time="2018-02-27T17:21:32Z" level=info msg="keeping protocol= TCP port= 80 instancePort= 80"
time="2018-02-27T17:21:32Z" level=info msg="keeping protocol= TCP port= 8080 instancePort= 8080"
time="2018-02-27T17:21:32Z" level=info msg="keeping protocol= TCP port= 8081 instancePort= 8081"
time="2018-02-27T17:21:32Z" level=info msg="keeping protocol= TCP port= 11111 instancePort= 11111"
time="2018-02-27T17:21:32Z" level=info msg="keeping protocol= TCP port= 7 instancePort= 7"
time="2018-02-27T17:21:32Z" level=info msg="listeners to create: []"
time="2018-02-27T17:21:32Z" level=info msg="listeners to change: []"
time="2018-02-27T17:21:32Z" level=info msg="listeners to remove: []"

After its removal, I redeployed that stack an everything went back to normal.

It should be noted that the initial service update failed a couple of times on different nodes because there was not enough memory available and the service process was forcefully terminated after receiving a TERM signal followed by a KILL signal each time.

FrenchBen commented 6 years ago

@Mobe91 thanks for the input - I forgot to mention, but next time this happens, it would be great to also capture a support dump via: https://docs.docker.com/docker-for-aws/faqs/#where-do-i-report-problems-or-bugs

Even if not a leader, the l4controller should still log some details about the elb. Only one manager should show the port opening, as it's the manager on which you took action (it's listening via the socket).

Ideally these type of bug end up in https://github.com/docker/infrakit as we use the LB controller as detailed here: https://github.com/docker/infrakit/blob/master/docs/controller/ingress/README.md

Mobe91 commented 6 years ago

Even if not a leader, the l4controller should still log some details about the elb. Only one manager should show the port opening, as it's the manager on which you took action (it's listening via the socket).

@FrenchBen When I redeployed the stack, indeed only one manager showed the port opening.

Here is the docker-diagnose output that I retrieved just now - I did not change anything in the Swarm since the problem happened yesterday (apart from redeploying the respective stack).

OK hostname=ip-172-31-2-175-eu-central-1-compute-internal session=1519796929-ljCwm8qvIZ0e4S0HePojkbwqq9bVWkZx
OK hostname=ip-172-31-17-101-eu-central-1-compute-internal session=1519796929-ljCwm8qvIZ0e4S0HePojkbwqq9bVWkZx
OK hostname=ip-172-31-27-54-eu-central-1-compute-internal session=1519796929-ljCwm8qvIZ0e4S0HePojkbwqq9bVWkZx
OK hostname=ip-172-31-16-68-eu-central-1-compute-internal session=1519796929-ljCwm8qvIZ0e4S0HePojkbwqq9bVWkZx
OK hostname=ip-172-31-20-18-eu-central-1-compute-internal session=1519796929-ljCwm8qvIZ0e4S0HePojkbwqq9bVWkZx
OK hostname=ip-172-31-11-30-eu-central-1-compute-internal session=1519796929-ljCwm8qvIZ0e4S0HePojkbwqq9bVWkZx
Done requesting diagnostics.
Your diagnostics session ID is 1519796929-ljCwm8qvIZ0e4S0HePojkbwqq9bVWkZx
Please provide this session ID to the maintainer debugging your issue.

Ideally these type of bug end up in https://github.com/docker/infrakit as we use the LB controller as detailed here: https://github.com/docker/infrakit/blob/master/docs/controller/ingress/README.md

So should I open a separate issue in infrakit and reference this one?