Expected behavior

Ability to deploy and rm docker stack without losing memory.

Actual behavior

Every iteration of "docker stack deploy --compose-file hello.yml hello" and "docker stack rm hello": all masters lose some memory (<100kb) and all nodes running the container lose around 1-2MB of memory. Eventually a node (usually a master) will run out-of-memory; and the out-of-memory node becomes unresponsive. Sometimes ASG terminates node immediately; sometimes it takes ASG time (over 1 hour) to auto-terminate the node. I've run this test on various Docker configurations: 1 master, 3 masters, 1 master and 1 worker, 3 masters and 3 workers.

Information

Here is my leader of a 3 master only configuration after testing but stopping before using all the memory. I've pruned all docker resources, yet the memory is never freed:

$ free -m
                   total       used       free     shared    buffers     cached
Mem:          3946       3824        121        185         12        201
-/+ buffers/cache:       3611        334
Swap:            0          0          0

Running docker-diagnose killed my leader node, right after execution =(

$  docker-diagnose
OK hostname=ip-172-31-36-46-us-west-2-compute-internal session=1511636693-K6FsbkUOfNCzcGE9wFIVxru763mUTd7i
OK hostname=ip-172-31-0-211-us-west-2-compute-internal session=1511636693-K6FsbkUOfNCzcGE9wFIVxru763mUTd7i
OK hostname=ip-172-31-21-134-us-west-2-compute-internal session=1511636693-K6FsbkUOfNCzcGE9wFIVxru763mUTd7i
Done requesting diagnostics.
Your diagnostics session ID is 1511636693-K6FsbkUOfNCzcGE9wFIVxru763mUTd7i
Please provide this session ID to the maintainer debugging your issue.

A reproducible case if this is a bug, Dockerfiles FTW
Steps to reproduce the behavior

I have reproduced this out-of-memory in many docker configurations of master/worker combinations. I've used micro instances (500MB RAM) up to large instances (8GB RAM). Less instance memory runs into out-of-memory condition faster.
1. Create docker compose file - hello.yml:
```
version: '3.0'
```

services: hellotest: image: hello-world deploy: mode: global


2. Create test shell script - test.sh

!/bin/sh

globals

how many times to loop through test

LOOPS=1000

starttime=date echo start: $starttime

counter=1 while [ $counter -le $LOOPS ] do echo loop number $counter

start docker

    cd /home/docker
    echo starting docker: docker stack deploy --compose-file hello.yml hello
    docker stack deploy --compose-file hello.yml hello
    sleep 60

    # stop docker
    echo stopping docker: docker stack rm hello
    docker stack rm hello
    sleep 60

    counter=$((counter+1))

done

endtime=date echo end: $endtime



3. Run shell script:
nohup ./test.sh &

docker-archive / for-aws

Docker Swarm Resource/Memory leak #124