When using swarm1.0, cannot start container, memory.swappiness: invalid argument

[fsfshijm]

swarm1.0 + docker1.9 + docker-compose1.5

$> docker info
Containers: 7
Images: 17
Role: primary
Strategy: spread
Filters: health, port, dependency, affinity, constraint
Nodes: 2
 EP003: 192.168.0.17:2375
  └ Containers: 5
  └ Reserved CPUs: 0 / 50
  └ Reserved Memory: 0 B / 131.9 GiB
  └ Labels: executiondriver=native-0.2, kernelversion=3.10.0-229.1.2.el7.x86_64, operatingsystem=Red Hat Enterprise Linux Server 7.1 (Maipo), storagedriver=devicemapper
 PEKdev233.dev.fwmrm.net: 192.168.0.233:2375
  └ Containers: 2
  └ Reserved CPUs: 0 / 4
  └ Reserved Memory: 0 B / 16.29 GiB
  └ Labels: executiondriver=native-0.2, kernelversion=3.10.0-123.el7.x86_64, operatingsystem=Red Hat Enterprise Linux Server 7.0 (Maipo), storagedriver=devicemapper
CPUs: 54
Total Memory: 148.2 GiB
Name: c34d6a96258c

$> docker-compose up -d
Starting manual_data_1
ERROR: Cannot start container e677d36e76dfe3486ee4fdd949cd6848d4f416917683384c86df4f7cf253217b: [8] System error: write /sys/fs/cgroup/memory/docker/e677d36e76dfe3486ee4fdd949cd6848d4f416917683384c86df4f7cf253217b/memory.swappiness: invalid argument

$> docker -v
Docker version 1.9.0, build 76d6bc9

$> docker-compose -v
docker-compose version: 1.5.0

$> docker images
REPOSITORY                             TAG                  IMAGE ID            CREATED             VIRTUAL SIZE
swarm                                  latest               6b40fe7724bd        9 days ago          15.6 MB

$> cat /proc/sys/vm/swappiness
10

[fsfshijm]

I can run 'docker-compose up -d' directly against the docker engines, (not swarm) with no problem My OS is RHEL7 Could I set swappiness on swarm?

[jimmyxian]

@fsfshijm Can you provide compose yml file?

[fsfshijm]

@jimmyxian

[fsfshijm]

@jimmyxian $> cat docker-compose.yml

script:
  image: registry.docker.dev.fwmrm.net/mrm-be:redhat7-script
  links:
    - ads
    - hourly
    - oltp
  volumes_from:
    - hourly
  environment:
    - OLTPHOST=oltp
    - ADSHOST=ads
    - ADSLOGHOST=hourly
    - VOLUMEDIR=/work/local_data/script_volume
  command: /usr/sbin/sshd -D

hourly:
  image: registry.docker.dev.fwmrm.net/mrm-be:redhat5-ads-log
  links:
    - ads
    - oltp
  volumes_from:
    - ads
  ports:
    - "54702:44702"
  environment:
    - VOLUMEDIR=/work/local_data/hourly_volume
  command: /usr/sbin/sshd -D

ads:
  image: registry.docker.dev.fwmrm.net/mrm-be:redhat7-ads
  links:
    - oltp
    - data
  volumes:
    - /work
  volumes_from:
    - data
  ports:
    - "44701:44701"
  environment:
    - VOLUMEDIR=/work/local_data/ads_volume
  command: /usr/sbin/sshd -D

oltp:
  image: registry.docker.dev.fwmrm.net/mrm-be:redhat7-oltp
  links:
    - data
  volumes:
    - /home/jmshi/work/be-ci-build/build/..//data/ci:/work/ci
  volumes_from:
    - data
  ports:
    - "3311:3310"
  environment:
    - VOLUMEDIR=/work/local_data/oltp_volume
  command: /usr/sbin/sshd -D

data:
  image: registry.docker.dev.fwmrm.net/mrm-be:redhat7-local-data
  volumes:
    - /home/jmshi/work/be-ci-build/build/..//docker/dockerfiles/local_data/volume:/work/local_data/volume
  command: /usr/sbin/sshd -D

thanks!

[jimmyxian]

@fsfshijm Can you test docker-compose up directly against the docker engine on both of the two machines? From the yml, I see you don't set swappiness. The default swappiness value is -1.

[fsfshijm]

@jimmyxian, I have tested docker-compose up on this two machines, they both are normal; '-1' is correct ? and how to set swappiness in docker-compose.yml?

thanks

[jimmyxian]

Yeah, -1 is correct. I test locally, but can not reproduce. Can you try this and see if container is created successfully?

docker -H <swarm_ip:swarm_port> run -d busybox sleep 100

[fsfshijm]

@jimmyxian Yes, it's successful,

-sh-4.2$ docker run -d busybox sleep 100
1bb7e9be8ea60e44f7980dcc7661d663ecc96d7374f3d167335290e95003c569
-sh-4.2$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED              STATUS              PORTS                          NAMES
1bb7e9be8ea6        busybox             "sleep 100"              6 seconds ago        Up 3 seconds                                       PEKdev233.dev.fwmrm.net/elated_northcutt

I also try this, I run containers separately，it's successfull; but when running them in docker-compose.yml, it's failed.

-sh-4.2$ docker images
REPOSITORY                             TAG                  IMAGE ID            CREATED             VIRTUAL SIZE
mongo                                  latest               db8536bb5482        5 days ago          261.5 MB
swarm                                  latest               6b40fe7724bd        12 days ago         15.6 MB
hello-world                            latest               0a6ba66e537a        4 weeks ago         960 B
bfirsh/compose-mongodb-demo            latest               ebcd20f3983f        4 months ago        684.8 MB
busybox                                latest               c51f86c28340        2 weeks ago         1.109 MB

-sh-4.2$ docker run -d mongo:latest
5e29522f7872146ec8d10e4d2dbede2db297b4068be30d335b12766970d20f5a

-sh-4.2$ docker run -d bfirsh/compose-mongodb-demo:latest
0af7464d4406fd4e3e5eccd61cf2247bab631f2206ee67361078f82fc19b4405

-sh-4.2$ docker ps
CONTAINER ID        IMAGE                                COMMAND                  CREATED             STATUS              PORTS                          NAMES
0af7464d4406        bfirsh/compose-mongodb-demo:latest   "/bin/sh -c 'python a"   3 seconds ago       Up 3 seconds        5000/tcp                       EP003/elegant_kirch
5e29522f7872        mongo:latest                         "/entrypoint.sh mongo"   19 seconds ago      Up 17 seconds       27017/tcp                      PEKdev233.dev.fwmrm.net/drunk_ramanujan
2176723233d9        jplock/zookeeper                     "/opt/zookeeper/bin/z"   25 hours ago        Up 25 hours         2181/tcp, 2888/tcp, 3888/tcp   EP003/grave_feynman

-sh-4.2$ cat docker-compose.yml
web:
    image: bfirsh/compose-mongodb-demo
    container_name: web
    ports:
        - "1234:5000"
mongo:
    image: mongo
    container_name: mongo

-sh-4.2$ docker-compose up -d
Starting web
ERROR: Cannot start container 138713d4d8fce0259615fdda814c9351cb6542d249357a7e51dc442586908854: [8] System error: write /sys/fs/cgroup/memory/docker/138713d4d8fce0259615fdda814c9351cb6542d249357a7e51dc442586908854/memory.swappiness: invalid argument

[AzenYao]

Did this problem has been fixed? I faced this problem too

[jimmyxian]

@AzenYao I'm trying to reproduce, but can not. Can you provide detail information about your environment and reproduce step. Thanks

[AzenYao]

@jimmyxian Are you in Huawei? I'm in Shanghai. Perhaps we can make a phone call or chatting via wechat.... cos it is complicated...

[AzenYao]

@jimmyxian I used Compose 1.5.1, Swarm 1.0.0 and Docker 1.9 in my host which is Centos7 the kernel version is 3.10.0-123.9.3.el7.x86_64

I could start container via compose and docker or swarm and docker, but when I want to combine swarm, compose together, it responded that Cannot start container 2669a535766c5e19c9e58e67fc411218ec1bc8151f5c5a220f985903f45eff36: [8] System error: write /sys/fs/cgroup/memory/system.slice/docker/2669a535766c5e19c9e58e67fc411218ec1bc8151f5c5a220f985903f45eff36/memory.swappiness: invalid argument

[jimmyxian]

@AzenYao Yeah, 342288824 is my wechat number. I will build a Centos7 environment, and try to reproduce.

[fsfshijm]

@jimmyxian, this is my docker daemon:

sudo /usr/local/bin/docker daemon -g /export/docker1.9.0 -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock --insecure-registry registry.docker.dev.fwmrm.net --exec-opt native.cgroupdriver=cgroupfs

and I setup my swarm like this:

#!/bin/bash

#token=d182585a357794f36dce91943a5760a0
token=`docker run --rm swarm create`
echo $token > cluster_id

docker run -d swarm join --addr=192.168.0.233:2375 token://$token
docker run -d swarm join --addr=192.168.0.17:2375 token://$token
docker run -d -p 2380:2375 swarm manage token://$token

I'm not sure if this can give some help for reproducing

thanks

[jimmyxian]

@AzenYao @fsfshijm Done to confirm. it's a bug.

• Compose --> Docker: When create container, compose create container without Memory swappiness, the value is set to 0/null. It should be -1

[root@ubuntu home]# docker inspect a40af8469798 | grep Mem
        "Memory": 0,
        "MemoryReservation": 0,
        "MemorySwap": 0,
        "KernelMemory": 0,
        "CpusetMems": "",
        "MemorySwappiness": null,

• Compose --> Swarm: Same as Compose-->Docker. Swarm should set this default value -1 if no Memory swappiness in request
• Centos: does not support memory swappiness?

[root@host4 home]# docker run --memory-swappiness=0 -d nginx
7df09881125d7e6c06d65b342bcc633946b77813308fbb5cbc62d3ee460d1a15
Error response from daemon: Cannot start container 7df09881125d7e6c06d65b342bcc633946b77813308fbb5cbc62d3ee460d1a15: [8] System error: write /sys/fs/cgroup/memory/system.slice/docker-7df09881125d7e6c06d65b342bcc633946b77813308fbb5cbc62d3ee460d1a15.scope/memory.swappiness: invalid argument
[root@host4 home]# docker run --memory-swappiness=1 -d nginx
a40af8469798517bb4a8f7b8c40f332f4db884c00051135b90d7fe3999043464
Error response from daemon: Cannot start container a40af8469798517bb4a8f7b8c40f332f4db884c00051135b90d7fe3999043464: [8] System error: write /sys/fs/cgroup/memory/system.slice/docker-a40af8469798517bb4a8f7b8c40f332f4db884c00051135b90d7fe3999043464.scope/memory.swappiness: invalid argument

I will fix Compose-->Swarm.

The other two problem, we should open issues in docker project.

/cc @abronan @vieux

[fsfshijm]

Got it, Thanks very much! @jimmyxian

[dnephin]

Compose does not set any value for MemorySwappiness, so it should use the default value of -1. I think something in swarm is translating null to 0.

[abronan]

@dnephin @jimmyxian PTAL: https://github.com/docker/docker/pull/18285

There is a PR up on docker to restore -1 as a default value when not specified. This should make this PR unnecessary on Swarm and Compose (good for us :cocktail:)

[jimmyxian]

@abronan Thanks. :) But I think we should also check this in our swarm. The reason is that in our swarm api, if not have memoryswappiness in request, we will set it 0. I will test as soon as that PR merged and make sure.

[abronan]

@jimmyxian @fsfshijm @AzenYao This should be fixed by now in docker:master :)

[jimmyxian]

@abronan Cool. I will test this and make sure whether we should fix in swarm side or not.

[johnjelinek]

@jimmyxian: the issue is not fixed alone in docker:master. See: https://github.com/docker/docker/issues/17879#issuecomment-161522894

[jimmyxian]

@johnjelinek Thanks for testing this. We should also fix this on swarm side.

[abronan]

Fixed by #1425

[kentoj]

I am unclear on what exactly setting swappiness to -1 does. The current documentation states "A value of 0 turns off anonymous page swapping." Does a value of -1 also do this? If so, I'll submit a documentation PR for the Docker Run and Docker Compose docs.