search

portworx / px-dev

PX-Developer is scale-out storage for containers. Run Cassandra, Jenkins, or any application in Docker, with enterprise storage functionality on commodity servers
270 stars 83 forks source link

ERROR: for portworx Cannot start service portworx: linux mounts: Path /opt/pwx/bin is mounted on / but it is not a shared mount #13

Closed akamalov closed 8 years ago

akamalov commented 8 years ago

Greetings,

Trying to get px-get working. Followed the quick-start guide, however encountered problems.

Environment:

NAME="Red Hat Enterprise Linux Server"
VERSION="7.2 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="7.2"
PRETTY_NAME="Red Hat Enterprise Linux Server 7.2 (Maipo)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.2:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.2
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.2"

Docker info:

Containers: 3
 Running: 2
 Paused: 0
 Stopped: 1
Images: 42
Server Version: 1.12.1
Storage Driver: overlay
 Backing Filesystem: xfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local rexray
 Network: null host bridge overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 3.10.0-327.4.4.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.2 (Maipo)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 11.58 GiB
Name: local1
ID: 2CLA:UCHL:R5J4:IFLP:UOYW:HUVM:DT4M:FUSP:UX6I:SQTB:67PH:5YIN
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: twistlockreader
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
 127.0.0.0/8

File: config.json

{
  "clusterid": "d6a1096b-3ba6-4e1e-b75b-b7f7a8fa6df7"
  "kvdb": [ 
              "etcd:http://127.0.0.1:2379"
         ],
  "storage": {
    "devices": [
      "/dev/sdh"
    ]
  }
}

Error:

[root@node261 quick-start]# docker-compose up -d 
Pulling portworx (portworx/px-dev:latest)...
latest: Pulling from portworx/px-dev
a2392627bec4: Pull complete
6f75e6962b30: Pull complete
f5e1684bcd4b: Pull complete
1af17c678254: Pull complete
b21c7216d97e: Pull complete
ba99d4dc1b96: Pull complete
c93ccce1ede9: Pull complete
0d377a9f3876: Pull complete
4d1e825eea0e: Pull complete
1fb846e844bb: Pull complete
a1df505be760: Pull complete
89407dacbba6: Pull complete
c823797e1d88: Pull complete
Digest: sha256:6c1c58b06e1efe7f198319b998d758874e35db0bf4287ceb5b51a7dfe92792c3
Status: Downloaded newer image for portworx/px-dev:latest
Creating px-dev

ERROR: for portworx  Cannot start service portworx: linux mounts: Path /opt/pwx/bin is mounted on / but it is not a shared mount.
ERROR: Encountered errors while bringing up the project.
[root@node261 quick-start]#

Any ideas why is it bombing out ?

Alex

akamalov commented 8 years ago

Answering my own question, as per Gao's post:

https://groups.google.com/forum/#!topic/portworx/A2vtTjjzSLI

akamalov commented 8 years ago

Verified, flag MountFlags=slave is not present in /usr/lib/systemd/system/docker.service. Still having the same issue.

File: /usr/lib/systemd/system/docker.service

[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network.target

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd
ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
#TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process

[Install]
WantedBy=multi-user.target
#
jsilberm commented 8 years ago

A couple requests: 1) Send output from "service docker status", to verify which docker.service file is being used. 2) Send the output from your docker-compose.yml file

If you want to bring up without docker-compose, you could just run this: docker run --restart=always --name px -d --net=host \ --privileged=true \ -v /run/docker/plugins:/run/docker/plugins \ -v /var/lib/osd:/var/lib/osd:shared \ -v /dev:/dev \ -v /etc/pwx:/etc/pwx \ -v /opt/pwx/bin:/export_bin:shared \ -v /var/run/docker.sock:/var/run/docker.sock \ -v /var/cores:/var/cores \ -v /usr/src:/usr/src \ --ipc=host \ portworx/px-dev

Please keep us posted.

akamalov commented 8 years ago

@jsilberm Thanks for the prompt reply. Here are the outputs:

[root@node261 quick-start]# service docker status
Redirecting to /bin/systemctl status  docker.service
● docker.service - Docker Application Container Engine
   Loaded: loaded (/etc/systemd/system/docker.service; disabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/docker.service.d
           └─docker.conf
   Active: active (running) since Thu 2016-10-20 09:57:57 EDT; 2h 31min ago
     Docs: https://docs.docker.com
  Process: 27967 ExecStartPre=/usr/sbin/ip link del docker0 (code=exited, status=0/SUCCESS)
 Main PID: 27972 (dockerd)
   Memory: 14.5M
   CGroup: /system.slice/docker.service
           ├─27972 dockerd --bip=192.168.98.1/24 --mtu=1450 -H tcp://0.0.0.0:2375 -H unix://var/run/docker.sock --...
           └─27991 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --shim docker-...

Oct 20 09:57:55 node261.local.net docker[27972]: time="2016-10-20T09:57:55.305756065-04:00" level=info...se"
Oct 20 09:57:57 node261.local.net docker[27972]: time="2016-10-20T09:57:57.043137427-04:00" level=info...e."
Oct 20 09:57:57 node261.local.net docker[27972]: time="2016-10-20T09:57:57.043783760-04:00" level=info...on"
Oct 20 09:57:57 node261.local.net docker[27972]: time="2016-10-20T09:57:57.043873354-04:00" level=info...2.1
Oct 20 09:57:57 node261.local.net systemd[1]: Started Docker Application Container Engine.
Oct 20 09:57:57 node261.local.net docker[27972]: time="2016-10-20T09:57:57.097270356-04:00" level=info...ck"
Oct 20 09:57:57 node261.local.net docker[27972]: time="2016-10-20T09:57:57.098234600-04:00" level=info...75"
Oct 20 09:58:03 node261.local.net docker[27972]: time="2016-10-20T09:58:03.705175012-04:00" level=erro...t."
Oct 20 10:04:06 node261.local.net docker[27972]: time="2016-10-20T10:04:06.859827737-04:00" level=erro...t."
Oct 20 10:05:36 node261.local.net docker[27972]: time="2016-10-20T10:05:36.829507324-04:00" level=erro...t."
Hint: Some lines were ellipsized, use -l to show in full.
[root@node261 quick-start]#

File: /usr/lib/systemd/system/docker.service.d:

[Unit]
After=flanneld.service
Requires=flanneld.service

[Service]
EnvironmentFile=-/run/flannel/docker
EnvironmentFile=/run/flannel/subnet.env
EnvironmentFile=/etc/sysconfig/docker
ExecStartPre=-/usr/sbin/ip link del docker0
ExecStart=
ExecStart=/usr/bin/docker daemon \
      --bip=${FLANNEL_SUBNET} \
      --mtu=${FLANNEL_MTU} \
      $OPTIONS \
      $DOCKER_STORAGE_OPTIONS \
      $DOCKER_NETWORK_OPTIONS \
      $INSECURE_REGISTRY

Deploy portworx without docker-compose:

[root@node261 quick-start]# docker run --restart=always --name px -d --net=host \
> --privileged=true \
> -v /run/docker/plugins:/run/docker/plugins \
> -v /var/lib/osd:/var/lib/osd:shared \
> -v /dev:/dev \
> -v /etc/pwx:/etc/pwx \
> -v /opt/pwx/bin:/export_bin:shared \
> -v /var/run/docker.sock:/var/run/docker.sock \
> -v /var/cores:/var/cores \
> -v /usr/src:/usr/src \
> --ipc=host \
> portworx/px-dev
5bcaab836d07d0738b312bc93bede763627ef718f422c521d5ffb95b28e6824e
docker: Error response from daemon: linux mounts: Path /opt/pwx/bin is mounted on / but it is not a shared mount..
[root@node261 quick-start]#

Thanks again.

jsilberm commented 8 years ago

@akamalov : Can you please share this file too? /etc/systemd/system/docker.service

Thanks.

gourao commented 8 years ago

@akamalov Hi Alex, When you run the following, what happens? docker run -v /mnt:/mnt:shared --rm -it busybox /bin/sh

We need shared to be able to deploy volumes into the other containers...

jvinod commented 8 years ago

@akamalov based on your systemd output, the file being loaded is /etc/systemd/system/docker.service

You'd need to remove MountFlags=slave``from there as well, follow it up withsystemct daemon-reload. Thedocker run` suggested by @gourao will help you verify if it took effect..

Generally curious how this machine was setup because with docker 1.12.1+, the systemd files don't have the MountFlags setting. Was it setup through docker machine?

jsilberm commented 8 years ago

Alex --- We also sent you a link to use the px-enterprise product, which removes many of the limitations around px-dev. Please feel free to use that and offer us your feedback. All the documentation is available at docs.portworx.com. Thanks.

akamalov commented 8 years ago

Thanks @jsilberm!

@jvinod if you scroll up,the file I posted /usr/lib/systemd/system/docker.service has no MountFlags=slave flag:

Verified, flag MountFlags=slave is not present in /usr/lib/systemd/system/docker.service. Still having the same issue.

File: /usr/lib/systemd/system/docker.service

[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network.target

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd
ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
#TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process

[Install]
WantedBy=multi-user.target
#

The environment originally was Docker 1.10, which was later upgraded to 1.11 and now 1.12.x

akamalov commented 8 years ago

@jsilberm Tried..getting the same error:

[root@node261 quick-start]# docker run --restart=always --name px-enterprise -d --net=host --privileged=true \
> -v /run/docker/plugins:/run/docker/plugins \
> -v /var/lib/osd:/var/lib/osd:shared \
> -v /dev:/dev \
> -v /etc/pwx:/etc/pwx \
> -v /opt/pwx/bin:/export_bin:shared \
> -v /var/run/docker.sock:/var/run/docker.sock \
> -v /mnt:/mnt:shared \
> -v /var/cores:/var/cores \
> -v /usr/src:/usr/src \
> --ipc=host \
> portworx/px-enterprise -t b1f55a36-96ee-11e6-b4e9-0242ac110003 \
> -s /dev/sdh -m ens192 -d ens192
Unable to find image 'portworx/px-enterprise:latest' locally
latest: Pulling from portworx/px-enterprise
a2392627bec4: Already exists 
6f75e6962b30: Already exists 
f5e1684bcd4b: Already exists 
1af17c678254: Already exists 
b21c7216d97e: Already exists 
ba99d4dc1b96: Already exists 
c93ccce1ede9: Already exists 
0d377a9f3876: Already exists 
4d1e825eea0e: Already exists 
1fb846e844bb: Already exists 
a1df505be760: Already exists 
3e8080920c18: Pull complete 
dfc14f519191: Pull complete 
Digest: sha256:4eb42810b2bd8edb5a1b05ec62af54f128d32a826b9744e1acc1240e7f0a2094
Status: Downloaded newer image for portworx/px-enterprise:latest
d70f1526255b895f6b61d619d11fa17ac70df6ce25f6424f0be5813ae1dd0e0a
docker: Error response from daemon: linux mounts: Path /mnt is mounted on /mnt but it is not a shared mount..
[root@node261 quick-start]#

@gourao the same error:

[root@node261 quick-start]# docker run -v /mnt:/mnt:shared --rm -it busybox /bin/sh
docker: Error response from daemon: linux mounts: Path /mnt is mounted on /mnt but it is not a shared mount..
[root@node261 quick-start]#
gourao commented 8 years ago

@akamalov Alex, this means that Docker is not setup (somehow) to allow shared mounts. docker-machine has an outstanding bug in their deployment... so can you look at /etc/systemd/system/docker.service

That must also have that flag disabled.

After unsetting that, please do a systemct daemon-reload

And restart docker...

akamalov commented 8 years ago

Hey @gourao , that's the whole point that the value is not there:

[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network.target

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd
ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
#TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process

[Install]
WantedBy=multi-user.target

This flag was on as of Docker 1.11.2, but current version running is 1.12.1 and the flag is absent. I tried different flags, including MountFlags=shared , but it didn't yield any results...

Thanks, Alex

gourao commented 8 years ago

@akamalov if we can do a quick screen share session, that would be great... want to see where that flag could be potentially being enforced from.

akamalov commented 8 years ago

Overcame the problem. Here is what I did, as per https://github.com/docker/docker/issues/19625

Compare the namespaces used for the system:

[root@node261 ~]# nsenter --mount=/proc/$(cat /var/run/docker.pid)/ns/mnt findmnt -o TARGET,PROPAGATION
TARGET                                                                                                                        PROPAGATION
/                                                                                                                             private,slave
├─/dev                                                                                                                        private,slave
│ ├─/dev/shm                                                                                                                  private,slave
│ ├─/dev/pts                                                                                                                  private,slave
│ ├─/dev/hugepages                                                                                                            private,slave
│ └─/dev/mqueue                                                                                                               private,slave
├─/proc                                                                                                                       private,slave
│ ├─/proc/sys/fs/binfmt_misc                                                                                                  private,slave
│ │ └─/proc/sys/fs/binfmt_misc                                                                                                private,slave
│ └─/proc/fs/nfsd                                                                                                             private,slave
├─/sys                                                                                                                        private,slave
│ ├─/sys/kernel/security                                                                                                      private,slave
│ ├─/sys/fs/cgroup                                                                                                            private,slave
│ │ ├─/sys/fs/cgroup/systemd                                                                                                  private,slave
│ │ ├─/sys/fs/cgroup/memory                                                                                                   private,slave
│ │ ├─/sys/fs/cgroup/net_cls                                                                                                  private,slave
│ │ ├─/sys/fs/cgroup/cpuset                                                                                                   private,slave
│ │ ├─/sys/fs/cgroup/freezer                                                                                                  private,slave
│ │ ├─/sys/fs/cgroup/cpu,cpuacct                                                                                              private,slave
│ │ ├─/sys/fs/cgroup/hugetlb                                                                                                  private,slave
│ │ ├─/sys/fs/cgroup/perf_event                                                                                               private,slave
│ │ ├─/sys/fs/cgroup/devices                                                                                                  private,slave
│ │ └─/sys/fs/cgroup/blkio                                                                                                    private,slave
│ ├─/sys/fs/pstore                                                                                                            private,slave
│ ├─/sys/kernel/config                                                                                                        private,slave
│ ├─/sys/kernel/debug                                                                                                         private,slave
│ └─/sys/fs/fuse/connections                                                                                                  private,slave
├─/run                                                                                                                        private,slave
│ ├─/run/user/0                                                                                                               private,slave
│ └─/run/docker/netns/default                                                                                                 private,slave
├─/var                                                                                                                        private,slave
│ ├─/var/lib/nfs/rpc_pipefs                                                                                                   private,slave
│ ├─/var/log                                                                                                                  private,slave
│ ├─/var/tmp                                                                                                                  private,slave
│ ├─/var/lib/rexray/volumes/whisper                                                                                           private,slave
│ ├─/var/lib/rexray/volumes/elasticsearch                                                                                     private,slave
│ ├─/var/lib/rexray/volumes/grafana                                                                                           private,slave
│ ├─/var/lib/rexray/volumes/log_graphite                                                                                      private,slave
│ ├─/var/lib/rexray/volumes/log_elasticsearch                                                                                 private,slave
│ └─/var/lib/docker/overlay                                                                                                   private
├─/diskdump                                                                                                                   private,slave
├─/users                                                                                                                      private,slave
├─/usr/bltemp                                                                                                                 private,slave
├─/tmp                                                                                                                        private,slave
│ ├─/tmp/mesos/slaves/bfc2b395-a610-4826-8e5f-280928d609cb-S0/frameworks/a979cde6-aa86-4286-b07f-e83e9ae4076e-0005/executors/d1082b91224121d2_k8sm-executor/runs/b56816e8-6195-4728-9c9a-75357e4c9099/pods/770a8d8e-fa85-11e5-be0c-005056b67cb9/volumes/kubernetes.io~secret/default-token-4hqwm
                                                                                                                              private,slave
│ ├─/tmp/mesos/slaves/bfc2b395-a610-4826-8e5f-280928d609cb-S0/frameworks/a979cde6-aa86-4286-b07f-e83e9ae4076e-0005/executors/d1082b91224121d2_k8sm-executor/runs/b56816e8-6195-4728-9c9a-75357e4c9099/pods/cf24a470-fa9b-11e5-be0c-005056b67cb9/volumes/kubernetes.io~secret/default-token-bgf6e
                                                                                                                              private,slave
│ ├─/tmp/mesos/slaves/bfc2b395-a610-4826-8e5f-280928d609cb-S0/frameworks/a979cde6-aa86-4286-b07f-e83e9ae4076e-0005/executors/d1082b91224121d2_k8sm-executor/runs/b56816e8-6195-4728-9c9a-75357e4c9099/pods/5b9ee67e-fc26-11e5-be0c-005056b67cb9/volumes/kubernetes.io~secret/default-token-4hqwm
                                                                                                                              private,slave
│ └─/tmp/mesos/slaves/a979cde6-aa86-4286-b07f-e83e9ae4076e-S6/frameworks/a979cde6-aa86-4286-b07f-e83e9ae4076e-0005/executors/d1082b91224121d2_k8sm-executor/runs/58d54d92-8792-400d-b16f-6315739510ba/pods/3f2b94ed-1bc3-11e6-be0c-005056b67cb9/volumes/kubernetes.io~secret/default-token-4hqwm
                                                                                                                              private,slave
├─/opt/bsa/bladelogic                                                                                                         private,slave
├─/boot                                                                                                                       private,slave
└─/mnt                                                                                                                        private,slave
[root@node261 ~]

Override the MountFlags value in the default unit with a systemd drop-in directory

mkdir -p /etc/systemd/system/docker.service.d/
cat <<EOF > /etc/systemd/system/docker.service.d/clear_mount_propagtion_flags.conf
[Service]
MountFlags=shared
EOF

Verify drop-in took place:

[root@node261 ~]# cat /etc/systemd/system/docker.service.d/clear_mount_propagtion_flags.conf
[Service]
MountFlags=shared
[root@node261 ~]#

Reload systemd files:

[root@node261 ~]# systemctl daemon-reload

Restart docker:

[root@node261 ~]# systemctl restart docker

Check the status of your mounts in the namespace of the Engine itself (should say shared):

[root@node261 ~]# nsenter --mount=/proc/$(cat /var/run/docker.pid)/ns/mnt findmnt -o TARGET,PROPAGATION
TARGET                                                                                                                          PROPAGATION
/                                                                                                                               shared
├─/dev                                                                                                                          shared
│ ├─/dev/shm                                                                                                                    shared
│ ├─/dev/pts                                                                                                                    shared
│ ├─/dev/hugepages                                                                                                              shared
│ └─/dev/mqueue                                                                                                                 shared
├─/proc                                                                                                                         shared
│ ├─/proc/sys/fs/binfmt_misc                                                                                                    shared
│ │ └─/proc/sys/fs/binfmt_misc                                                                                                  shared
│ └─/proc/fs/nfsd                                                                                                               shared
├─/sys                                                                                                                          shared
│ ├─/sys/kernel/security                                                                                                        shared
│ ├─/sys/fs/cgroup                                                                                                              shared
│ │ ├─/sys/fs/cgroup/systemd                                                                                                    shared
│ │ ├─/sys/fs/cgroup/memory                                                                                                     shared
│ │ ├─/sys/fs/cgroup/net_cls                                                                                                    shared
│ │ ├─/sys/fs/cgroup/cpuset                                                                                                     shared
│ │ ├─/sys/fs/cgroup/freezer                                                                                                    shared
│ │ ├─/sys/fs/cgroup/cpu,cpuacct                                                                                                shared
│ │ ├─/sys/fs/cgroup/hugetlb                                                                                                    shared
│ │ ├─/sys/fs/cgroup/perf_event                                                                                                 shared
│ │ ├─/sys/fs/cgroup/devices                                                                                                    shared
│ │ └─/sys/fs/cgroup/blkio                                                                                                      shared
│ ├─/sys/fs/pstore                                                                                                              shared
│ ├─/sys/kernel/config                                                                                                          shared
│ ├─/sys/kernel/debug                                                                                                           shared
│ └─/sys/fs/fuse/connections                                                                                                    shared
├─/run                                                                                                                          shared
│ └─/run/user/0                                                                                                                 shared
├─/var                                                                                                                          shared
│ ├─/var/lib/nfs/rpc_pipefs                                                                                                     shared
│ ├─/var/log                                                                                                                    shared
│ ├─/var/tmp                                                                                                                    shared
│ ├─/var/lib/rexray/volumes/whisper                                                                                             shared
│ ├─/var/lib/rexray/volumes/elasticsearch                                                                                       shared
│ ├─/var/lib/rexray/volumes/grafana                                                                                             shared
│ ├─/var/lib/rexray/volumes/log_graphite                                                                                        shared
│ ├─/var/lib/rexray/volumes/log_elasticsearch                                                                                   shared
│ └─/var/lib/docker/overlay                                                                                                     private
├─/diskdump                                                                                                                     shared
├─/users                                                                                                                        shared
├─/usr/bltemp                                                                                                                   shared
├─/tmp                                                                                                                          shared
│ ├─/tmp/mesos/slaves/bfc2b395-a610-4826-8e5f-280928d609cb-S0/frameworks/a979cde6-aa86-4286-b07f-e83e9ae4076e-0005/executors/d1082b91224121d2_k8sm-executor/runs/b56816e8-6195-4728-9c9a-75357e4c9099/pods/770a8d8e-fa85-11e5-be0c-005056b67cb9/volumes/kubernetes.io~secret/default-token-4
                                                                                                                                shared
│ ├─/tmp/mesos/slaves/bfc2b395-a610-4826-8e5f-280928d609cb-S0/frameworks/a979cde6-aa86-4286-b07f-e83e9ae4076e-0005/executors/d1082b91224121d2_k8sm-executor/runs/b56816e8-6195-4728-9c9a-75357e4c9099/pods/cf24a470-fa9b-11e5-be0c-005056b67cb9/volumes/kubernetes.io~secret/default-token-b
                                                                                                                                shared
│ ├─/tmp/mesos/slaves/bfc2b395-a610-4826-8e5f-280928d609cb-S0/frameworks/a979cde6-aa86-4286-b07f-e83e9ae4076e-0005/executors/d1082b91224121d2_k8sm-executor/runs/b56816e8-6195-4728-9c9a-75357e4c9099/pods/5b9ee67e-fc26-11e5-be0c-005056b67cb9/volumes/kubernetes.io~secret/default-token-4
                                                                                                                                shared
│ └─/tmp/mesos/slaves/a979cde6-aa86-4286-b07f-e83e9ae4076e-S6/frameworks/a979cde6-aa86-4286-b07f-e83e9ae4076e-0005/executors/d1082b91224121d2_k8sm-executor/runs/58d54d92-8792-400d-b16f-6315739510ba/pods/3f2b94ed-1bc3-11e6-be0c-005056b67cb9/volumes/kubernetes.io~secret/default-token-4
                                                                                                                                shared
├─/opt/bsa/bladelogic                                                                                                           shared
├─/boot                                                                                                                         shared
└─/mnt                                                                                                                          shared
[root@node261 ~]#

Test it:

[root@node261 ~]# docker run -v /mnt:/mnt:shared --rm -it busybox /bin/sh
/ # exit
[root@node261 ~]#

Deploy Portworx:

docker run --restart=always --name px-enterprise -d --net=host --privileged=true -v /run/docker/plugins:/run/docker/plugins -v /var/lib/osd:/var/lib/osd:shared -v /dev:/dev -v /etc/pwx:/etc/pwx -v /opt/pwx/bin:/export_bin:shared -v /var/run/docker.sock:/var/run/docker.sock -v /mnt:/mnt:shared -v /var/cores:/var/cores -v /usr/src:/usr/src --ipc=host portworx/px-enterprise -t b1f55a36-96ee-11e6-b4e9-0242ac110003 -s /dev/sdh -m ens192 -d ens192

[root@node261 ~]# docker run --restart=always --name px-enterprise -d --net=host --privileged=true -v /run/docker/plugins:/run/docker/plugins -v /var/lib/osd:/var/lib/osd:shared -v /dev:/dev -v /etc/pwx:/etc/pwx -v /opt/pwx/bin:/export_bin:shared -v /var/run/docker.sock:/var/run/docker.sock -v /mnt:/mnt:shared -v /var/cores:/var/cores -v /usr/src:/usr/src --ipc=host portworx/px-enterprise -t b1f55a36-96ee-11e6-b4e9-0242ac110003 -s /dev/sdh -m ens192 -d ens192
13d6c5aaed4e1c5a0c57157d72c5a50630ee2da95516bf3f38a4beca9a169163
[root@node261 ~]#

Display running containers:

[root@node261 ~]# docker ps -a
CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS              PORTS               NAMES
13d6c5aaed4e        portworx/px-enterprise   "/docker-entry-point."   12 seconds ago      Up 11 seconds                           px-enterprise

Display Portworx container log:

[root@node261 ~]# docker logs 13d
Tue Oct 25 14:55:12 UTC 2016 : Running on Linux node261.local.net 3.10.0-327.4.4.el7.x86_64 #1 SMP Thu Dec 17 15:51:24 EST 2015 x86_64 x86_64 x86_64 GNU/Linux
checking /usr/src/kernels/3.10.0-327.4.4.el7.x86_64
found /usr/src/kernels/3.10.0-327.4.4.el7.x86_64
make -C /usr/src/kernels/3.10.0-327.4.4.el7.x86_64 M=/home/px-fuse clean
make[1]: Entering directory `/usr/src/kernels/3.10.0-327.4.4.el7.x86_64'
make[1]: Leaving directory `/usr/src/kernels/3.10.0-327.4.4.el7.x86_64'
make -C /usr/src/kernels/3.10.0-327.4.4.el7.x86_64 M=/home/px-fuse modules
make[1]: Entering directory `/usr/src/kernels/3.10.0-327.4.4.el7.x86_64'
  CC [M]  /home/px-fuse/pxd.o
  CC [M]  /home/px-fuse/dev.o
  CC [M]  /home/px-fuse/iov_iter.o
  CC [M]  /home/px-fuse/px_version.o
  LD [M]  /home/px-fuse/px.o
  Building modules, stage 2.
  MODPOST 1 modules
  CC      /home/px-fuse/px.mod.o
  LD [M]  /home/px-fuse/px.ko
make[1]: Leaving directory `/usr/src/kernels/3.10.0-327.4.4.el7.x86_64'
insmod px.ko
insmod: ERROR: could not insert module px.ko: Cannot allocate memory
make: *** [insert] Error 1
Failed to load PX module dependencies
Tue Oct 25 14:55:26 UTC 2016 : Running on Linux node261.local.net 3.10.0-327.4.4.el7.x86_64 #1 SMP Thu Dec 17 15:51:24 EST 2015 x86_64 x86_64 x86_64 GNU/Linux
checking /usr/src/kernels/3.10.0-327.4.4.el7.x86_64
found /usr/src/kernels/3.10.0-327.4.4.el7.x86_64
make -C /usr/src/kernels/3.10.0-327.4.4.el7.x86_64 M=/home/px-fuse clean
[root@node261 ~]#

I am worried about insmod error displayed above, failing to allocate memory. Any tips ?

Thanks,

Alex

jvinod commented 8 years ago

@akamalov glad the MountFlags propagation got resolved - the defaults would soon be 'shared' and you wouldn't have to go through this.

I have actually seen one other instance of "Cannot allocate memory, it is a rare problem - the message is misleading, we have root caused it and will fix. A temporary workaround is to reboot the machine.

Thanks

jsilberm commented 8 years ago

Hi Alex : * /etc/systemd/system/docker.*service \ is the file that is likely the culprit.

Can you please show us the contents?

Thanks, -Jeff

On Thu, Oct 20, 2016 at 12:46 PM, Alex Kamalov notifications@github.com wrote:

Thanks @jsilberm https://github.com/jsilberm!

@jvinod https://github.com/jvinod if you scroll up,the file I posted /usr/lib/systemd/system/docker.service has no MountFlags=slave flag:

Verified, flag MountFlags=slave is not present in /usr/lib/systemd/system/docker.service. Still having the same issue.

File: /usr/lib/systemd/system/docker.service

[Unit] Description=Docker Application Container Engine Documentation=https://docs.docker.com After=network.target

[Service] Type=notify

the default is not to use systemd for cgroups because the delegate issues still

exists and systemd currently does not support the cgroup feature set required

for containers run by docker

ExecStart=/usr/bin/dockerd ExecReload=/bin/kill -s HUP $MAINPID

Having non-zero Limit*s causes performance problems due to accounting overhead

in the kernel. We recommend using cgroups to do container-local accounting.

LimitNOFILE=infinity LimitNPROC=infinity LimitCORE=infinity

Uncomment TasksMax if your systemd version supports it.

Only systemd 226 and above support this version.

TasksMax=infinity

TimeoutStartSec=0

set delegate yes so that systemd does not reset the cgroups of docker containers

Delegate=yes

kill only the docker process, not all processes in the cgroup

KillMode=process

[Install] WantedBy=multi-user.target #

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/portworx/px-dev/issues/13#issuecomment-255208657, or mute the thread https://github.com/notifications/unsubscribe-auth/AFkHUkyLhiUk6-SJjNtlZItJ15wXXZBmks5q18UmgaJpZM4KcJVE .

jsilberm commented 8 years ago

Hi Alex,

We really need to see * /etc/systemd/system/docker.*service

That is very likely the source of the problem.

Once you remote the MOUNT=shared from that file, and do a "system docker restart", then you should be good to go.

Hope this helps, -Jeff

On Thu, Oct 20, 2016 at 1:20 PM, Alex Kamalov notifications@github.com wrote:

@jsilberm https://github.com/jsilberm Tried..getting the same error:

[root@node261 quick-start]# docker run --restart=always --name px-enterprise -d --net=host --privileged=true \

-v /run/docker/plugins:/run/docker/plugins \ -v /var/lib/osd:/var/lib/osd:shared \ -v /dev:/dev \ -v /etc/pwx:/etc/pwx \ -v /opt/pwx/bin:/export_bin:shared \ -v /var/run/docker.sock:/var/run/docker.sock \ -v /mnt:/mnt:shared \ -v /var/cores:/var/cores \ -v /usr/src:/usr/src \ --ipc=host \ portworx/px-enterprise -t b1f55a36-96ee-11e6-b4e9-0242ac110003 \ -s /dev/sdh -m ens192 -d ens192 Unable to find image 'portworx/px-enterprise:latest' locally latest: Pulling from portworx/px-enterprise a2392627bec4: Already exists 6f75e6962b30: Already exists f5e1684bcd4b: Already exists 1af17c678254: Already exists b21c7216d97e: Already exists ba99d4dc1b96: Already exists c93ccce1ede9: Already exists 0d377a9f3876: Already exists 4d1e825eea0e: Already exists 1fb846e844bb: Already exists a1df505be760: Already exists 3e8080920c18: Pull complete dfc14f519191: Pull complete Digest: sha256:4eb42810b2bd8edb5a1b05ec62af54f128d32a826b9744e1acc1240e7f0a2094 Status: Downloaded newer image for portworx/px-enterprise:latest d70f1526255b895f6b61d619d11fa17ac70df6ce25f6424f0be5813ae1dd0e0a docker: Error response from daemon: linux mounts: Path /mnt is mounted on /mnt but it is not a shared mount.. [root@node261 quick-start]#

@gourao https://github.com/gourao the same error:

[root@node261 quick-start]# docker run -v /mnt:/mnt:shared --rm -it busybox /bin/sh docker: Error response from daemon: linux mounts: Path /mnt is mounted on /mnt but it is not a shared mount.. [root@node261 quick-start]#

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/portworx/px-dev/issues/13#issuecomment-255216807, or mute the thread https://github.com/notifications/unsubscribe-auth/AFkHUrBvZHLi7zBnrqDmZ9lRG9TpAsRWks5q180jgaJpZM4KcJVE .

gourao commented 8 years ago

@akamalov Hi Alex, if the reboot worked, can you let us know and/or close this issue?

akamalov commented 8 years ago

Yep, reboot worked. closing the ticket. Thanks Gou!!