Err": "VolumeDriver.Mount: {\"Error\":\"open /dev/xvdf: operation not permitted\"}\n",

[ajeetraina]

I have 2 node Amazon EC2 instance and I ran this container in global mode.

root@ip-172-31-31-235:~# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 738d64301e74 basi/rexray:latest "/bin/sh -c /entrypoi" About an hour ago Up About an hour rexray.0.b2yklecexmgaut3kl147gtkan root@ip-172-31-31-235:~#

I am trying to run Wordpress application under Docker 1.12 which uses MySQL. While I try to:

docker service create --replicas 1 --name wordpressdb1 --network collabnet --mount type=volume,source=vol_collab,target=/var/lib/mysql,volume-driver=rexray --env MYSQL_ROOT_PASSWORD=collab123 --env MYSQL_DATABASE=wordpress mysql:latest

It just starts but throws error:

root@ip-172-31-31-235:~# docker service ls ID NAME REPLICAS IMAGE COMMAND 0b8mhlirjgfe rexray global basi/rexray 0h158az9rnp5 wordpressdb1 0/1 mysql:latest root@ip-172-31-31-235:~# docker service ps wordpressdb1 ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR a54da45pbuxw6ave7ggp3r2tj wordpressdb1.1 mysql:latest ip-172-31-29-88 Ready Rejected 1 seconds ago "VolumeDriver.Mount: {"Error":…" erewtmvzt7b6w6bidhzs4qe5b _ wordpressdb1.1 mysql:latest ip-172-31-31-235 Shutdown Rejected 2 seconds ago "VolumeDriver.Mount: {"Error":…" root@ip-172-31-31-235:~#

root@ip-172-31-31-235:~# docker inspect 1osa13k53mc3p429rwy1aziiz [ { "ID": "1osa13k53mc3p429rwy1aziiz", "Version": { "Index": 711 }, "CreatedAt": "2016-09-13T09:25:16.272351258Z", "UpdatedAt": "2016-09-13T09:25:21.338945122Z", "Spec": { "ContainerSpec": { "Image": "mysql:latest", "Env": [ "MYSQL_ROOT_PASSWORD=collab123", "MYSQL_DATABASE=wordpress" ], "Mounts": [ { "Type": "volume", "Source": "vol_collab", "Target": "/var/lib/mysql", "VolumeOptions": { "DriverConfig": { "Name": "rexray" } } } ] }, "Resources": { "Limits": {}, "Reservations": {} }, "RestartPolicy": { "Condition": "any", "MaxAttempts": 0 }, "Placement": {} }, "ServiceID": "0h158az9rnp5h8h6nz7ntblg9", "Slot": 1, "NodeID": "6pwuaibhnh0zfyvnsguxp5ffn", "Status": { "Timestamp": "2016-09-13T09:25:18.529371545Z", "State": "rejected", "Message": "preparing", "Err": "VolumeDriver.Mount: {\"Error\":\"open /dev/xvdf: operation not permitted\"}\n", "ContainerStatus": {} }, "DesiredState": "shutdown", "NetworksAttachments": [ { "Network": { "ID": "0dwmxk3ht92316201awf1449p", "Version": { "Index": 35 }, "CreatedAt": "2016-09-13T08:00:28.505174871Z", "UpdatedAt": "2016-09-13T08:00:28.510428839Z", "Spec": { "Name": "collabnet", "DriverConfiguration": { "Name": "overlay" }, "IPAMOptions": { "Driver": { "Name": "default" } } }, "DriverState": { "Name": "overlay", "Options": { "com.docker.network.driver.overlay.vxlanid_list": "257" } }, "IPAMOptions": { "Driver": { "Name": "default" }, "Configs": [ { "Subnet": "10.0.0.0/24", "Gateway": "10.0.0.1" } ] } }, "Addresses": [ "10.0.0.3/24" ] } ] } ]

If you closely look at the logs it is trying to fetch "Err": "VolumeDriver.Mount: {\"Error\":\"open /dev/xvdf: operation not permitted\"}\n", which I doubt is trying to reach out to container or host. I mean container tries to reach out to host machine volume but in our condition we are running RexRay inside the container.

Do suggest.

[vfarcic]

I'm experiencing the same problem. Did you get a solution for it?

[ajeetraina]

I remember I fixed it somehow but missed to update this thread. Let me go back and check.

[bvis]

Hi,

@ajeetraina, I'm sorry. I missed totally your comment.

I finally installed the rexray for docker swarm via commands executed in the shell. I used this approach after some problems I couldn't resolve because the swarm services still do not allow "privileged" flags, the only solution I found for this was to use a mount helper, which does not provide a "clear" solution at all. I hope this will be added in some way to 1.13 in the GA version.

By now I have a script that executes these commands on each cluster node:

echo "libstorage:
 service: efs
 server:
   services:
     efs:
       driver: efs
       efs:
         accessKey:      ${AWS_ACCESS_KEY_ID}
         secretKey:      ${AWS_SECRET_ACCESS_KEY}
         securityGroups: ${AWS_SECURITY_GROUP_ID}
         region:         ${AWS_DEFAULT_REGION}
         tag:            ${CLUSTER_NAME}" > /etc/rexray/config.yml

# Needed by Amazon EFS
sudo apt-get install -y nfs-common
sudo curl -sSL https://dl.bintray.com/emccode/rexray/install | sh -s -- stable 0.5.1
sudo rexray service start

There's a newer version of RexRay (v0.6) which I have not tried. I have used EFS instead of EBS because it can be shared across different availability zones, then I don't care if one task dies and starts in a node that belongs to another AZ. In my cluster I've mounted 3 managers, each one in one AZ, and the workers follow the same sequence, it is the typical approach:

node1 -> zone A
node2 -> zone B
node3 -> zone C
node4 -> zone A
node5 -> zone B
etc.

The formula could be seen as AZ(node(x)) = (x MOD 3) + 1

But I'm interested to hear what you did to make it work! ;)